Solving Roaming Users: HTTP Out for the Splunk Universal Forwarder

The release of version 8.1.0 of the Splunk Universal Forwarder introduced a brand new feature to support sending data over HTTP. Traditionally, a Splunk Universal Forwarder uses the proprietary Splunk-to-Splunk (S2S) protocol for communicating with the Indexers. Using the ‘HTTP Out Sender for Universal Forwarder’ it can now send data to a Splunk Indexer using HTTP. What this feature does is effectively encapsulates the S2S message within a HTTP payload. Additionally, this now enables the use of a 3rd party load-balancer between Universal Forwarders and Splunk Receivers. To date, this is a practice which has not been recommended, or supported, for traditional S2S based data forwarding.

Where the new HTTP Out feature is especially useful is in scenarios such as collecting data from systems in an edge location or collecting data from a roaming user’s device. Typically in these situations it would require more complex network configuration, or network traffic exceptions, to support traditional S2S for the connection from the Universal Forwarder to the Indexers. HTTP Out now allows the Universal Forwarder to make use of a standard protocol and port (443), which is generally open and trusted, for outgoing traffic.

Use Case: The Roaming User

Let’s take a look at how we can use the HTTP Out feature of the Splunk Universal Forwarder to transmit data from the laptop of a roaming user, or generally a device outside of our corporate perimeter, which is an occurrence that has become more and more common with the shift to work from home during the pandemic.

For the purpose of this demonstration, we will be working with the following environment configuration:

  1. Splunk environment in AWS with 2 Indexers and 1 Search Head
  2. Internet-facing AWS Load Balancer
  3. Laptop with the Splunk Universal Forwarder (8.1.0)

Step 1: Configure The Receiver

On our Splunk Indexers we have already configured the HTTP Event Collector (HEC) and created a token for receiving data from the Universal Forwarder. Detailed steps for enabling HEC and creating a token can be found on the Splunk Documentation site here.

Step 2: Configure The Load Balancer

The next thing we need is a Load Balancer which is Internet facing. HTTP Out on the Splunk Universal Forwarder supports Network Load Balancers and Application Load Balancers.  For this use case, we have created an Application Load Balancer in AWS. The Load Balancer has a listener created for receiving connection requests on port 443 and forwards them to the Splunk Indexer on port 8088 (the default port used for HEC). The AWS Application Load Balancer provides a DNS A record which we will be using in the Universal Forwarder outputs configuration.

Step 3: Configure The Universal Forwarder

The last step is to install Splunk Universal Forwarder on the roaming user’s laptop and configure HTTP Out using the new httpout stanza in outputs.conf.

We have installed the Universal Forwarder on one of our laptops and created the following configuration within the outputs.conf file. For ease of deployment, the outputs.conf configuration file is packaged in a Splunk application and deployed to the laptop to enable data forwarding via HTTP.

httpEventCollectorToken = 65d65045-302c-4cfc-909a-ad70b7d4e593
uri = 

The URI address within this configuration is the Load Balancer DNS address which will handle the connection requests to Splunk HTTP Event Collector endpoints on the Indexers. 

The Splunk Universal Forwarder HTTP Out feature also supports batching to reduce the number of transactions used for sending out the data. Additionally, a new configuration LB_CHUNK_BREAKER is introduced in props.conf. Use this configuration on the Universal Forwarder to break events properly before sending the data out. When HTTP Out feature is used with a 3rd party load balancer, LB_CHUNK_BREAKER prevents partial breaking of a data, and sends a complete event to an Splunk Indexer. Please refer to the Splunk Documentation site here for detailed information on the available parameters.

Test and Verify Connectivity

Now that we have our configuration in place we need to restart the Splunk Universal Forwarder service. After this restart occurs we can immediately see that the internal logs are being received by our Splunk Indexers in AWS. This is a clear indicator that the HTTP Out connection is working as expected and data is flowing from the Universal Forwarder to the Load Balancer and through to our Splunk Indexers.

To demonstrate the roaming use case, we have written a small PowerShell script that will run on the laptop. The PowerShell script will generate events printing the current IP address, user, location, city, etc. The Splunk Universal Forwarder will execute this PowerShell script as a scripted input and read the events generated by it. Now, when we search within our Splunk environment we can see that the events being generated by the PowerShell script are flowing correctly, and continuously, to our Splunk Indexers. The laptop connects to the Load Balancer via a home network with no special requirements for network routing or rules.

Let’s now move to a different network by tethering the laptop through a mobile phone for Internet connectivity. This is something that may be common for people while on the road or in areas with minimal wifi access. What we will now observe is that data forwarding to our Splunk Indexers continues without any interruption even though we are on a completely new network with its own infrastructure, connectivity rules, etc. The screenshot below shows that the location and IP address of the laptop has changed however the flow of events from the laptop has not been interrupted.

This configuration could now be deployed to an entire fleet of roaming user devices to ensure that no matter where they are or what network they are on, there is continuous delivery of events using an Internet-facing Load Balancer. This will help IT and Security teams make sure they have the necessary information at all times to support, and protect, their corporate devices.

Looking to expedite your success with Splunk? Click here to view our Splunk Professional Service offerings.

© Discovered Intelligence Inc., 2021. Unauthorised use and/or duplication of this material without express and written permission from this site’s owner is strictly prohibited. Excerpts and links may be used, provided that full and clear credit is given to Discovered Intelligence, with appropriate and specific direction (i.e. a linked URL) to this original content.