Deployment Server Clustering – An Easier Way to Manage Splunk Forwarders

Are you having trouble managing thousands and thousands of Splunk forwarders? Until now, many organizations have typically used numerous deployment servers in a scaling configuration, to manage a massive number of forwarders.

However, to make life easier, Splunk has introduced a new feature called ‘Deployment Server Clustering’, which was introduced with Splunk version 9.2.

What is the Splunk Deployment Server Cluster?

The Splunk Deployment Server Cluster is a combination of the Horizontal Deployment Server Scaling and the Tiered Deployment Server Scaling, but it is much simpler and easier to use. There are new configurations for the Deployment Server Cluster in Splunk v9.2 that can be used to configure this new clustering.

How many clients can a Splunk Deployment Server Cluster support?

To calculate the maximum number of deployment clients that the cluster can support, multiply the number of deployment servers by 25K. The maximum number of deployment servers in a cluster is limited to 3. Therefore, a cluster of 3 deployment servers can service a total of 75K clients.

25000  (max # of clients a deployment server could support)
x 3    (max # of deployment servers in a cluster)
____
75,000 (max # of clients a deployment server cluster can service)

Implementing the Deployment Server Cluster

Pre-requisites

Before you begin, you will need to ensure the following:

• Splunk needs to be running version of 9.2 or higher for the Deployment server.
• At least two deployment servers (the maximum number of deployment servers that you can have in a cluster is three)
• A shared drive, which will be used for all the apps and configurations that the deployment servers in the cluster will use. This shared drive must be a network shared drive, such as an NFS host.
• A load balancer (or a DNS record that is mapped with all deployment servers).
• The Splunk forwarders do not need to be upgraded to version 9.2 or higher.

Steps

1. Add the following configuration to a serverclass.conf (you won’t have this configuration file when starting from scratch) on your deployment servers in a cluster and then restart Splunkd.
   [global]
syncMode=sharedDir
   
   The above configuration is new in v9.2. This configuration will allow deployment apps to share across multiple deployment servers.
   
2. Mount the following directories to the shared drive.
   • $SPLUNK_HOME/var/log/client_events
     • The directory called ‘client_events’ is a new directory from v9.2. The directory tracks the clients’ phone home events via log entries.
   • $SPLUNK_HOME/etc/deployment-apps
     • Within the ‘deployment-apps’ directory, there is a new directory called ‘_splunk_ds_info’. It has a set of shared server class configuration files among the deployment servers in a cluster.
       
3. Configure the load balancer point to the deployment server cluster. Note: You may skip this step when you use a DNS record for the deployment servers.
   
4. Configure deploymentclient.conf on the forwarders to phone home to the load balancer or the DNS record for the deployment servers.

Results

Events are available in the new Splunk internal indexes (those indexes present by default).

• _dsphonehome – tracks the last phone home time of clients.
• _dsclient – contains a list of all clients.
• _dsappevent – displays the status of deployed apps on clients.

On the forwarder management page from the Splunk Web UI of each deployment server you will see:

• Identical lists of apps, serverclasses, and clients
• Identical (or similar) phone home time from all clients.

Considerations

• It is recommended NOT to edit the contents within the $SPLUNK_HOME/etc/deployment-apps/_splunk_ds_info/ directory.
• You will receive duplicated logs on all _ds* indexes because the internal logs are coming from each deployment server.
• If you are configuring server classes with CLI, you must run a reload command to make the changes available to all deployment servers in a cluster. If you configure server classes from Web UI, Splunk automatically triggers the reload after saving.

Troubleshooting

If you face an issue where clients are visible only on one of your deployment servers:

• Stop splunkd on a deployment server that was showing all clients.
• Wait until clients successfully phone home to other deployment servers.
• Start splunkd on a deployment server that was shut down.

If you have duplicated GUID among clients, they may not show up on the forwarder management, and apps may not get deployed. You can check the GUID of the forwarder by looking at $SPLUNK_HOME/etc/instance.cfg. You can reset GUID by removing $SPLUNK_HOME/etc/instance.cfg and restart Splunkd. You will notice instance.cfg repopulated with a new GUID.

---

Looking to expedite your success with Splunk? Click here to view our Professional Service offerings.

© Discovered Intelligence Inc., 2024. Unauthorized use and/or duplication of this material without express and written permission from this site’s owner is strictly prohibited. Excerpts and links may be used, provided that full and clear credit is given to Discovered Intelligence, with appropriate and specific direction (i.e. a linked URL) to this original content.