Workload management is a powerful Splunk Enterprise feature for users to delegate CPU and memory resources to various Splunk workloads, based on their preferences. As Splunk continues to develop new attributes for the defining of rules, the number of Splunk users who are enabling workload management in their environment is gradually increasing.
Setting up workload rules in Splunk can be challenging when defining the workload rule ‘predicates’ (conditions). It is very important to identify which existing searches are going to be affected by the rules before enabling them. The following guidance should be taken into consideration while implementing workload rule predicates.
Using Role vs User Workload Rule Predicates
When using a role predicate type for a workload rule predicate, you must consider which roles will be affected. All roles that are inherited from a role you define in the predicate will be impacted. For example the admin role will include the user role because the admin role inherits the user role by default in Splunk. To minimize the confusion, it is sometimes easier to use a user predicate type instead of the role predicate type. The user predicate type allows you to select individual or wildcarded users. Alternatively you could create a Splunk role that does not have any capabilities and role inheritances, then assign the role to users that should be included by the predicate.
Using the search_type Workload Rule Predicates
When using the search_type workload rule predicate type, you should consider which searches are going to be affected by the rule. Available search types include:
search_type=adhoc in your rule, you should consider that there are many searches considered to be adhoc searches by Splunk:
- Searches from the search bar
- Searches running inside dashboards
- REST based searches from external sources or users
- Built-in internal searches, typically owned by the splunk-system-user
The other search types relate to scheduled searches. To identify the search_type associated with each of your scheduled searches you can run the following Splunk search:
| stats count by _time savedsearch_name search_type workload_pool user
When using a
search_type relating to scheduled searches in your rule, there are a number of aspects to consider about each scheduled search type:
search_type=summary_indexonly works correctly within ‘Workload Rules’ and not ‘Admission Rules’.
search_type=datamodel_accelerationin your rule with user or role attributes (e.g.
user=test AND search_type=datamodel_acceleration) will not work, because Splunk runs all datamodel accelerations with ‘nobody’ user, even though the datamodel is owned by a different user.
The screenshot below shows ‘Splunk’s Internal Audit Logs – TEST’ data model is owned by the ‘test’ user.
However, the screenshot below shows the data model acceleration search actually runs with a user of ‘nobody’.
search_type=report_accelerationattribute in your rule will not be applied to the underlying scheduled search itself, but will instead be applied to the report acceleration searches linked to the search. As you can see in the screenshot below, the workload of the search test_report_acceleration run with a ‘default’ workload_pool and the report_acceleration run with ‘high’ workload_pool.
Using the search_mode Workload Rule Predicates
There are two different values that can be used when using the search_mode workload rule predicate type –
alltime. Be careful when using
search_mode=alltime predicate in your rule, as any searches that use ‘All time’ as a time range will be impacted. This is likely more than you may first think and includes:
- Any scheduled searches that are not affected by the time range (e.g.
|restsearches), which are typically set to ‘All time’ as the time range.
- Saved searches that use inline time attributes (e.g.
latest_time), may be set to run ‘All time’ as the time range, as users often do not set the time range when using inline time attributes.
- Dashboards may have searches or time ranges set to ‘All time’ by default.
- Several built-in Splunk internal searches, owned by the ‘splunk-system-user’ may be set to run over All time.
Splunk workload management can be a very useful feature to enable. However, it is best to ensure that your Splunk environment – and especially your scheduled searches – are in good shape and cleaned up, before configuring workload management. You should remove any duplicate scheduled searches or any searches that are no longer needed.
In addition, it is important to exercise some search governance for your scheduled searches and adjust timings as necessary. For example, looking at the scheduled time, vs. the time range, vs. the total run time for each search.
Monitoring Splunk Workload Management
There are pre-built dashboards to monitor the Workload Management activities on both Splunk Cloud and Splunk Enterprise. You can find the dashboard under ‘Resource Usage’ in the Monitoring Console of Splunk Enterprise, or you can find ‘Workload management monitoring’ in the ‘Cloud Monitoring Console’ app on Splunk Cloud.
The dashboard displays all the of the scheduled searches by name, which have been affected by the rules you created. The dashboard also shows the adhoc searches that are affected, but does not provide visibility into the syntax of the adhoc search, making it challenging to identify the search that the rule applied to.
© Discovered Intelligence Inc., 2023. Unauthorised use and/or duplication of this material without express and written permission from this site’s owner is strictly prohibited. Excerpts and links may be used, provided that full and clear credit is given to Discovered Intelligence, with appropriate and specific direction (i.e. a linked URL) to this original content.