What to Consider When Creating Splunk Workload Management Rule Conditions

Setting up workload rules in Splunk can be challenging when defining the workload rule ‘predicates’ (conditions). It is very important to identify which existing searches are going to be affected by the rules before enabling them. The following guidance should be taken into consideration while implementing workload rule predicates.

Using Role vs User Workload Rule Predicates

When using a role predicate type for a workload rule predicate, you must consider which roles will be affected. All roles that are inherited from a role you define in the predicate will be impacted. For example the admin role will include the user role because the admin role inherits the user role by default in Splunk. To minimize the confusion, it is sometimes easier to use a user predicate type instead of the role predicate type. The user predicate type allows you to select individual or wildcarded users. Alternatively you could create a Splunk role that does not have any capabilities and role inheritances, then assign the role to users that should be included by the predicate.

Using the search_type Workload Rule Predicates

When using the search_type workload rule predicate type, you should consider which searches are going to be affected by the rule. Available search types include: adhoc, scheduled, summary_index, report_acceleration, and datamodel_acceleration.

When using search_type=adhoc in your rule, you should consider that there are many searches considered to be adhoc searches by Splunk:

1. Searches from the search bar
2. Searches running inside dashboards
3. REST based searches from external sources or users
4. Built-in internal searches, typically owned by the splunk-system-user

The other search types relate to scheduled searches. To identify the search_type associated with each of your scheduled searches you can run the following Splunk search:

index=_internal sourcetype=scheduler
| stats count by _time savedsearch_name search_type workload_pool user

When using a search_type relating to scheduled searches in your rule, there are a number of aspects to consider about each scheduled search type:

• Using search_type=summary_index only works correctly within ‘Workload Rules’ and not ‘Admission Rules’.
• Using search_type=datamodel_acceleration in your rule with user or role attributes (e.g. user=test AND search_type=datamodel_acceleration) will not work, because Splunk runs all datamodel accelerations with ‘nobody’ user, even though the datamodel is owned by a different user.

The screenshot below shows ‘Splunk’s Internal Audit Logs – TEST’ data model is owned by the ‘test’ user.

However, the screenshot below shows the data model acceleration search actually runs with a user of ‘nobody’.

• Using search_type=report_acceleration attribute in your rule will not be applied to the underlying scheduled search itself, but will instead be applied to the report acceleration searches linked to the search. As you can see in the screenshot below, the workload of the search test_report_acceleration run with a ‘default’ workload_pool and the report_acceleration run with ‘high’ workload_pool.

Using the search_mode Workload Rule Predicates

There are two different values that can be used when using the search_mode workload rule predicate type – historical and alltime. Be careful when using search_mode=alltime predicate in your rule, as any searches that use ‘All time’ as a time range will be impacted. This is likely more than you may first think and includes:

• Any scheduled searches that are not affected by the time range (e.g. |inputlookup searches or |rest searches), which are typically set to ‘All time’ as the time range.
• Saved searches that use inline time attributes (e.g. earliest_time and latest_time), may be set to run ‘All time’ as the time range, as users often do not set the time range when using inline time attributes.
• Dashboards may have searches or time ranges set to ‘All time’ by default.
• Several built-in Splunk internal searches, owned by the ‘splunk-system-user’ may be set to run over All time.

General Recommendations

Splunk workload management can be a very useful feature to enable. However, it is best to ensure that your Splunk environment – and especially your scheduled searches – are in good shape and cleaned up, before configuring workload management. You should remove any duplicate scheduled searches or any searches that are no longer needed.

In addition, it is important to exercise some search governance for your scheduled searches and adjust timings as necessary. For example, looking at the scheduled time, vs. the time range, vs. the total run time for each search.

Monitoring Splunk Workload Management

There are pre-built dashboards to monitor the Workload Management activities on both Splunk Cloud and Splunk Enterprise. You can find the dashboard under ‘Resource Usage’ in the Monitoring Console of Splunk Enterprise, or you can find ‘Workload management monitoring’ in the ‘Cloud Monitoring Console’ app on Splunk Cloud.

The dashboard displays all the of the scheduled searches by name, which have been affected by the rules you created. The dashboard also shows the adhoc searches that are affected, but does not provide visibility into the syntax of the adhoc search, making it challenging to identify the search that the rule applied to.

---

Looking to expedite your success with workload management? Click here to view our Splunk Professional Service offerings.

© Discovered Intelligence Inc., 2023. Unauthorised use and/or duplication of this material without express and written permission from this site’s owner is strictly prohibited. Excerpts and links may be used, provided that full and clear credit is given to Discovered Intelligence, with appropriate and specific direction (i.e. a linked URL) to this original content.