Beyond Smart: When ‘Always On’ Mode is the Best Choice for Cribl Persisent Queues
If your Cribl environment was set up a few years ago, it might be time to revisit some of your settings—particularly the Persistent Queue (PQ) settings on your source inputs. Recently, while troubleshooting an issue, I discovered that the PQ settings were the root cause of the problem. I wanted to share my findings in case they help you optimize your Cribl setup.
The issue I encountered involved failed Splunk Universal Forwarder (UF) connections to Cribl. After verifying that the UF settings were correct, I focused on Cribl. During my investigation, I discovered that one of my Splunk TCP inputs consistently triggered backpressure. As I dug deeper, I found that the input’s PQ mode was set to “Smart.” This wasn’t surprising, as “Smart” was the default PQ mode when the environment was first configured. However, Cribl’s newer releases now default to “Always On.”
Cribl PQ Mode: Smart vs. Always On
Curious if switching the mode could improve performance, I changed the PQ mode from “Smart” to “Always On.” Before making the change, I used Cribl’s built-in Monitoring tools to capture critical metrics for the worker nodes, including Free Memory, CPU Load, and input throughput for the source input. After switching the PQ mode, I was pleasantly surprised to see a 50% increase in throughput, with no adverse effects on the worker nodes.
I was curious about the significant difference in throughput, so I did some research. By “research,” I mean I spent some time browsing Cribl’s Slack channel and asking a few questions. Smart mode requires the CPU to process each event to determine whether PQ needs to be activated. This consumes CPU cycles that could otherwise be used to process events. In contrast, “Always On” mode—true to its name—simply writes events directly to disk without checking, allowing the CPU to focus on processing events instead. The best part is that “Always On” mode won’t cause blocking unless your storage is full.
While I can’t guarantee the same 50% boost in every environment – since results can vary – it’s worth investigating. Testing the different PQ modes could make a big difference if you’re running into backpressure issues or want to optimize your setup.
Looking to expedite your success with Splunk and Cribl? View our Cribl Professional Service and Splunk Professional Service offerings.
Discovered Intelligence Inc., 2024. Unauthorized use and/or duplication of this material without express and written permission from this site’s owner is strictly prohibited. Excerpts and links may be used, provided that full and clear credit is given to Discovered Intelligence, with appropriate and specific direction (i.e. a linked URL) to this original content.
