Migrating Syslog to Cribl Stream: The Art of the “Zero Change” Migration
We’ve all been there. You’re ready to modernize your observability pipeline. You’ve got the green light to move from legacy syslog servers (like syslog-ng) to Cribl Stream. It sounds like a straightforward lift-and-shift, right? But then you flip the switch, and suddenly your downstream SIEM is screaming about unparsed events, your timestamps are drifting, and your load balancers are pinning traffic to a single node.
The cornerstone of a successful migration is simple: No disruption of service. The end user shouldn’t even know we migrated the data. It should look exactly the same as it did when it came through the original source.
In this post, we walk through the critical best practices for migrating Syslog to Cribl Stream. Covering format fidelity, the timestamp trap, and the hidden “gotchas” of TCP load balancing that you won’t find in the basic documentation.
The Golden Rule: Format Fidelity and Zero Change
The first rule of migration club? Don’t change the data. Unless explicitly permitted by the customer or the requirements, the data delivered by Cribl Stream must have precisely the same format as the data supplied by the original syslog server.
However, there is no guarantee that raw data coming into Cribl will match the format your downstream tool (like Splunk) was used to receiving from syslog-ng. Legacy servers often have configuration settings that alter formats before writing to disk.
The Solution: Master the Mask Function
You need to be prepared to manipulate headers on the fly. The mask function is your best friend here. For example, we often see data arriving with a priority header followed by a version number (e.g., <34>1) that wasn’t present in the previous feed.
If you try to fix this live during a cutover, it’s incredibly stressful. We recommend building an arsenal of mask function examples in your pipeline beforehand. This allows you to surgically remove priority headers or fix version numbers without the end user ever noticing a difference.
The Timestamp Trap: handling “Loose” Standards
Accuracy in our data is fundamentally based on the timestamp. If you don’t have an accurate timestamp, your correlations are worthless. While RFC 5424 gave us a standard with timestamps and years, many older devices play loose with these rules, often omitting time zones or years entirely.
When you migrate, you don’t have control over the source device’s configuration. You have to deal with what you get.
3 Ways to Handle Missing Time Zones
If you’re ingesting data that lacks time zone information, you have three options:
- Use a Pre-processor (The Hard Way): You can use the Cribl Syslog Pre-processor Pack to look up hosts in a table and assign time zones. The problem? You have to build and maintain that table. If you have 6,000 devices, that’s a management nightmare.
- Multiple Ports (The Worst Way): We’ve seen deployments where admins open different ports for different time zones (e.g., port 9514 for EST, 9515 for PST). This creates a massive management overhead.
- Default Time Zone Setting (The Best Way): Here is a feature many admins miss. In the Syslog Input configuration under Advanced Settings, you can specify a Default Time Zone. If an event arrives without a time zone, Cribl applies this default immediately. It’s cleaner, faster, and requires zero lookup tables.
Transport Protocols: TCP, UDP, and the Load Balancing “Gotcha”
We all know TCP is the recommended transport. It offers reliability, flow control, and security. However, here is a pro tip from the field: Always configure your Source for both TCP and UDP.
Why? Because even when a client swears their device supports TCP, they often enable it only to find it doesn’t work. By configuring your load balancer and Cribl Input for both protocols upfront, you can switch the load balancer to UDP instantly if TCP fails, without needing to re-engage the customer or change your Cribl config.
The TCP Pinning Problem
In a typical deployment, you have a load balancer sitting in front of your Cribl Worker Nodes. The problem is that standard load balancers are “dumb” – they don’t understand where a syslog message starts and stops. They see a continuous stream of data on a TCP connection and “pin” it to a single Worker Process on a single Node.
This creates a bottleneck where one CPU core is pegged at 100% while the others sit idle.
The Fix and the Hidden Cost
Cribl introduced TCP Load Balancing to solve this. It acts as a receiver that distributes the stream across all Worker Processes on the node. But there is a catch that tripped us up recently.
When you enable TCP Load Balancing, Cribl effectively dedicates one Worker Process to handle that distribution. In a standard configuration, the software reserves two processes for overhead (API, metrics, etc.), denoted as -2.
If you enable TCP Load Balancing without adjusting your settings, you lose another worker for processing. You need to manually increase the worker process count setting from -2 to -3 to ensure the system holds back enough resources for the load balancer service without impacting your processing pipelines.
Resilience: Why You Need Persistent Queues (PQ)
Syslog is a “fire and forget” protocol. It doesn’t buffer. If your destination goes down or applies backpressure, that data is lost.
To mitigate this, we recommend enabling Persistent Queue (PQ) on your Syslog Sources in “Always On” mode.
We hear the objection often: “Won’t writing to disk slow everything down?” Technically, yes, it adds latency. But remember, you are migrating from a legacy syslog server that was already writing everything to disk. If your old infrastructure could handle the I/O, your new Cribl infrastructure can too. “Always On” PQ ensures that if downstream pressure builds, your data is safe on the disk rather than vanishing into the ether.
Conclusion
Migrating syslog isn’t just about repointing IP addresses; it’s about ensuring data fidelity, time accuracy, and architectural resilience. By respecting the Zero Change rule, utilizing the Default Time Zone settings, and properly configuring TCP Load Balancing (remember the -3 setting!), you can ensure a migration that is invisible to your end users and robust for your operations team.
Ready to bulletproof your data migration strategy? View our Cribl Professional Service offerings.
Discovered Intelligence Inc., 2026. Unauthorized use and/or duplication of this material without express and written permission from this site’s owner is strictly prohibited. Excerpts and links may be used, provided that full and clear credit is given to Discovered Intelligence, with appropriate and specific direction (i.e. a linked URL) to this original content.
