Splunk Data Integration – Getting Data Out of Splunk
There are several ways of integrating Splunk within your environment or with your cloud service providers. In this post, we will outline some of the many methods you can use to get data out of Splunk. In a related post, we outline some of the many ways to get data into Splunk.
Discovered Intelligence has implemented all the output methods outlined below for customers. Contact us today to find out more.
Data Output Method Summary
The following table provides a summary of methods that can be used to get data out of Splunk. Each method is then explained further below.
|Splunk GUI||Export data directly from the Splunk GUI|
|Splunk Forwarding||Configure Splunk to stream data out of Splunk to a third-party application|
|Splunk REST API||Remotely execute Splunk searches and export the results|
|Splunk Webhooks||Define custom callbacks on a web resource|
|Splunk ODBC Driver||Connect third-party analytics tools to Splunk via ODBC to export data|
|Splunk Hadoop Connect||Export data from Splunk to Hadoop|
|Splunk Hadoop Data Roll||Archive data to Hadoop or Amazon S3|
A user can export results of a search directly from the Splunk GUI. Data can be exported to as a text file containing the raw events or exported in tabulated/structured CSV, XML or JSON formats. In addition, search results can be e-mailed through alert actions or by executing the sendemail search command. Another useful search command is outputcsv, which will store the search results into a CSV on the Search Head in the following directory: $SPLUNK_HOME/var/run/splunk/csv. The dump search command can also be used to perform a oneshot export of search results to the local disk in the following directory: $SPLUNK_HOME/var/run/splunk/dispatch/<sid>/dump.
One of the main issues with all these GUI based exporting approaches is that they typically do not allow for exporting of massive amounts of data. However, they are great way to export reports or result sets. The capability to export data in this way may also be limited, depending on user access controls that have been set by your administrator.
More information on using the Splunk GUI to export data.
By making revisions to the outputs, props and transforms configuration files, Splunk can be made to forward or stream data to a third-party application using any available network port using a standard syslog format. This can also be accomplished through Splunk apps, such as the Splunk App for CEF, which syslogs data in CEF format. Data can be forwarded from Splunk at index time (i.e. as the data is indexed into Splunk) or at search time (i.e. execute a Splunk search and forward the results on). It is important to execute care when implementing forwarding to other systems, to ensure Splunk queues do not back up and the data being forwarding is accurate and complete.
More information on forwarding data.
Splunk REST API
A user can call the Splunk REST API to export search results. The REST API can be leveraged to execute saved searches or to perform ad-hoc searches. Data can be exported in JSON, CSV or XML formats. The Splunk REST API can be used to export data from any Splunk environment, including on-premise and cloud deployments. The Splunk REST API is feature rich and allows the exporting of massive volumes of data from Splunk – although some skill is required to perform this accurately to maintain data integrity. The REST API is often used by other applications to export data from Splunk via REST or to run saved searches remotely.
More information on exporting data using the REST API.
A webhook custom alert action allows for callbacks to a web resource. For example it might be leveraged to have an alert message pop up in a chat room, to post a notification to a web page or to raise a ticket in an external ticketing system. When an alert is triggered in Splunk, an HTTP POST request is made on a URL. JSON formatted information about the alert is then passed via the webhook. Webhook alert actions are available in Splunk 6.3 or higher.
More information on webhook alert actions.
Splunk ODBC Driver
The Splunk ODBC driver provides connectivity between third-party analytics tools and Splunk. It allows users to combine the data collected by Splunk with existing data from elsewhere within the organization. The ODBC driver is currently compatible with Microsoft Excel, Tableau and MicroStrategy. Features include: role-based access (control access to sensitive information), machine data isolation (only run saved searched so custom queries cannot be executed) and machine data integrity (only read access so the original data is never altered).
More information on ODBC Driver.
Splunk Hadoop Connect
The Splunk Hadoop Connect app allows you to use Splunk to collect and index data in real-time, then send all or a subset of events in a reliable and predictable way to HDFS for archiving, further processing or additional batch analytics. You can optionally pre-process data in Splunk before exporting the results into Hadoop, selecting both the format type as well as specific fields to include. Alternatively, you can simply export the raw events to Hadoop.
More information on exporting data to HDFS or mounted file system.
Splunk Hadoop Data Roll
Splunk Hadoop Data Roll is bundled with Splunk 6.5 and allows warm, cold and frozen data to be archived into the Hadoop file system (HDFS) or Amazon S3. It enables organizations to search data no longer available in Splunk, perform batch processing analysis for archived data and meet data retention policies without using space on the Indexers.
One of the downsides is that searching the archived data will be much slower than if the data were indexed and available in Splunk. Splunk can search across both the data indexed in Splunk and the archived data in Hadoop at the same time. Searches can be tweaked to limit the number of archived buckets that are searched to improve search performance.
More information on Hadoop Data Roll.
Once the data is archived, it can be made available to other Hadoop tools such as Hive and Pig using the Bucket Reader functionality. More information on Bucket Reader.
Looking to expedite your success with Splunk? Click here to view our Splunk service offerings.
© Discovered Intelligence Inc., 2017. Unauthorised use and/or duplication of this material without express and written permission from this site’s owner is strictly prohibited. Excerpts and links may be used, provided that full and clear credit is given to Discovered Intelligence, with appropriate and specific direction (i.e. a linked URL) to this original content.