Splunk and the Internet of Things (IoT)

What’s new in Splunk 6.3?

The release of Splunk Enterprise 6.3 saw improvements in performance, management and Total Cost of Ownership (TCO). Data integrity control, intelligent job scheduling, custom alert actions, geospatial visualization, anomaly detection and the addition of the HTTP event collector are only a few of the new features available in 6.3. This blog posting will discuss two of these features in more detail and what it means for Splunk’s push into the Internet of Things (IoT).

Anomaly Detection

Anomaly detection uses machine learning analytics to recognize patterns and events which deviate from what is normal, or in other words, expected. The concept of machine learning is not new, but implementing such a system generally requires an extended amount of time, resources and human configuration. Another limitation, which is a result of the amount of human input necessary, is that the system cannot look for what it does not know. Therefore, the system will not recognize valuable information its users have not identified for it.

Splunk has simplified the process of analyzing these patterns into an anomalydetection search time command. This new command will help to identify anomalous events by computing a probability for each event and then detecting unusual small probabilities. Some use cases for this feature include:



Information Technology Identify a DDoS attack from IP address ranges
Marketing Rare but high-value customer purchase patterns
Product Rare or previously unknown method of using a product yielding better results or yielding results more efficiently
Security Faster-than-human transaction – Detecting when transactions are performed more quickly by one user than by others, which may indicate a bot or an attempt to probe security measures
HTTP Event Collector

The HTTP Event Collector is a token-based JSON API for sending events to Splunk from anywhere without the use of a forwarder. The only options Splunk users had until this point included the use of TCP/UDP inputs or the rest API, both of which have limitations and are not built for this purpose. The HTTP Event Collector can help users gain visibility into applications within Splunk, capture events from external systems and capture external events for devices. This new feature is scalable and capable of handling millions of events per second and works out of the box. Some use cases for this feature include:



Manufacturing Sensors sending data to determine mechanical problems for predictive maintenance
Marketing Receiving data about internet activity from customer devices while they are connected to a Wi-Fi network
Smart Buildings & Smart Cities Sensors sending data for predictive maintenance, building conditions and tenant behaviour
Transportation Telematics, asset management and logistics

Before looking into how these features will integrate with IoT technology, let us look at what is necessary for its growth and the challenges the industry faces.

The Internet of Things

It is difficult to determine exactly how many M2M connections there will be in the next decade, but experts agree IoT is the next big ICT wave which will transform the world we live in. The range in the estimates listed below is impacted by a few factors: how each source defines IoT and its components, how these machines are connected (some sources may not include Wi-Fi connection or include only devices enabled by a cellular network), the future state of the global economy in the next 5 years and the accessibility to IoT enabling technologies.

Estimated Number of M2M Connections Globally by 2020 (Billions)

M2M Connections

Data adapted and retrieved from: Ericsson Press Release, Machina Research, Cisco, Gartner Press Release and Lund et al., IDC.

The growth of IoT though raises several questions. How will all these devices communicate with one another? How can we simplify the integration of different machines participating in dynamic environments? How will we manage the amount of data being produced? Who does the data belong to? What kind of information will organizations be allowed to collect on customers and citizens?

As IoT evolves, these are the types of questions that must be answered and will either enable or inhibit the growth of IoT. In some cases, work is already being done to resolve these issues, but others will take decades i.e. the standardization of radio frequencies at a global level for device interoperability.

How Does Splunk Fit in with the Internet of Things (IoT)?

With so many devices communicating in a number of different methods and formats, and generating an unprecedented amount of data, the question on everyone’s mind has become what story and value do these 1’s and 0’s tell? Organizations need a platform capable of ingesting diverse forms of data from a variety of systems and provide the ability to easily analyze the environment in its entirety. With Splunk’s HTTP Event Collector, organizations are capable of receiving and analyzing data from devices they were not previously able to communicate with.

There already exist standardization efforts to enable better technology integration. Telecoms, software providers, developers, hardware vendors and professional service companies are already forming partnerships to provide customers from all verticals with end-to-end solutions to make the shift to IoT less complicated. Standardization efforts will take time and until then platforms such as Splunk, with the ability to ingest diverse forms of data, will be essential.

There are also an enormous number of security vulnerabilities due to wireless connectivity between devices. IoT networks will be dynamic with devices constantly being added or removed and this is where Splunk’s anomaly detection becomes useful. Enterprise security practices traditionally perform analysis on historical data – trying to identify attacks in the past and how it can be prevented in the future. However, by that time, it is more than likely that attackers will have extracted information. 75% of attacks begin data exfiltration in minutes, but half of them persist without detection for months or years.[1] Organizations are already having difficult managing security. In 2014, the number of detected incidents rose to almost 43 million, an increase of close to 50% from the previous year.[2] With more attacks, came more financial damage. In 2014 there was also a 51% increase in the number of companies reporting a loss of $10 million USD or more.[3]

Splunk’s anomaly detection is a way of alerting abnormal behaviour. With IoT being so dynamic, humans will be unable to configure a system for an ever changing environment. As well, with IoT being relatively new, people are still unaware of what to look for, what data points are useful and the value (if any). This type of statistical analysis allows for the Splunk platform to be deployed in different environments i.e. smart buildings, transportation logistics, smart cities, energy usage, etc. Splunk is not the only platform being touted with these capabilities, but Splunk’s versatility and the addition of these new features is positioning the company to become one of the major players for the Internet of Things.

[1] “Cisco Annual Security Report,” Cisco, 2014.

[2] “Cybersecurity 2014 Breaches and Costs Rise, Confidence and Budgets Are Low,” CSO Magazine, 2014.

[3] “Verizon 2014 Data Break Investigations Report,” Verizon, 2014.

Looking to expedite your success with Splunk? Click here to view our Splunk service offerings.

© Discovered Intelligence Inc., 2016. Unauthorised use and/or duplication of this material without express and written permission from this site’s owner is strictly prohibited. Excerpts and links may be used, provided that full and clear credit is given to Discovered Intelligence, with appropriate and specific direction (i.e. a linked URL) to this original content.