Log I am your father

Editor's Notes

• #14: At Splunk, our mission is to make machine data accessible, usable and valuable to everyone. And this overarching mission is what drives our company and product priorities.
• #16: Splunk products are being used for data volumes ranging from gigabytes to hundreds of terabytes per day. Splunk software and cloud services reliably collects and indexes machine data, from a single source to tens of thousands of sources. All in real time. Once data is in Splunk Enterprise, you can search, analyze, report on and share insights form your data. The Splunk Enterprise platform is optimized for real-time, low-latency and interactivity, making it easy to explore, analyze and visualize your data. This is described as Operational Intelligence. The insights gained from machine data support a number of use cases and can drive value across your organization.
• #17: Splunk provides an open, fully integrated platform. That means you can collect, index, analyze, report and predict on machine-generated data from a single product. It’s enterprise-ready with high availability and disaster recovery features, role-based access control and scales to index hundreds of terabytes per day. It’s an open platform with over 500 Splunk Apps available and allows for custom development.
• #18: More than 10,000 customers in 100 countries have purchased the enterprise license of Splunk. This includes a majority of the Fortune 100. Enterprises, service providers and government agencies in 100 countries use Splunk to improve service levels, reduce IT operations costs, mitigate security risks and drive new levels of operational visibility. As they gain new visibility into their real-time and historical machine data, Splunk’s customers are finding answers and solving the most challenging issues facing IT and the business.
• #21: Now you can onboard data directly from any application or device– opening up new types of machine data to the benefits of Splunk analysis. The new Event Collector makes it simple and efficient to collect this data, scaling to millions of events per second, using a developer-friendly, standard HTTP/JSON API and logging libraries And NO FORWARDERS. Today it is possible to send data directly to Splunk using Modular Inputs or a TCP connection, however this is not an efficient or scalable solution. While log files and forwarders provide an efficient mechanism for typical log and syslog files, use of files and forwarders is not possible or necessarily a desired data collection method for the world of custom applications DevOps, Docker, and other packaged application environments. The same is true for the world of IoT event data, where devices/apps need have no local storage, and even intermediate event collection systems and partners would prefer to use a real-time interface to Splunk rather than create specific log files and use forwarders.   The HTTP Event Collector (EC) uses a standard API and high-volume Splunk endpoint to allow events to be directly sent/collected at extreme velocity. The HTTP/JSON API is a developer-standard, whose simple but powerful functionality will be attractive to DevOps and custom application developers and operations managers. Without requiring new system configuration, log creation or administration support, developers can instrument their applications to understand usage flows, performance, error conditions and more. The interface/functionality is also a fit for IoT software developers to connect their devices either directly or via intermediate collection services. The data volumes supported by Splunk are ideal for the transactional and diagnostic data of devices such as Point-Of-Sale systems, vending machines, gaming consoles, automobiles and other devices/systems – opening up a new world of machine data to the benefits of Splunk analysis An ecommerce saas provider uses EC to analyze the performance of their Docker apps A large video game company uses EC to monitor security of a popular sports game. 16B events collected in first 2 months
• #22: NY Air Brake – Optimizing fuel efficiency per route Large Telecom – Main use case is to forecast possible telecom network outages - “intelligent performance-based alarming”. Happy ITSI customer who wants to do more free form ML with more data.  Another Large Telecom – Proactive cell tower outage prediction and avoidance Online real-estate marketplace and app – First use case is website app troubleshooting; Second is identifying fraudulent web visitors. Electronics company specializing in video games – Predicting and avoiding online services outages Financial Services company – Detecting lateral movement inside network Another Financial Services company – ML services to app developers and support Large food and beverage company – Real-time inventory forecasting
• #24: Industry: Automotive/Manufacturing Use case: IoT and Analytics More can be found at: http://www.techworld.com/news/big-data/volkswagen-group-uses-splunk-underpin-iot-experiment-3575744/ Splunk have been working with VW’s data labs on their connected car and IoT program. Splunk is being used to analyse sensor data from VW’s E-Up cars. Splunk’s ability to search, alert, report and analyse IoT data allowed VW to see a fleet of E-Up cars at CeBIT and analyse metrics such as speed, RPM, battery level, range, temperature, when the doors were open and when windscreen wipers were on. This data could also be combined with location, maps and smart watches to show where vehicles had been, heatmaps, and driver heart rate
• #25: Custom Alert Actions provide the ability to use Splunk Alerts to trigger custom actions or pre-packaged integrations with 3rd party products such as trouble ticketing or support systems. Developers can build and publish integrations or custom action packages that users or admins can use via a simple menu within the Splunk Alert Interface. Splunk and partners provide a growing set of integrations including, ServiceNow, xMatters, Webhooks and more. Previously these integrations were complex, ad-hoc efforts requiring custom scripts. The new scheme makes it simple for partners (and customers) to create and contribute out-of-the-box integration templates, and for customers to use them via a simple pull-down menu.
• #26: There are many free add-ons and Apps for Splunk software that simplify the connection and collection of data from both industrial systems and the Internet of Things. These include:   Rest API Modular Input: Poll local and remote REST APIs and index the responses. Amazon Kinesis Modular Input: Index data from Amazon Kinesis, a fully managed service for real-time streaming data. Apache Kafka Modular Input: Index messages from Apache Kafka messaging brokers, including clusters managed by Zookeeper. DB Connect 2: Integrate structured data sources with your Splunk real-time machine data collection. Universal Forwarder for Linux (ARM – Raspberry Pi): Dedicated Splunk package for Linux and ARM based systems where data needs to be collected directly from embedded devices such as the Raspberry Pi. MQTT Modular Input: Index messages from MQTT, a machine-to-machine connectivity protocol, by subscribing Splunk software to MQTT Broker Topics. AMQP Modular Input: Index data from message queues provided by AMQP brokers. JMS Modular Input: Poll and index message queues and topics from messaging queues and topics, including MQTT messages, provided by message providers, including TibcoEMS, Weblogic JMS and ActiveMQ. Protocol Data Inputs: Recieve data via a number of different data protocols such as TCP , TCP(s) ,HTTP(s) PUT/POST/File Upload , UDP , Websockets , SockJS. Splunk App for Stream: Capture, filter and index real-time streaming wire data and network events. COAP Modular Input: Index messages from a COAP (Constrained Application Protocol) Server. SNMP Modular Input: Collect data by polling SNMP attributes and catching SNMP traps from datacenter infrastructure devices providing cooling and power distribution. In addition, Splunk has a powerful ecosystem of technology partners. Kepware Technologies – Connects Splunk software with thousands of industrial devices communicating on over a hundred proprietary industrial protocols. Stream real-time data to Splunk from industrial control systems, including SCADA. Carvoyant – Connected car platform, integration with Splunk software allows enterprises to monitor their automobile fleets, including geo-location, engine parameters and diagnostics. B&B SmartWorx – Intelligent sensors and gateways. Integration with Splunk (Splunk App) will include sensor data collection (via MQTT), and gateway and sensor network diagnostics and cyber security. Bluvision– Intelligent beacons. Integration with Splunk (Splunk App) will include beacon data collection (via Websockets). Powerful retail applications. ThingWorx (PTC) – The leading IoT Application Development Platform. Seamless data exchange between ThingWorx applications and Splunk Enterprise and Splunk Cloud, and ThingWorx customers can access Splunk search and analytics through the ThingWorx mashup builder. Buddy.com – Cloud services for connected devices. Integration (Splunk App) will allow Splunk to stream data from any device connected to the buddy platform. Octoblu (Citrix) – IoT developer platform. Has created libraries that allow any Octoblu-enabled device to stream its data to Splunk software and allows those same devices to use Splunk search and analytics to inform their own decisions and logic Red Balloon Security – Security platform for the defense of embedded systems in the enterprise (IP Phones, Printers, switches and routers, etc). Uses proprietary firmware level protection and appliance-based endpoint monitoring, and is integrating (Splunk TA/ES Compliant CIM) with Splunk software to allow Enterprise Security monitoring of threats to embedded enterprise devices. Bayshore Networks – Content-aware cyber security platform for industrial networks. Is integrating (Splunk TA/ES Compliant CIM) with Splunk software to allow Enterprise Security monitoring of threats to SCADA and other industrial networks. Foxguard Solutions– Cyber security and compliance solutions for industrial networks built with Splunk. NERC-CIP compliance specialists. UltraElectronics-3eti – Cyber security platform for industrial networks. Building ES compliant TA to allow collection and analysis of security relevant ICS data in Splunk. Distrix – Software defined networking for industrial networks and the internet of things. Simplifies connectivity and delivers and enhances data over extremely complex networks. Distrix’s SDN supports Splunk to Splunk communication, and can enhance other data, including timestamping and meta-data enrichment, for ingestion in Splunk. Prelert – Anomaly detection app for Splunk Enterprise. Valuable app for management of sensors and devices where rapid identification of anomalies in sensor readings or operations are critical. Predikto – Leverages the power of Predictive Analytics enabling organizations to use their data to predict future asset failures. N3N – Custom, advanced user interfaces for Splunk specializing in isometric views of industrial facilities. R Project App – harness the power of R statistical processing language directly from Splunk interfaces and search processing language. D3.js – Data driven documents for powerful user experiences. HTML5 – Advanced web interfaces and applications for browser and mobile based user experience.
• #34: Coca-cola uses Splunk to understand connected vending machines, including the Coca-Cola Freestyle. They are able to better understand patterns in consumption as well as the performance of the machines, both of which ultimately lead to better customer satisfaction.
• #40: Splunk have been working with VW’s data labs on their connected car and IoT program. Splunk is being used to analyse sensor data from VW’s E-Up cars. Splunk’s ability to search, alert, report and analyse IoT data allowed VW to see a fleet of E-Up cars at CeBIT and analyse metrics such as speed, RPM, battery level, range, temperature, when the doors were open and when windscreen wipers were on. This data could also be combined with location, maps and smart watches to show where vehicles had been, heatmaps, and driver heart rate
• #46: Visuals need to be worked on
• #47: New York Air Brake’s Train Dynamic Systems Division is using Splunk to manage inter-train forces, the “slinky factor” inherent in large freight trains with 6 inches of flex between cars. With splunk, they are able to produce insight and reports allowing the owners of the locomotives they manage to better train the engineers, and better manage the acceleration and braking of the trains throughout thousand mile journeys. Managing this data with Splunk, they can produce 5-10% fuel savings for customers. For their largest customers this can mean a billion dollars in savings a year.
• #48: Energy efficiency is one of the most critical factors in any building project. It's estimated that commercial buildings consume nearly 20% of all energy in the U.S., the majority of which is used for lighting and indoor climate control. From site selection and layout to building materials and mechanical systems, energy efficiency is a primary goal at every stage in a building's lifecycle. In early 2012, McKenney's was enlisted to assist Gulf Power and its partner Chevron Energy Solutions in implementing a new energy management system at Eglin Air Force Base through Gulf Power's GSA contract services in Florida. At 724 square miles, Eglin is one of the largest military bases in the world and includes hundreds of buildings and a base population of about 17,000. The Enterprise Intelligence Group leveraged its in-house experience with Splunk to provide Eglin with continuous collection and aggregation of data from most of the energy management, building control systems and utility metering and monitoring systems on base, helping to monitor and analyze tens of thousands of sensors and data inputs from HVAC systems in more than 100 Eglin buildings. The new Eglin energy management system (EMS) will leverage the Splunk to provide dashboards that will help base maintenance staff to assess building performance and energy efficiency, generate automated Air Force/DoD energy usage reports, compare current energy usage with historical data, and enable the deployment of load shedding and load shifting strategies to take advantage of favorable electric rates. The project is projected to save about $2.5 million annually, with a payback period of less than three years.
• #49: Comcast Corporation (Nasdaq: CMCSA, CMCSK) (www.comcast.com) is one of the world’s leading media, entertainment and communications companies. Comcast has many different Splunk use cases. One of their use cases involves taking data from the set-top boxes to gain real time insights in to customer interaction with content served up by the set top box. Each set top box has a media access control (MAC) address that is unique and is associated with a specific customer. The set top box is capturing all customer interaction with device including which content the customer searched for, what the date of search was, what search results were displayed (this information is recorded a unique identifiers called IDA numbers) and what content was purchased. However, the set top box does not have any information on the customer including their profile. That information is stored in the billing system. Comcast is using Splunk to correlate data across set top boxes and billing systems to gain real-time business insights. Using the correlation criteria of MAC address, content displayed in search and time of purchase, Comcast is gaining a broad range of business insights into their customers. For example, these insights are helping Comcast understand revenues driven by search. By overlaying this information with geo location data, they are able to improve content mix and drive higher monetization. These insights are also helping Comcast improve content promotion based on region. Comcast is using the Splunk and Hadoop integration to visualize Comcast setbox log information. The setbox data comes to Hadoop, get pre processed and moved to Splunk for visualization.  Hadoop Input = High volume of data from many systems along a complex workflow, Developers expressing artistic prerogative on log formats, Many different data sources and formats Splunk Output = Drive operational intelligence, Improve user experience, Troubleshooting, root cause analysis, Track and measure success, Reports, alarms