Machine Data 101: Turning Data Into Insight

Editor's Notes

• #4: A story about Dan Houston, IT manager @ USPS Dan observed that data was trapped in departments – data silos silos led to vulnerabilities in operations and security 239 data types 1000’s of systems 134 billion events indexed each day that’s 3 TB of information T: Journey to understanding value of machine data For those new to Splunk, let me pause for a moment and explain a few key concepts that are important to understand   Let me start with machine data     Before we talk about Splunk, let me tell you about Dan Houston, an IT manager at the United States Postal Service. As the organization responsible for handling 40 percent of the world’s mail volume, the USPS has to manage an enormous, and rapidly growing, amount of information. Dan and his team found that USPS employees were tracking information differently across internal departments and the data was trapped in silos – everybody was doing their own thing from a monitoring and a data collection perspective. The over 2000 IT professionals at the USPS lacked a holistic view of the environment. Without the ability to search, analyze and correlate various data sources from across the organization, the USPS was vulnerable to operational and security issues, like the security breach they experienced in 2014.    <click – build slide> Enter Splunk. Dan and the IT team at USPS now sends data from over 1,100 sources across multiple operating systems and over 239 data types with over 134 billion events indexed (approximately 3 terabytes per day) into Splunk. Splunk provides USPS a single, shared view of its environment for real-time data sharing and collaboration across IT, business units and security to help prevent issues from happening and rapidly dealing with them when they do.  
• #5: Let’s switch to how are customers are using Splunk to get value from their machine data? As I mentioned Splunk is platform - enabling a range of uses cases. Many customer start with IT- realize the value of the machine data in their organization, and expand their use of Splunk across the organization. Yelp started with Splunk for IT operations, then moved to app delivery and business analytics use cases - all in support of enhancing their customers’ experience. Yelp was challenged with scaling and correlating across fractured data sets. Once they brought in Splunk…visibility into 10+ TB of log data. Let’s hear more from Yelp…[video]
• #13: Looking out into the audience, I can see that most of you have gotten the chance, first hand, to see how much the “connectedness” of our world has changed everything. Every industry, every business, in every country is experiencing the effects of digitization and change. Our world is in the midst of massive change This technical renaissance or digital transformation has only just begun and it’s accelerating Software defined everything, Big data, Web scale, Containerization and microservices, Cloud, Mobile, Analytics, and the world of IoT Add to this, customer and end user expectations have never been higher
• #14: There's one thing in common across all of these digital transformation examples - machine data. When someone orders a Lyft, makes a purchase online, or a robot manufactures a car - Machine data is created every step of the way. That's why machine data is the biggest and most valuable aspect of big data.
• #15: The rise of big data has forced IT organizations to transition from a focus on structured, relational data, to accommodate unstructured data, driven by the volume, velocity and variety of today’s applications and systems. As the data has changed from structured data to unstructured data, the technology approach needs to change as well. When you don’t know what data types you’ll need to analyze tomorrow or what questions you need to ask in a week, flexibility becomes a key component of your technology decisions. The ability to index any data type, search across silos and avoid being locked into a rigid schema opens a new world of analytics and business insights to your organization. Schema at Read – Enables you ask any question of the deal Search – Enables rapid, iterative exploration of the data along with advanced analytics Universal Indexing – Enables you to ingest any type of machine data Horizontal scaling over commodity hardware enables big data analytics
• #16: Splunk is the platform for machine data, it digests all machine data and allow users to quickly analyze their data and rapidly obtain insight. The platform was designed around the premise of being able to consume any machine data even if the format changes. A relational database would cannot effectively support constantly changing underlying schemas. Splunk solves this by creating a schema on the fly… Splunk Cloud is only available in the U.S. and Canada.
• #17: The Splunk platform consists of multiple products to fit your needs. The products can pull in data from virtually any source to support multiple use cases. And we continue to invest heavily as new sources become available. There are 1,000s of apps built specifically to extend and simplify deployments by providing pre-packaged content designed for specific use cases and data types. Premium apps that are built by Splunk are available for Security, IT Service Intelligence and User Behavior Analytics .
• #18: Will spend most of our time on the first 3 topics.
• #30: Splunk’s mission statement is to make machine data accessible, useful and valuable to everyone. Splunk can take any machine data and automatically index it for fast searching. Because Splunk doesn’t use a database, there are no additional licenses, and most importantly, no pre-defined schema to limit how you use your information. Examples include the configuration files, syslog, Windows events and registry settings, as well as WMI. But the most important thing to note is how easy it is to get data into Splunk and make it useful.
• #34: The Splunk Stream software captures real-time wire data from distributed infrastructures, including private, public and hybrid clouds with on-the-fly deployment and fine-grained filtering capabilities.
• #39: Splunk DB Connect delivers reliable, scalable, real-time integration between Splunk Enterprise and traditional relational databases. With Splunk DB Connect, structured data from relational databases can be easily integrated into Splunk Enterprise, driving deeper levels of operational intelligence and richer business analytics across the organization. Organizations can drive more meaningful insights for IT operations, security and business users. For example, IT operations teams can track performance, outage and usage by department, location and business entities. Security professionals can correlate machine data with critical assets and watch-lists for: incident investigations, real-time correlations and advanced threat detection using the award-winning Splunk Enterprise. Business users can analyze service levels and user experience by customer in real-time to make more informed decisions.
• #47: To address the needs of developers, operations and product management, you need Operational Intelligence for your mobile apps. This is what we call mobile intelligence. Mobile intelligence provides real-time insight on how your mobile apps are performing, and can correlate with and enhance Operational Intelligence.    Splunk software enables organizations to search, monitor, analyze and visualize machine-generated data from websites, applications, servers, networks, sensors and mobile devices. Splunk MINT helps organizations monitor mobile app usage and performance, gain deep visibility into mobile app transactions and accelerate development Deliver better performing, more reliable apps When a user has a problem with a mobile app, the issue could be isolated or spread across all app versions, handsets and OS types. With Splunk MINT, you can see issues with app performance or availability in real time. Bugs can be addressed quickly, and app developers can gain a head start in creating and delivering valuable app updates. Achieve End-to-End visibility When mobile apps fail, there are many potential sources of failure. With Splunk MINT, you can analyze overall transaction performance. And using Splunk MINT, you can correlate this data with information from back-end apps to gain detailed insight on transaction problems. As a result, operations can reduce MTTR and better anticipate future mobile app back-end requirements. Deliver real-time analytics Mobile apps give enterprises new ways of conducting digital business. With mobile app information in Splunk Enterprise, you can correlate usage and performance information— some call this omni-channel analytics—to better understand how users are engaging all aspects of your organization.
• #48: Splunk MINT Express provides a dashboard that offers and at a glance view of Mobile app health and usage. This includes an overall index called “MobDex”, which provides a blended view of Application usage, crashes, engagement in and abandonment. The insight boxes provide top-level aggregated information, which you can click on to get more specific information, and context.
• #49: Hadoop data roll is an option available to customers who would like to retain their historical Splunk data in their Hadoop data lake. This functionality used to be part of the Hunk product, but it now integrated within Splunk Enterprise and included with your license. It is compatible with most popular Apache Hadoop distributions as well as Amazon EMR running on S3 storage. The main benefit of Hadoop data roll is TCO reduction achieved by reducing the storage footprint and lower cost storage hardware. Additionally, your Hadoop applications will be able to use data that was originally indexed in Splunk. Reduction in storage footprint is achieved by reducing Splunk search optimization data that are primarily used to speed up ”needle in the haystack” type searches. The storage footprint reductions can range from 40-80%, depending on the characteristics of the underlying data. Once data is rolled into Hadoop, search, reporting, and analysis functionality within Splunk Enterprise is retained through virtual indexes, though with performance tradeoffs. Another consideration is that Splunk premium solutions like ES and ITSI don’t yet support use of data that has been rolled into Hadoop.
• #52: Splunk MINT Express provides a dashboard that offers and at a glance view of Mobile app health and usage. This includes an overall index called “MobDex”, which provides a blended view of Application usage, crashes, engagement in and abandonment. The insight boxes provide top-level aggregated information, which you can click on to get more specific information, and context.
• #68: The data for example may have a userid but you want to search on a name. Splunk’s lookup capability can enrich the raw data by adding additional fields at search time by. Some common use cases including event and error code description fields. Think “Page not Found” instead of “404”. Enriching your data can lead to entirely new insight. In the example shown, Splunk took the userid and looked up the name and role of the user from an HR database. Similarly, it determined the location of the failed log in attempt by correlating the IP address. Even though these fields don’t exist in the raw data, Splunk allows you to search or pivot on them at any time. You can also mask data. For example, you may want social security numbers to be replaced with all X’s for regular users but not masked for others. Removing data can also be useful, such as filtering PII, before writing it to an index in Splunk.
• #105: Splunk 6 takes large-scale machine data analytics to the next level by introducing three breakthrough innovations: Pivot – opens up the power of Splunk search to non-technical users with an easy-to-use drag and drop interface to explore, manipulate and visualize data Data Model – defines meaningful relationships in underlying machine data and making the data more useful to broader base of non-technical users Analytics Store – patent pending technology that accelerates data models by delivering extremely high performance data retrieval for analytical operations, up to 1000x faster than Splunk 5 Let’s dig into each of these new features in more detail.
• #106: Data Models are created using the Data Model Builder and are usually designed and implemented by users who understand the format and semantics of their indexed data, and who are familiar with the Splunk Search Processing Language (SPL). They define meaningful relationships in the data. Unlike data models in the traditional structured world, Splunk Data Models focus on machine data and data mashups between machine data and structured data. Splunk software is founded on the ability to flexibly search and analyze highly diverse machine data employing late-binding or search-time techniques for schematization (“schema-on-the-fly”). And Data Models are no exception. They define relationships in the underlying data, while leaving the raw machine data intact, and map these relationships at search time. They are therefore highly flexible and designed to enable users to rapidly iterate. Security is also a key consideration and data models are fully permissionable in Splunk 6.
• #107: Data Models are created using the Data Model Builder and are usually designed and implemented by users who understand the format and semantics of their indexed data, and who are familiar with the Splunk Search Processing Language (SPL). They define meaningful relationships in the data. Unlike data models in the traditional structured world, Splunk Data Models focus on machine data and data mashups between machine data and structured data. Splunk software is founded on the ability to flexibly search and analyze highly diverse machine data employing late-binding or search-time techniques for schematization (“schema-on-the-fly”). And Data Models are no exception. They define relationships in the underlying data, while leaving the raw machine data intact, and map these relationships at search time. They are therefore highly flexible and designed to enable users to rapidly iterate. Security is also a key consideration and data models are fully permissionable in Splunk 6.
• #108: Data Models are created using the Data Model Builder and are usually designed and implemented by users who understand the format and semantics of their indexed data, and who are familiar with the Splunk Search Processing Language (SPL). They define meaningful relationships in the data. Unlike data models in the traditional structured world, Splunk Data Models focus on machine data and data mashups between machine data and structured data. Splunk software is founded on the ability to flexibly search and analyze highly diverse machine data employing late-binding or search-time techniques for schematization (“schema-on-the-fly”). And Data Models are no exception. They define relationships in the underlying data, while leaving the raw machine data intact, and map these relationships at search time. They are therefore highly flexible and designed to enable users to rapidly iterate. Security is also a key consideration and data models are fully permissionable in Splunk 6.
• #122: What does this platform look like? The platform consists of 2 layer: A core engine and an interface layer On top of the platform you can’t run a broad spectrum of content that supports use cases Use cases range from application mgmt. and IT operations, to ES and PCI compliance, to web analytics The core engine provides the basic services for real time data input, indexing and search as well alerting, large scale distributed processing and role based access The Interface layer consist of the basic UI for search, reporting and visualization – it contains developer interfaces, the REST API, SDKs and Web Framework The SDKs provide a convenient access to core engine services in a variety of programing language environments. The Web Framework enables developers to quickly create Splunk Apps by using the modern web programming paradigm including pre-built components, styles, templates, and reusable samples as well as supporting the development of custom logic, interactions, components, and UI. Developers can choose to program their Splunk App using Simple XML, JavaScript or Django (or any combination thereof). These programmatic interfaces allow you to eithe:r: extend Splunk integrate Splunk with other applications build completely new applications from scratch that require OI or analytical services that Splunk provides
• #123: What have developers been building using Splunk Enterprise? Examples include the following: Run searches and retrieve Splunk data from existing Customer Service/Call Center applications (Comcast use case) Integrate Splunk data into existing BI tools and dashboard (Tableau, MS Excel) Build mobile applications with KPI dashboards and alerts powered by Splunk (Otto Group use case) Log directly to Splunk from remote devices (Bosch use cases) Build customer-facing dashboards powered by user-specific data in Splunk (Socialize, Hurricane Labs use cases) Programmatically extract data from Splunk for long-term data warehousing We hope this is just the beginning. We hope to open up a whole new world of enterprise apps.
• #124: The Splunk Web Framework Toolkit is a resource to help developers learn how to build rich applications using the Splunk Web Framework. This app contains a collection of examples that show you how to use the components of the Web Framework. You will find documentation and reference information, along with code templates and additional components that you can use within your own Splunk Apps.
• #125: BUILD SPLUNK APPS The Splunk Web Framework makes building a Splunk app looks and feels like building any modern web application.   The Simple Dashboard Editor makes it easy to BUILD interactive dashboards and user workflows as well as add custom styling, behavior and visualizations. Simple XML is ideal for fast, lightweight app customization and building. Simple XML development requires minimal coding knowledge and is well-suited for Splunk power users in IT to get fast visualization and analytics from their machine data. Simple XML also lets the developer “escape” to HTML with one click to do more powerful customization and integration with JavaScript.   Developers looking for more advanced functionality and capabilities can build Splunk apps from the ground up using popular, standards-based web technologies: JavaScript and Django. The Splunk Web Framework lets developers quickly create Splunk apps by using prebuilt components, styles, templates, and reusable samples as well as supporting the development of custom logic, interactions, components, and UI. Developers can choose to program their Splunk app using Simple XML, JavaScript or Django (or any combination thereof). EXTEND AND INTEGRATE SPLUNK Splunk Enterprise is a robust, fully-integrated platform that enables developers to INTEGRATE data and functionality from Splunk software into applications across the organization using Software Development Kits (SDKs) for Java, JavaScript, C#, Python, PHP and Ruby. These SDKs make it easier to code to the open REST API that sits on top of the Splunk Engine. With almost 200 endpoints, the REST API lets developers do programmatically what any end user can do in the UI and more. The Splunk SDKs include documentation, code samples, resources and tools to make it faster and more efficient to program against the Splunk REST API using constructs and syntax familiar to developers experienced with Java, Python, JavaScript, PHP, Ruby and C#. Developers can easily manage HTTP access, authentication and namespaces in just a few lines of code.   Developers can use the Splunk SDKs to: - Run real-time searches and retrieve Splunk data from line-of-business systems like Customer Service applications - Integrate data and visualizations (charts, tables) from Splunk into BI tools and reporting dashboards - Build mobile applications with real-time KPI dashboards and alerts powered by Splunk - Log directly to Splunk from remote devices and applications via TCP, UDP and HTTP - Build customer-facing dashboards in your applications powered by user-specific data in Splunk - Manage a Splunk instance, including adding and removing users as well as creating data inputs from an application outside of Splunk - Programmatically extract data from Splunk for long-term data warehousing Developers can EXTEND the power of Splunk software with programmatic control over search commands, data sources and data enrichment. Splunk Enterprise offers search extensibility through: - Custom Search Commands - developers can add a custom search script (in Python) to Splunk to create own search commands. To build a search that runs recursively, developers need to make calls directly to the REST API - Scripted Lookups: developers can programmatically script lookups via Python. - Scripted Alerts: can trigger a shell script or batch file (we provide guidance for Python and PERL). - Search Macros: make chunks of a search reuseable in multiple places, including saved and ad hoc searches.    Splunk also provides developers with other mechanisms to extend the power of the platform. - Data Models: allow developers to abstract away the search language syntax, making Splunk queries (and thus, functionality) more manageable and portable/shareable. - Modular Inputs: allow developers to extend Splunk to programmatically manage custom data input functionality via REST.
• #126: Splunk Enterprise empowers developers with application intelligence across the entire product development lifecycle, from monitoring code check-ins and build servers, to pinpointing production issues in real-time and gaining valuable insights on application usage and user preferences. Splunk Enterprise is a robust platform that enables developers to integrate data and functionality from Splunk software into applications across the organization using Software Development Kits (SDKs) for Java, JavaScript, C#, Python, PHP and Ruby. Developers can extend the power of Splunk software with programmatic control over search commands, data sources and data enrichment. Developers can use the tools and languages they know to build Splunk apps with custom dashboards, flexible UI and custom data visualizations, using the Splunk Web Framework.
• #127: Here are just some of the new Splunk Apps that have been delivered over the past year. Their goal is to make it easier to use Splunk for specific technologies and use cases – prepackaging inputs, field extractions, searches and visualizations. Highlight a few apps. These apps along with 100’s of others have been developed not only by Splunk but by partners, customers and members of the Splunk community.
• #130: “After this workshop, if you want more information, all the product documentation is available online. The documentation is divided into several manuals. For reporting and dashboards you will likely be most interested in the User and Developer Manuals.”
• #131: “For a more interactive approach to getting your questions addressed there is Splunk Answers. It is a web based Splunk community of Splunkers like you. Splunk employees are also regular experts on the site.”
• #132: “It is not possible to cover everything you need to know about building reports and dashboards in 30-45 minutes. For more structured training with labs, consider Splunk education courses. These are available as instructor-led web-based courses or onsite if there is enough participants per class.”
• #133: 15 Get Started Videos Select From X Classes Get Splunk Certified in 5 Days Delivered Online, Classroom, Self-Paced, Custom  
• #134: Certification 8 Certification Tracks - Title list chart with checks Image / website
• #135: Certification 8 Certification Tracks - Title list chart with checks Image / website
• #136: Certification 8 Certification Tracks - Title list chart with checks Image / website
• #137: Course Topics Overview of ITSI features ITSI architecture and deployment Installing ITSI Designing and implementing services and entities Configuring correlation searches and notable events Creating deep dive pages Creating glass tables ITSI troubleshooting
• #138: Course Topics Overview of ITSI features ITSI architecture and deployment Installing ITSI Designing and implementing services and entities Configuring correlation searches and notable events Creating deep dive pages Creating glass tables ITSI troubleshooting
• #140: Are you looking to learn, share, and participate with other Splunk users? Visit usergroups.splunk.com, search for <<City Name>>, and join the local user group to receive updates on upcoming meetings!