SplunkLive! München 2016 - Splunk Enterprise 6.3 - Data Onboarding

Editor's Notes

• #2: Splunk Enterprise Security
• #4: One of of the key differentiators of Splunk is the ability to digest all machine data and allow users to quickly analyze it for insight. We call this the universal machine data platform. We’ll look at this in more detail in a bit, but for now, understand that the platform was designed around the premise of being able to consume any machine data even if the format changes; something a relational database cannot do. (Splunk Cloud is only available in the U.S. and Canada.)
• #5: Splunk software reliably collects and indexes all the streaming data from IT systems, technology devices and the Internet of Things in real-time - tens of thousands of sources in unpredictable formats and types. Splunk software is optimized for real-time, low latency and interactivity. Organizations use Splunk software and their data the following ways: 1. Find and fix problems dramatically faster 2. Automatically monitor to identify issues, problems and attacks 3. Gain end-to-end visibility to track and deliver on IT KPIs and make better-informed IT decisions 4. Gain real-time insight from operational data to make better-informed business decisions This is described as Operational Intelligence: visibility, insights and intelligence from operational data.
• #6: Our customers typically start with Splunk to solve a specific problem, and then expand from there to address a broad range of use cases, across application troubleshooting, IT infrastructure monitoring, security, business analytics, Internet of things, and many others that are entirely innovated by our customers. Here’s how it works. Splunk software and cloud services reliably collect and index machine data, from a single source to tens of thousands of sources. All in real time. - Once data is in Splunk, you can search, analyze, report-on and derive insights from all your data - across real-time or historical data that may be stored in Hadoop or other NoSQL data sources.
• #7: Splunk Enterprise is the industry leading software for machine data analytics and has been driving innovation and setting the standard for Operational Intelligence since 2006. In the beginning, we were first to introduce the paradigm of ‘search’ to IT – to troubleshoot IT operations and application management issues much faster than ever before and to find the proverbial “needle in the haystack”. When asking customers, they often referred to it as “google for the datacenter”. As the product evolved, Splunk 4 - the engine for machine data - introduced enterprise-class features – dashboards and apps, real-time search and alerts, universal collection and indexing, enterprise controls and map-reduce for horizontal scalability on commodity servers. And then in 2012 we introduced Splunk 5 – this release represented the evolution of Splunk as an Enterprise Platform for Operational Intelligence. It introduced breakthrough innovations and platform features that included:   A new reporting architecture and transparent summarization technology delivering dramatically faster reports A new high availability architecture delivering enterprise-class scale and resilience, even while scaling on commodity servers and storage A robust developer API and SDKs available in mainstream programming languages to enable enterprise developers to leverage Splunk software Big data ecosystem integrations that included Splunk Hadoop Connect, Splunk DB Connect and the Splunk App for HadoopOps And continuing our strategy of delivering you the Platform for Operational Intelligence we introduce you to Splunk 6 - The most advanced version of Splunk software ever. Splunk 6 delivers new and powerful analytics features designed for broader use: non-technical and technical users alike. Splunk 6 is our most advanced version of Splunk software ever – the industry-leading machine data platform. Powerful Analytics: Splunk Enterprise 6 takes large-scale machine data analytics to the next level by introducing three breakthrough innovations: Pivot – opens up the power of analytics to non-technical users with an easy-to-use drag and drop interface to explore, manipulate and visualize data Data Model – defines meaningful relationships in underlying machine data and makes this data more useful to a broader base of users, in particular non-technical users Analytics Store – patent-pending technology that accelerates data models by delivering extremely high performance data retrieval for analytical processing, up to 1000x faster than Splunk Enterprise 5   The new Pivot interface, combined with Data Models and Analytics Store makes it dramatically easier for non-technical users and technical users alike to analyze and visualize data in Splunk. Now more users than ever are empowered by Splunk software to get insights from their machine data.   Intuitive User Experience: Splunk Enterprise 6 includes powerful productivity features for users with a more intuitive user experience: The new Home Experience – gives users instant access to the data, apps and content they care about The Enhanced Search Experience – brings search and reporting together – so users can author rich – dynamic reports - build visualizations – tables – and custom searches – faster than ever before Simplified Management We’ve made Splunk Enterprise 6 easier to deploy, configure and manage – even as customers expand their Splunk Enterprise deployments to the multi-terabyte scale Simplified Cluster Management – deliver easier management of mission-critical Splunk software deployments providing everything the Splunk admin needs to monitor high availability on a centralized dashboard Forwarder Management – support big data scale with easy configuration and management of thousands of forwarders across multiple geographies   Rich Developer Environment And now Splunk Enterprise 6 provides a more powerful developer environment with the integrated Web Framework. Developers can build custom Splunk Apps, customize dashboards, or add advanced functionality - using standard web technologies, such as JavaScript and Django. Splunk 6 represents a significant milestone in our mission to make machine data accessible, usable and valuable by everyone. Find out more at www.splunk.com/6
• #8: One of the great things about Splunk is that you don’t have to start big. Many organizations start using Splunk to solve a single problem. From there, they quickly see the value and begin to expand within the organization. Let’s take a look at how customer deployments often mature. The point I want you to see here is that it’s not unusual to start small, but make sure you plan to go big. Let’s start with Search and investigation, which is where many customers start. Using Splunk, organizations identify and resolve issues up to 70% faster and reduce costly escalations by up to 90%. Splunk is one place to find and fix problems, and investigate incidents across all your IT systems and infrastructure. In the Proactive monitoring stage we begin to monitor IT systems in real time to identify issues, problems and attacks before they impact your customers, services and revenue. Splunk keeps watch of specific patterns, trends and thresholds in your machine data so you don't have to. We trigger notifications in real-time via email or RSS, execute a script to take remedial actions. We can also send an SNMP trap to your system management console or generate a service desk ticket. From here we get into Operational visibility. We can now see the whole picture, track performance and make better decisions. We can visualize usage trends to better plan for capacity; spot SLA infractions, track how you are being measured by the business. We do all of this using your existing machine data without spending millions of dollars instrumenting your IT infrastructure. And finally we reach Real-time business insight. We can now make better-informed business decisions by understanding trends, patterns and gaining Operational Intelligence from your machine data. See the success of new online services by channel or demographic, reconcile 3rd-party service provider fees against actual use, find your heaviest users and heaviest abusers, and more. Because machine data captures every behavior, the possibilities are game changing. You'll find the lead times to get to this intelligence dramatically less than other solutions - measured in minutes/hours instead of months.
• #10: Splunk is the industry-leading platform for Operational Intelligence, delivering both cloud and on-premise solutions tailored to meet the needs of any size organization. Splunk is increasingly being used as a mission-critical, enterprise-wide operational intelligence source, processing 100's of terabytes of data per day. Release 6.3 continues our journey to support the ever-expanding requirements of the most demanding organizations Release 6.3 is especially targeted to meet their needs for scalability and management, extended analysis features, analysis of high-volume data from application and IoT events, and new flexible connectivity options to their business and operational systems. Release 6.3 is a platform release. All 6.3 features are supported on Splunk Enterprise, most on Splunk Cloud, and select features are supported on the Hunk and Splunk Light products
• #11: Organizations are increasingly standardizing their datacenter operations on economically priced servers supporting 16 or more CPU cores. Splunk Enterprise Release 6.3 now supports vertical scaling capabilities to take better advantage of this available power to:   Improve search and reporting performance(Double the performance of most search and reporting activities) Increase data onboarding capacity (Double the peak data onboarding speed vs Double the data onboarding speed) Reduce operating costs(Reduce operating costs by 20% or more)   Previously, Splunk made use of available CPU cores to execute multiple simultaneous searches while indexing data. Release 6.3 vertical scaling uses allows both individual searches and the data indexing process to execute more efficiently by using multiple CPU cores per task. For systems with available CPU cores, the benefits are broad performance improvements in search processing, report generation, data on-boarding capacity and data forwarding efficiency. Why capacity gain overall? Intelligent scheduling should increase capacity somewhat by optimally scheduling jobs Allowing indexing to use additional cores means that burst data can be handled on the same system, and generally that more data/day overall can be processed. This does not necessarily require totally free CPUs to be permanently available, it can just use additional when needed If there is some available CPU capacity, then running searches faster may mean that more can be done We think most customers are not using their systems to full capacity today. Cores do not have to be otherwise idle in order for gains to be seen The net effect of all of this is a 20%+ gain. 50% for typical security scenarios TCO Influencers Indexer HW reduction System capacity gains – data/searches; job scheduling Standardization of datacenter HW configuration on higher core systems Simpler management: DMC, indexer auto discovery, single-instance indexers and forwarders
• #12: Report 1H vs 10 mins – assumes 5 or 6 cores are used. (in next release you can control core usage per search) Data ready in half the time – this is moving from 4 to 8 cores for indexing – so a burst takes half 20% capacity reflects our guidance changing from 250 to 300 GB/day 20% indexing HW – same reasoning Tripled since 2013 is our guidance moving from 100 to 300 (6.0 was 100) Expansion drop 50% - reflects 1/3 less indexer HW, but overall TCO is more than that, so downgraded to 50% instead of saying 66% TCO reduction 1/3 less HW – based on 100 to 300 increase New cost 50% lower – same as expansion cost