Splunk Ninjas: New Features and Search Dojo

Editor's Notes

• #2: Here is what you need for this presentation: Link to videos on box: <coming soon> You should have the following installed: 6.4 Overview OI Demo 3.2 – Note this is not on Enablement yet. Please request this from sluedtke@splunk.com. The enablement link will be placed here once availabile. NOTE: Configure your role to search the oidemo index by default, otherwise you will have to type “index=oidemo” for the examples later on. There is a lot to cover in this presentation! Try to go quickly and at a pretty high level. When you get through the presentation judge the audience’s interest and go deeper in whichever section. For example, if they want to know more about Choropleths and polygons spend some time there, or if they want to go deeper on the search commands talk through the extra examples.
• #4: Objective: We want to help you change from this..
• #5: Today I’m going to show you some of the new features available in Splunk 6.4. For TCO & Performance Improvements we’ve created new options to reduce your storage footprint as well as a new event sampling feature to optimize query performance and help you answer questions faster. For Platform Security and Management we have added new single sign-on capabilities, new features to the HTTP Event Collector and finally new views and dashboards to the Distributed Management Console. Then for my favorite part, the new Interactive Visualizations. Not only did we double the amount of visualizations available in Splunk, but we’ve provided a way for developers, partners and the community to create their own and integrate with the Splunk interface natively. Lastly we will go through some of the most commonly used search commands and how they are used so you can become a Splunk Ninja in 6.4!
• #6: To this…
• #7: Splunk safe harbor statement.
• #8: Let’s start with TCO & Performance Improvements.
• #9: Extra Material: Q: How does it affect performance? Can I still search the data? A: You can access the data in all of the normal ways, and for many search and reporting activities there is little impact. But for “needle in the haystack” ad-hoc searches, the performance will no longer be optimal. For “dense” searches (searches whose results return most of the data for the time range searched), the performance impact will be minimal.  For “sparse” or “needle in the haystack” searches (searches that return very few results), searches that typically return in seconds will now return in minutes.  Note: This feature can be selectively applied to any index to provide the greatest amount of flexibility to our customers.   The goal is to apply this feature to data that is less frequently accessed – data for which you are willing to sacrifice some performance in order to gain a very significant cost savings. Splunk specialists can help you set the right policies for the right data.   Q: Do apps and Premium Solutions still work? A: Yes. Apps and Premium Solutions will work. Q: How does it affect performance? Can I still search the data? A: You can access the data in all of the normal ways, and for many search and reporting activities there is little impact. But for “needle in the haystack” ad-hoc searches, the performance will no longer be optimal. For “dense” searches (searches whose results return most of the data for the time range searched), the performance impact will be minimal.  For “sparse” or “needle in the haystack” searches (searches that return very few results), searches that typically return in seconds will now return in minutes.  Note: This feature can be selectively applied to any index to provide the greatest amount of flexibility to our customers.   The goal is to apply this feature to data that is less frequently accessed – data for which you are willing to sacrifice some performance in order to gain a very significant cost savings. Splunk specialists can help you set the right policies for the right data. Q: How do I control what data is minimized? Can I bring data back to the standard state? A: You set policy by data age and by the type of data (index). Different data can have different time criteria for minimization. You can return data to the original state if needed. Splunk specialists can help you set the right policies for the right data.   Q: Why does your optimization data take up so much space? A: Even including the optimization data, Splunk compression techniques have already reduced the customer’s storage requirements by over 50% during indexing. The optimization metadata (TSIDX – time-series index) is what enables the customer to ask any question of their data and handle any type of investigation or use case in real time.   By keeping data in its original unstructured state, Splunk offers the flexibility to ask any question of the data, handling any type of investigation or use case. Splunk structures the answer to each query on the fly, rather than forcing the customer to create a fixed data structure that limits the questions that can be asked. The TSIDX data enables us to deliver this unique flexibility with real-time speed.   Q: Why is the savings range so large (40-80%)? A: The storage used by TSIDX varies depending on the nature and cardinality (uniqueness) of the data indexed. So the savings will vary as well across data types. Repetitive data fields will have a lower savings while unique (high cardinality) data will see a higher savings.  Typical syslog data, for example will fall in the middle – about 60-70%.   High cardinality data returns a higher savings because it requires more index entries to describe it. When the TSIDX is reduced, the savings are larger. We expect most customers will see an overall benefit of 60% or more. We expect the average savings to be 60% or more.
• #10: Platform Security & Management
• #11: DMC In 6.3 we re-worked the Distributed Management Console. In 6.4 we enhanced it even more adding new views and monitoring capabilities for things such as: - HTTP Event Collector Views - Performance tracking for the HTTP Event Collector feature including breakdowns by authorization token. - TCP Inputs - A partner to the Forwarder performance views in DMC tracking TCP queue health and other TCP input statistics. Deployment Wide Search Statistics - Identify top Search Users across a multi-Search Head deployment including frequent and long running searches. - Distributed Search View - A dashboard dedicated to tracking metrics for search in distributed deployments. Includes views for bundle replication performance and dispatch directory statistics. - Resource Usage, I/O - In addition to useful data on CPU and Memory consumption, now also see I/O bandwidth utilization for any Splunk host or across hosts. - Index Performance, Multi-pipeline - Updated views in the Deployment-wide and Instance-scoped Indexing Performance pages to accommodate multi-pipeline indexing. - Threshold Control - Fine-grain controls for visual thresholds for DMC displays containing CPU, Memory, Indexing Rate, Search Concurrency, and Up/Down Status. HTTP Event Collector In 6.3 we added the HTTP Event Collector. Now we’ve improved it by enabling unrestricted data for payloads (besides JSON) and data indexing acknowledgements so customers can verify data was received. SAML And finally we’ve added additional Single Sign On Options for added flexibility
• #12: Platform Security & Management
• #13: Release 6.4 delivers an array of new pre-built visualizations, a visualization developer framework, and an open library to make it simple for customers to access, develop and share interactive visualizations 15 new pre-built visualizations help customers analyze and interact with data sets commonly found in IT, security, and machine learning analysis A new developer framework allows customers and partners to easily create or customize any visualization to suit their needs Splunkbase now contains a growing library of visualizations provided by Splunk, our partners and our community Doubles the visualizations in Splunk today and creates an open environment for the unlimited creation and sharing of new visualizations Once a visual is imported from SplunkBase it is treated the same as any native Splunk feature, and is available for general use in the Visualizations dropdown.
• #14: 15 new pre-built visualizations help customers analyze and interact with data sets commonly found in IT, security, and machine learning analysis. We survey out customer and field to choose an initial set that would meet many common needs.
• #15: The new Event Sampling feature makes it faster to characterize very large datasets and focus your investigations. It is an integrated option of Search, offering a dropdown menu to control sampling 1 per 10, per 100, 1000, 10,000 etc. Of course the performance is equally as fast – a 1 per 1000 search runs 1000x faster.
• #16: Main algo used – Kalgan filter Algorithmic improvements Support bi-variate time series by taking covariance between the individual time series into account. Predict for multiple time series at the same time - this treats individual time series independently, i.e. without computing covariance Predicting missing values in time series and accounting for that during prediction via missing value imputation methods (i.e., “No value was recorded, but it was most likely 5”)
• #17: Use Splunk Ninja App and Demo Instructions
• #18: For more information, or to try out the features yourself. Check out the overview app which explains each of the features and includes code samples and examples where applicable.
• #19: <This section should take ~15 minutes>Search is the most powerful part of Splunk.
• #20: The Splunk search language is very expressive and can perform a wide variety of tasks ranging from filtering to data, to munging, and reporting. The results can be used to answer questions, visualize results, or even send to a third party application in whatever format they require. Although there are 135 documented search commands; however, most questions can be answered by using just a handful.
• #21: These are the five commands you should get very familiar with. If you know how to use these well, you will be able to solve most data questions that come your way. Let’s take a quick look at each of these.
• #22: <Walk through the examples with a demo. Hidden slides are available as backup. NOTE: Each of the grey boxes is clickable. If you are running Splunk on port 8000 you won’t have to type in the searches, this will save time.>
• #24: sourcetype=access* | eval http_response = if(status == 200, "OK", "Error") | eventstats avg(bytes) AS avg_bytes by http_response | timechart latest(avg_bytes) avg(bytes)
• #26: Note: Chart is just stats visualized. Timechart is just stats by _time visualized.
• #27: sourcetype=access* | eval KB=bytes/1024 | stats sum(KB) AS "Sum of KB"
• #28: sourcetype=access* | stats values(useragent) avg(bytes) max(bytes) by clientip
• #29: sourcetype=access* | stats values(useragent) avg(bytes) max(bytes) by clientip
• #30: Eventstats let’s you add statistics about the entire search results and makes the statistics available as fields on each event. <Walk through the examples with a demo. Hidden slides are available as backup>
• #31: Eventstats let’s you add statistics about the entire search results and makes the statistics available as fields on each event. Let’s use eventstats to create a timechart of the average bytes on top of the overall average. index=* sourcetype=access* | eventstats avg(bytes) AS avg_bytes | timechart latest(avg_bytes) avg(bytes)
• #32: We can turn this into a moving average simply by adding “by date_hour” to calculate the average per hour instead of the overall average. index=* sourcetype=access* | eventstats avg(bytes) AS avg_bytes by date_hour | timechart latest(avg_bytes) avg(bytes)
• #34: Streamstats calculates statistics for each event at the time the event is seen. So for example, if I had an event with a temperature reading I could use streamstats to create a new field to tell me the temperature difference between the event and one or more previous events. Similar to the delta command, but more powerful. In this example, I’m going to take the bytes field of my access logs and see how much total data is being transferred code over time.
• #35: To create a cumulative sum: sourcetype=access* | timechart sum(bytes) as bytes | streamstats sum(bytes) as cumulative_bytes | timechart max(cumulative_bytes)
• #36: sourcetype=access* | reverse | streamstats sum(bytes) as bytes_total by status | timechart max(bytes_total) by status
• #37: sourcetype=access* | timechart avg(bytes) as avg_bytes | streamstats avg(avg_bytes) AS moving_avg_bytes window=10 | timechart latest(moving_avg_bytes) latest(avg_bytes) Bonus: This could also be completed using the trendline command with the simple moving average (sma) parameter: sourcetype=access* | timechart avg(bytes) as avg_bytes | trendline sma10(avg_bytes) as moving_average_bytes | timechart latest(avg_bytes) latest(moving_average_bytes) Double Bonus: Cumulative sum by period sourcetype=access* | timechart span=15m sum(bytes) as cumulative_bytes by status | streamstats global=f sum(cumulative_bytes) as bytes_total
• #38: A transaction is any group of related events that span time. It’s quite useful for finding overall durations. For example, how long did it take a user to complete a transaction. This really shows the power of Splunk. Think about it, if you are sending all your data to splunk then you have data from multiple subsystems (think database, webserver, and app server), you can see the overall time it’s taking AND how long each subsystem is taking. So many customers are using this to quickly pinpoint whether slowness is because of the network, database, or app server.
• #39: sourcetype=access* | transaction JSESSIONID
• #40: sourcetype=access* | transaction JSESSIONID | stats min(duration) max(duration) avg(duration)
• #41: NOTE: Many transactions can be re-created using stats. Transaction is easy but stats is way more efficient and it’s a mapable command (more work will be distributed to the indexers). sourcetype=access* | stats min(_time) AS earliest max(_time) AS latest by JSESSIONID | eval duration=latest-earliest | stats min(duration) max(duration) avg(duration)
• #42: There is much more each of these commands can be used for. Check out answers.splunk.com and docs.splunk.com for many more examples.
• #44: Android coming soon!
• #45: Now go do this Fu in your own environment!
• #46: But don’t just say you know the “Fu”…
• #48: <If you have time, feel free to show one of your favorite commands or a neat use case of a command. The cluster command is provided here as an example > “There are over 135 splunk commands, the five you have just seen are incredibly powerful. Here is another to add to your arsenal.”
• #49: You can use the cluster command to learn more about your data and to find common and/or rare events in your data. For example, if you are investigating an IT problem and you don't know specifically what to look for, use the cluster command to find anomalies. In this case, anomalous events are those that aren't grouped into big clusters or clusters that contain few events. Or, if you are searching for errors, use the cluster command to see approximately how many different types of errors there are and what types of errors are common in your data.
• #50: Decrease the threshold of similarity and see the change in results sourcetype=access* | cluster field=bc_uri showcount=t t=0.1| table cluster_count bc_uri _raw | sort -cluster_count