SplunkLive! Frankfurt 2018 - Data Onboarding Overview

Editor's Notes

• #4: Today’s goal is to talk about Data Onboarding or “Getting Data Into Splunk” from a ”New to Splunk” perspective. More specifically we’ll talk about the following and then do a little bit of demo. 1. Splunk Platform – a refresher You’ve seen the Splunk Overview, but I want to quickly go through a few overview slides and relate why data onboarding is important to them 2. What can Splunk Eat Then we’ll identify not only the data sources that Splunk can collect, but the methods of collection as well 3. Apps and Add-ons Next we’ll discuss how Apps and Add-ons from the ecosystem play a role 4. Data Onboarding Examples/Demos We’ll get into a few demos 5. Data Onboarding Best Practices and Next Steps And finally we’ll get into some common best practices and what to do from here!
• #6: 1. Explain the different components at a high level 2. The forwarder is one of the many ways to collect data in Splunk – we will discuss setting up and using a forwarder in more detail later in the presentation
• #7: 1. Spend some time talking about each collection method 2. Today we will concentrate on and demo the ones highlighted in blue
• #8: Universal Forwarders provide reliable, secure data collection from remote sources and forward that data into Splunk software for indexing and consolidation. They can scale to tens of thousands of remote systems, collecting terabytes of data.
• #9: Heavy forwarders allow for the aggregation, filtering and routing of data, as well as serving as a “data collection node” for applications such as DB Connect and other API-driven data sources. They are typically *not* used for local data collection.
• #10: HTTP Event Collector (HEC pronounced H-E-C) is a new, robust, token-based JSON/raw API for sending events to Splunk from anywhere without requiring a forwarder. It is designed for performance and scale. Using a load balancer in front, it can be deployed to handle millions of events per second. It is highly available and it is secure. It is easy to configure, easy to use, and best of all it works out of the box.* A few other cool tidbits, it supports gzip compression, batching, HTTP keep-alive and HTTP/HTTPs.
• #12: Splunk apps and add-ons: what & why? Splunk apps allow developers to extend data ingestion and processing capabilities of Splunk Enterprise for your specific needs. Apps facilitate more efficient completion of domain-specific tasks by the end user. High-level perspective A Splunk app is a prebuilt collection of additional capabilities packaged for a specific technology, or use cases, which allows a more effective usage of Splunk Enterprise. You can use Splunk apps to gain the specific insights you need from your machine data. Depending on the type and complexity of those use cases, and also whether the developer wants certain app parts to be configured or distributed separately (potentially by a third party), an app may rely on various add-ons. An add-on is a technical component that can be re-used across a number of different use cases and packaged with one or more Splunk apps. Add-ons may contain one or more knowledge objects, which encapsulate a specific functionality focused on a single concern and its configuration. Using an add-on should help to reduce the technical risk and cost of building an app. 
• #14: Additionally we have the community! The community provides thousands of apps and add-ons that can help you onboard and ingest thousands of different data types and new content is added everyday!
• #15: Let’s look at how we would use an Add-on from Splunkbase to get data in. Use an example that you are comfortable with and showcases using an add-on to get data in and mapped properly.
• #16: < If you have another data source or want to improvise a little here feel free – otherwise you can use the following demo flow below > < Support files can be found here: LINK > 1. Install an instance of Splunk on your laptop. 2. Create an inputs.conf that monitors a directory that will contain the PANW logs files, using the PANW sourcetype from the TA.  Leave the directory empty for now. 3. Show the data preview wizard with the apache data.  Show how Splunk understands (and assigns an appropriate sourcetype) to the data.  Show proper field extractions when ingest is complete. 4. Use the wizard to upload one of the 5 PANW data files.  Show how the sourcetype is *not* automatically set, and that there are no relevant choices in the sourcetype picker in the Wizard.   Set the sourcetype to some arbitrary value.  Show that there are no relevant field extractions after ingest. 5. Now, install the PANW app. Make sure to RESTART. 6. Use the wizard to import the next PANW data file.  Now show that there *is* a relevant sourcetype in the picker.  Select it. 7. Show how fields are extracted properly in the data.  *HOWEVER* -- note that the original sourcetype is automatically changed by the TA, and you will get no results when jumping from the wizard into the search window.  Instead, show the 5 or 6 new sourcetypes that get generated as a result of the TA doing its thing. 8. Lastly, deposit the 3rd PANW data file into the monitor directory set up earlier.  Show the data in search, correctly sourcetyped. 9. Move the file to a “backup” filename in the monitored directory.  Show how Splunk does *not* reingest the data. 10. Add the 4th PANW data sample.  Show how the UF handles this.
• #17: In this next section we are only looking at the tip of the iceberg. Data Onboarding can quickly become an advanced topic so the point of this next session is to introduce you to some of the most important/key points to get you started. After that you’ll need to do some research and learn the specifics yourself.
• #18: These are the components that make up a successful Splunk program – both large and small. In a very large deployment, individual people (or more) can be dedicated to each of these components.
• #19: Appropriate staffing will ensure these components are properly addressed. The person responsible for data onboarding from an architectural perspective is the Knowledge Manager
• #20: It is important to have a defined, documented, and repeatable process for data onboarding.
• #21: Explain Index Time Spend some time saying why these are so important for Splunk. Mention there will be references and resources at the end of the presentation to help dive deeper into these topics.
• #22: It is important to not only get the technical details right, but also the data stewardship issues: Who owns the data, who can see it, and how long to keep it?
• #23: It is important to “do the homework” prior to onboarding, not only to get the index-time parameters correct (previous slide) but also to ensure the resultant data in Splunk will be of value to the widest variety of people and use cases
• #24: Make sure to show this in the demo, this slide is just a follow up reminder
• #25: These are min number of parameters, that should be set when creating a new data source. Again, I like said when I flashed up the Splunk Apps site.. Find something similar to your source and re-work it. But make sure it includes these parameters.
• #26: Normalizes data from different sources – Host and hostname discussion
• #27: Syslog represents almost 50% of a typical Splunk installation’s data. And yet syslog itself is simply the protocol over which a number of devices’ log data flows. Be sure to *not* use syslog as the sourcetype, but rather that of the originating data. Use appropriate syslog tools to pre-filter data. They’re good at it, they’re free, they’re well-documented, and they integrate well with Splunk.
• #28: HEC is the newest, and most scalable, way to collect syslog-based data.
• #29: In addition to live, .conf, docs, answers, meetups etc etc