Hunk - Unlocking the Power of Big Data

Editor's Notes

• #3: But listening to your machine data isn’t as easy as it sounds. Machine Data is different: It is voluminous unstructured time series data with no predefined schema It is generated by all IT systems– from servers and applications, to RFIDs and wire data. It is non-standard data and characterized by many unpredictable and changing formats   Because of this, machine data cannot be managed using traditional approaches. Traditional approaches require you to transform your data and force fit it into a brittle schema – They aren’t designed to handle the inconsistent machine data formats Traditional approaches are designed with specific use cases and queries in mind – they limit the problems that you can solve Traditional approaches rely on siloed tools that are designed for structured data approaches and legacy computing environments – They are inherently limited in their ability to scale   To listen to your machine data, you need a solution with no limits: No limits on the formats of data No limits on where you can collect the data from No limits on the questions that you can ask and the use cases you can solve. And no limits on scale.   You need a solution that can keep up with Machine Data.  
• #4: Since then, Splunk has invested significantly to expand from a search tool to a mission-critical platform. The platform includes hundreds of data types and can scale to massive volumes Today, it’s more than Splunk Enterprise, we’ve added Splunk Cloud, Hunk, Splunk MINT for mobile intelligence; and have more than 600 Apps. Machine data is more than logs! It’s wire data, mainframe data, mobile device data, sensor data, metrics Your use cases have evolved well beyond troubleshooting so we’re investing in solutions that leverage the power of Splunk Enterprise to provide you with packaged views into your data for faster, deeper insights. Our most well-known solution is Splunk Enterprise Security and if you aren’t using it yet, we encourage you to find out why it’s turning the traditional SIEM market upside down.
• #6: How has big data evolved over time. For a long time, ‘big data’ was was simply a large database. The database industry – in order to handle large data – moved to smaller databases, but many of them. Horizontal partitioning (Also known as Sharding) is a database design principle whereby rows of a database table are held separately (For example, A -> D in one database E -> H in a second database, etc ..) Hadoop was introduced by Google and was adapted as the de-facto big data system. Hadoop is an open source project from Apache that has evolved rapidly into a major technology movement. It has emerged as a popular way to handle massive amounts of data, including structured and complex unstructured data. Its popularity is due in part to its ability to store and process large amounts of data effectively across clusters of commodity hardware. Apache Hadoop is not actually a single product but instead a collection of several components. For the most part, Hadoop is a batch oriented system. ** Teradata Aster Data & SQL on Hadoop are SQL interface systems that can talk to Hadoop ** Cassandra & HBase are NoSQL databases that can process data using a Key / Value in real-time. Splunk = Temporal, Unstructured, Heterogeneous, real-time analytics platform.
• #7: Splunk allows you divide up the work of search and indexing across as many servers as you need to achieve the performance and scale you require. Using work dividing techniques such as MapReduce, Splunk can take a single search and query as many indexers as you need to complete the job, allowing you to use inexpensive commodity hardware in massively parallel clusters. For example, if you had 1 million events to search, one Indexer can easily complete that search. But it will take a little time – let’s say 30 seconds. However, if the same million events was spread across 10 indexers, the same search would complete in 3 seconds. How fast and how large you want your searches is yours to control by adding indexers as desired.
• #8: For the most part, you can use monitor to add nearly all your data sources from files and directories. However, you might want to use upload to add one-time inputs, such as an archive of historical data. You can enable Splunk to accept an input on any TCP or UDP port. Splunk consumes any data sent on these ports. Use this method for syslog (default port is UDP 514), or set up netcat and bind to a port. TCP is the protocol underlying Splunk's data distribution and is the recommended method for sending data from any remote machine to your Splunk server. Splunk can index remote data from syslog-ng or any other application that transmits via TCP. However, there are times when you want to use scripts to feed data to Splunk for indexing, or prepare data from a non-standard source so Splunk can properly parse events and extract fields. You can use shell scripts, python scripts, Windows batch files, PowerShell, or any other utility that can format and stream the data that you want Splunk to index. You can stream the data to Splunk or write the data from a script to a file. All data that comes into Splunk enters through the parsing pipeline as large chunks. During parsing, Splunk breaks these chunks into events which it hands off to the indexing pipeline, where final processing occurs. During both parsing and indexing, Splunk acts on the data, transforming it in various ways. Most of these processes are configurable, so you have the ability to adapt them to your needs. To kick off a real-time search in Splunk Web, use the time range menu to select a preset Real-time time range window, such as 30 seconds or 1 minute. You can also specify a sliding time range window to apply to your real-time search. This defines a real-time buffer. The Splunk Index is the repository for Splunk Enterprise data. Splunk Enterprise transforms incoming data into events, which it stores in indexes.
• #10:   Faster Recovery II -   If you look at the screen – 2 indexers on the left with green cylinders – searchable copies of the data, 2 indexers on the right – only raw data What happens when a peer goes down, master waits for hb timeout and marks the peer down. <Cl> Reassigns primaries to another peer. Then tries to enforce the replication policy, makes copies of the raw data and search files In 5.0, search files are generated on each peer from the raw data, In 6.0, the search files are copied over from a peer that already has them instead of regenerating. <Cl> These statistics are from our internal tests... Another point to note is generating search files from the rawdata is cpu intensive as compared to copying search files.  
• #12: Quick to set-up, scales to multiple concurrent databases Enrich machine data with structured data from relational databases Execute database queries directly from the Splunk user interface Browse and navigate database schemas and tables Combine machine data with structured data from relational databases
• #14: Quick to set-up, scales to multiple concurrent databases Enrich machine data with structured data from relational databases Execute database queries directly from the Splunk user interface Browse and navigate database schemas and tables Combine machine data with structured data from relational databases
• #15: Search execution: The Hunk Search head takes the list of content of directories in the virtual index. The search head filters directories & files based on the search & time range (partition pruning) The NameNode and JobTracker (MapReduce Resource Manager in YARN) read data from MapReduce framework and feed it to search process. The process computes File Splits, constructs and submits the MapReduce jobs. Hunk streams a few File Splits from HDFS and processes them in the Search Head to provider quick previews. The search head consumes and merges the MapReduce results (provide incremental previews) while the MapReduce jobs kick off. The data nodes run a copy of splunkd to process the the jobs and write them to a working directory in HDFS. Final results are stored in the Hunk search head. Hunk utilizes the Splunk Search Processing Language, the industry-leading method to enable interactive data exploration across large, diverse data sets. There is no requirement to "understand" data up front. For customers of Splunk Enterprise, reuse your Search Processing Language knowledge and skill set for data stored in Hadoop. Any commands whose output depends on the event input order would yield different results – this is because Splunk guarantees events to be delivered in descending time order. Hunk doesn’t. This is the reason why transaction and localize do not work. We can see the results from the intermediate Hadoop Map jobs getting steamed into the Splunk UI even before all the Map jobs are finished, and once all the Hadoop Maps are done processing the results, Splunk displays the full results. In essence, Splunk acts as the Hadoop Reduce phase and there is no need to use Hadoop for that phase.
• #16: Hunk starts the streaming and reporting modes concurrently. Streaming results show until the reporting results come in. Allows users to search interactively by pausing and refining queries. This is a major, unique advantage of Hunk compared to alternative approaches such as Hive or SQL on Hadoop which require fixed schema in an effort to speed up searches, while Hunk retains the combination of schema on the fly with results preview.
• #19: In this new feature, planned for release in the next Hunk release (version 6.2.1), archive buckets to Hadoop (the Hadoop Distributed File System, or HDFS) instead of freezing buckets or throwing data away. This significantly lowers the total cost of ownership (TCO) for Splunk Enterprise installations while giving security analysts, risk managers and marketers access to months or years of historical data integral for their job success. Store old data up to 1/10 cheaper in Hadoop cheap batch storage instead of SANs Optimize Splunk Enterprise search head performance for real-time monitoring, alerting and dashboarding with short-term historical context Hunk search, analyze and visualize months or years of historical data in Hadoop Run federated queries and dashboards across Splunk Enterprise and Hunk
• #20: Indexing