Taking Splunk to the Next Level - Architecture Breakout Session

Editor's Notes

• #8: By allowing Splunk Enterprise to be split into multiple roles, any portion of Splunk can be scaled as needed. Customers are using Splunk to index hundreds of TB/s a day and search over petabytes of data. Splunk can take a single search and query as many indexers as are needed to complete the job, allowing you to use inexpensive commodity hardware in massively parallel clusters. Besides achieve massive scale, splitting the roles enabled user to meet location and data segmentation requirements.
• #9: Searches can be distributed from a single search head to any number of indexers. These indexers can all be local for massive parallelization for Big Data problems, or spread across a global enterprise to help you keep data wherever makes the most sense for your network, availability, and security requirements. Splunk Enterprise can be deployed on premise, in the cloud, or a combination of both. There is also an Amazon Machine Image available or if you don’t want to host or administer Splunk, it can be managed as a service by our experts using “Splunk Cloud”.
• #10: These are multiple logical roles, a Splunk instance can be one or more of the roles. The search head is what most users interact with. It is the webserver and app interpreting engine that provides the primary, web-based user interface. Since most of the data interpretation happens as-needed at search time, the role of the search head is to translate user and app requests into actionable searches for it’s indexer(s) and display the results. The Splunk web UI is highly customizable, either through our own view and app system, or by embedding Splunk searches in your own web apps or our API. Additional search heads can be deployed to scale with user or search load. The core of the Splunk infrastructure is indexing. An indexer does two things – it accepts and processes new data, adding it to the index and compressing it on disk. The indexer also services search requests, looking through the data it has via it’s indices and returning the appropriate results to the searcher over a secure compressed communication channel. Indexers scale out almost limitlessly and with almost no degradation in overall performance, allowing Splunk to scale from single-instance small deployments to truly massive Big Data challenges. The Splunk forwarder is an optional component that can be installed to forward data from servers, desktops, mainframes, and even ARM based devices. There are two types of forwarders; the full Splunk distribution or a dedicated “Universal Forwarder”. The full Splunk distribution can be configured to filter data before transmitting, execute scripts locally, or run SplunkWeb. This gives you several options depending on the footprint size your endpoints can tolerate. The universal forwarder is an ultra-lightweight agent designed to collect data in the smallest possible footprint. Both flavors of forwarder come with automatic load balancing, SSL encryption and data compression, and the ability to route data to multiple Splunk instances or third party systems. The Cluster Master coordinates which indexers have copies of which buckets to ensure we have met the proper number of replication and searchable copies of each bucket. All clustered Indexers check in with the Master to alert them of their status, and the status of each of their replicated indexes and buckets. It also manages the apps and configurations on clustered indexers. We will talk more about buckets later. The Deployment Server can be used to manage your Splunk forwarders, for centrally managed data collection. More on this to come. Listed for completeness are the license master and DMC roles, these typically coexist with other roles such as the Deployment server.
• #22: This slide shows the way we used to calculate data storage requirements.
• #24: This app makes it far easier to size the environment’s storage requirements. And it includes clustering configurations, which we’ll talk about in a sec.
• #26: Splunk’s clustering technology allows you to choose how many raw copies and searchable copies of your data you would like to keep. It also allows you to chose which indexers you want to store the copies on. This capability allows servers or even datacenters to go down without losing the ability to access the data. In addition, the search affinity capability allows users to fetch data from the closest or fastest location where there is a copy of the data which can not only save the time it takes to do a search but bandwidth by eliminating the need to use the WAN when there is a local copy.
• #29: Default 3/2 cluster uses 3*.15 + 2*.35 = 115% of license usage for that redudancy Processing : a little more CPU and more network this is much better in current versions, the indexed data (tsidx, etc) is streamed to the replica peer, rather than forcing the peer to re-index.
• #30: With Search factor / rep factor variables in the mix - what had been simple without clustering now becomes more challenging. Demo sizing calculator if time allows. Hot/Warm vs. Cold, in different RAID configurations. Sample indexes.conf is generated too.
• #31: As discussed – default parameters require *more than* original log size
• #36: Uniform user experience among pooled search heads No single point of failure Search job failure aware Does not require external storage such as NFS
• #37: Note the Deployer on this image. Deployer virtualizes very well. The deployer pushes apps & configurations to the search head cluster members.
• #41: Putting it all together
• #42: What’s the best way to roll out some of these features? It depends on customer environment. But one common method is shown here.