Taking Splunk to the Next Level - Architecture Breakout Session
1 like983 views
This document provides an agenda for scaling a Splunk deployment beyond initial use cases. It discusses growing use cases and data volume over time. As Splunk becomes mission critical, the document recommends implementing high availability through indexer and search head clustering. It also suggests using a distributed management console and centralized configuration management. Finally, the document briefly discusses Splunk Cloud and hybrid deployments as options to scale without waiting for additional on-premise hardware.
Taking Splunk to the Next Level - Architecture Breakout Session
- 1. Copyright © 2015 Splunk Inc. Splunk @ Level++ Stefan Sievert Client Architect PNW Majors ssievert@splunk.com Calgary, May 14th 2015
- 2. 2 Splunk at the Next Level Time to move beyond initial Splunk environment • More use cases – how to tackle? • More data – how do we scale? • Splunk is mission critical == HA • Global deployments • Splunk user experience Screenshot here
- 3. 3 Agenda Use cases Business Cases Simple Scaling Indexer Clustering (+Multi-site Clustering, Search Affinity) Search Head Clustering Distributed Management Console Centralized Configuration Management Splunk Cloud & Hybrid Deployments Q&A
- 4. 4 Growing your Splunk Deployment Many customers start with a single use case… • Ex: Monitor the web servers • Help ensure up-time & response times • Track usage, errors • Provides business value
- 5. 5 Growing your Splunk Deployment Value statement for each overall service Your services exist in a larger context than just one app, or one tier. What is the value of the service as a whole? What are CIO commitments for the service? • The company’s web store is one of the most critical parts of the business. • Performance of the overall environment must be maintained at all times. • Failures in any portion of the web store must be quickly identified, send notification to the appropriate parties. • Dependencies on external processes must be monitored as well.
- 6. 6 Growing your Splunk Deployment The larger context • Failure in one system cascades • Map dependencies, estimate costs • Use Splunk to track all dependencies. • What happens when it is down? Dependencies often include: • Networking dependencies • Shared storage • Databases, middleware, custom apps • Virtualization layer Screenshot here
- 7. 7 Scaling Multiple factors Indexer: IOPs, daily rate Storage: Usage & retention Search Head usage
- 8. 8 Scaling - Indexers Sizing for index performance Indexers are usually storage-bound Indexers: 150 to 250 GB per day each. (With suitable storage) Ref HW: 12 cores (2 GHz+), 12 GB RAM, 800+ IOPs Optimal HW (normal disk): 16 CPU cores, 48 GB RAM Optimal HW (SSD): 24 CPU cores, 132 GB RAM Questions?
- 9. 9 SSD Advantage http://blogs.splunk.com/2012/05/10/quantifying-the-benefits-of- splunk-with-ssds/ • Low cost random seeks • Writes are not that much faster – no great improvement with Indexing • Significant improvements with Sparse/needle-haystack searches • Dense searches become CPU bound • Searches run faster allowing for more completed searches/min
- 10. 10 Scaling - Storage Simple storage to complex Raw data rate net compression of ~ 50% on disk. Simple: rate * compression * retention 200 GB / day * 50% * 100 days = 10TB Consider cold storage on NAS – Changes storage story. – Retention on fast, retention on slow Clustering – Changes storage story
- 11. 11 Scaling - Storage Sizing Calculator: http://splunk-sizing.appspot.com/
- 12. 12 Scaling - Storage RAID + SSD deep dive • For spinning disks, Splunk recommends RAID 1+0 with 1k IOPs • SSDs provide extremely high IOPs (45,000 +) • RAID 5 SSD arrays give great Splunk performance in most scenarios. Additional details: Splunk Docs, Capacity Planning Manual
- 13. 13 Forwarder Load Balancing Have UF balance across multiple indexers DNS round robin Multiple hosts in outputs LB not needed! Geography-based routing
- 14. 14 Indexer Clustering High-Availability, Out of the Box Splunk indexer clustering Active-Active= better performance Specific terms: – Master Node – Peer Node – Search Factor – Replication Factor Additional details: Splunk Docs, Distributed Deployment Manual
- 15. 15 Multi-site Clustering Search Affinity by location “Search locally”, “Store Globally” DR scenarios
- 16. 16 Scaling the Search Heads Splunk Search is critical, too! Splunk Search high availability needs Scale to handle # of concurrent queries
- 17. 17 SHP vs SHC SHC • SHP • Available since v4.2 • Sharing configurations through NFS • Single point of failure • Performance issues • No NFS • Replication using local storage • Commodity hardware NFS
- 19. 19 Search Head Clustering Use “Captain” for Master to avoid confusion with Index-Clustering Minimum 3 nodes required. Odd is always preferred. Cluster takes certain key decisions based on *majority* (consensus) In multi-site setup have more nodes in main datacenter
- 20. 20 Distributed Management Console Manage Splunk 6.2 environments Replaces Deployment Monitor App Incorporates SOS app prior to 6.2
- 21. 21 Deployment Server Central management of Splunk Forwarders Deployment Server manages Apps, Configs Select one or more classes for each host Class defines apps & configs Works by phone-home Notes: DS does not push forwarder binaries Use Cluster Master to manage indexers in cluster, not DS
- 22. 22 Cloud & Hybrid Scale without waiting for hardware
- 23. 23 2 www.splunk.com/apptitude July 20th, 2015 Submission deadline
- 24. 24 24 The 6th Annual Splunk Worldwide Users’ Conference • September 21-24, 2015 • The MGM Grand Hotel, Las Vegas • 4000 IT & Business Professionals • 2 Keynote Sessions • 3 days of technical content – 165+ sessions • 3 days of Splunk University – Sept 19-21, 2015 – Get Splunk Certified for FREE! – Get CPE credits for CISSP, CAP, SSCP, etc. – Save thousands on Splunk education! • 80 Customer Speakers • 80 Splunk Speakers • 35+ Apps in Splunk Apps Showcase • 65 Technology Partners • Ask The Experts and Security Experts, Birds of a Feather, Chalk Talks and a new & improved Partner Pavilion! • Register at conf.splunk.com
- 25. 25 We Want to Hear your Feedback! After the Breakout Sessions conclude Text Splunk to 878787 And be entered for a chance to win a $100 AMEX gift card!
- 26. Thank You
Editor's Notes
- #24: ----- Meeting Notes (4/22/15 10:47) ----- Splunk Apptitude is live and open. You've got 90 days. To win more than $150,000 in cash and prizes. Last day to submit is July 20th, 2015. We'll announce the winners at Black Hat in August. Good luck!
- #25: 2 inspired Keynotes – General Session and Security Keynote 150+ Breakout sessions addressing all areas and levels of Operational Intelligence – IT, Business Analytics, Mobile, Cloud, IoT, Security…and MORE! Join the 50%+ of Fortune 100 companies who attended .conf2014 to get hands on with Splunk. You’ll be surrounded by thousands of other like-minded individuals who are ready to share exciting and cutting edge use cases and best practices. You can also deep dive on all things Splunk products together with your favorite Splunkers. Head back to your company with both practical and inspired new uses for Splunk, ready to unlock the unimaginable power of your data! Arrive in Vegas a Splunk user, leave Vegas a Splunk Ninja!