SplunkLive! Customer Presentation – Ticketmaster
6 likes2,051 views
Ticketmaster has been using Splunk for around 5 years to gain visibility into their operations and customer usage. They ingest over 1.6TB of log data per day from their various systems like web servers, databases, and applications. Splunk is used for monitoring systems and applications, troubleshooting issues, and providing analytics on customer search patterns and event demand. It has helped Ticketmaster improve customer service, protect against ticket brokers, and increase revenue by better understanding concert demand.
SplunkLive! Customer Presentation – Ticketmaster
- 1. Copyright © 2014 Splunk Inc. Splunk @ TicketMaster
- 2. 2 Shakeel Sorathia VP Systems Engineering
- 3. 3 Ticketmaster Overview • We are part of Live Nation Entertainment • You may have heard of us… • We sell a few tickets • 7 data centers worldwide • 20,000+ OS images (VMs and bare metal) • Transactions > $16B worldwide • Onsales > $1M/minute • 255M+ user accounts
- 4. 4 About Me • Been with Ticketmaster for almost 10 years • Responsible for the infrastructure and site operations • My name is pretty well known (but for different reasons) • Splunk, because ninjas are too busy!
- 5. 5 How We Got Started Adopted Splunk for ~5 years – Before Splunk: Lots of ‘grep’ping – Lack of understanding of what was going on with web properties – Very time consuming to try and troubleshoot issues Over the last 2 years, numerous new software launches demanding – Visibility into usage, performance, availability for engineering and development – View of blocks vs open reservations, comparison of popularity of events, planning and predicting for high-volume events
- 6. 6 Splunk at Ticketmaster Today • Keep applications and operations running – 900 users organization-wide Monitoring ticket process for failures Monitor Splunk NOC dashboards for capacity problems, availability issues, forensics Transaction tracing, counts, durations, failed transactions • Provide analytics to product managers/business owners What is the response to new events? High enough to create more similar events at different venues? Are we experiencing too much block activity – potential illegal resale? What are people searching for the most?
- 7. 7 Splunk at Ticketmaster 1.6TB/day 16 indexers ~2000 forwarders 80 indexes with many data types: .Net, Apache, JBOSS, weblogic, Java, Perl, python, C, C++ application logs 100s of applications across 17 different ticketing systems Apps used: Exchange, AD, NetApp E-series, SoS Offload search load to Splunk search heads Auto load-balanced forwarding to Splunk indexers Send data from thousands of servers using any combination of Splunk forwarders
- 8. 8 Developer Guidelines • Ticketmaster does 120 deployments/month • Splunk used to correlate production issues with new release deployments • Developers given logging guidelines initially: - Key value pairs in logs helps faster on boarding, greater visibility when the code is in production - Shorten variable names for ease of use (moving towards this) - Selling to developers is critical for fast problem solving
- 9. 9 Splunk Powering Our Operations Problem Customer service call from ticket buyers not receiving their tickets via mail results in hours of grepping email logs to verify claims Reduce Time to Resolution By putting mail logs in Splunk and creating an app for customer service, we could fix email issues quickly! Problem Correlated monitoring of the application stack was not possible; monitoring was siloed across environments, it took minutes (an eternity) to discover errors End-to-End Insights “The EOS Splunk dashboard represented a quantum leap in the information available to us during ‘on sale’ with real-time monitoring and real-time analysis of what is happening on the website and our hosts systems” an internal customer
- 10. 10 Splunk Powering Our Operations And The Business Problem Ticket broker automation was blocking access to open inventory for ticket fans Secure Inventory By using Splunk we protected inventory by responding quickly to evolving broker automation tactics Problem Inability to track concert demand in real time resulted in risks associated with adding new shows: either undersold new shows or didn't add shows and left money on the table Increase Revenue By having up-to-the-minute information ticket demand we can respond quickly by adding new shows based on actual demand metrics, resulting in more revenue!
- 11. 11 Screenshots Screenshot here
- 12. 12 Best Practice Recommendations • Think through log formatting as you through your deployments – better logs accelerate success • Define who owns which data – will help clarify what the use is • Plan to scale – its viral, everyone jumps on it, plan for success • Infrastructure • Semantic logging (use short variable names!)
- 13. 13 What’s Next • Greater use of Splunk across the enterprise • Making Splunk accessible via mobile devices • Splunk access to customer support for self-service resolution • Democratize data: more self-service access, more correlation use cases
- 14. Thank You