Logging and machine data at Scale. re:Invent 2017
Download as PPT, PDF1 like719 views
The document discusses the monetization of by-products in various industries, emphasizing the importance of user experience over system health in observability. It highlights the integration of logging, tracing, and monitoring in real-time data analysis across sectors such as healthcare, e-commerce, and security. Additionally, it mentions the growing significance of cloud-native tools and compliance regulations impacting data usage.
Logging and machine data at Scale. re:Invent 2017
- 1. 1 Logging and machine data at Scale James Governor @monkchips co-founder.
- 2. 2 “The lumber industry sells what used to be waste — sawdust, chips, and shredded wood — for a pretty profit. Today you’ll find these by-products in synthetic fireplace logs, concrete, ice strengtheners, mulch, particle board, fuel, livestock and pet bedding, winter road traction, weed killing and more” – Jason Fried, CEO Basecamp
- 3. 3
- 4. 4
- 5. 5
- 6. 6
- 7. 7
- 8. 8
- 9. 9 Zero Users Care What The System Health is All Users Care What Their Experience Is Nines Don’t Matter if Users Aren’t Happy Charity Majors, Honeycomb.io
- 10. 10 Observability bridges tracing, logging, and monitoring Logs are no longer about history Real time, streams, and adhoc queries
- 12. 12 Use Cases Compliance DevOps, IT Service management/root cause analysis ecommerce, digital marketing optimisation Fraud detection IoT and integration with data from physical assets Healthcare Real time security
- 13. 13 Always Compliance Anti-money laundering PCI GDPR is coming – fines potentially 4% of global turnover
- 14. 14 Splunk Dubai Airport instruments literally everything – Even hand washing in the bathroom from electronic taps and soap Rackspace ingests 3TB of data every day – Security monitoring and management Tactical Assault Light Operator Suit (TALOS) – Monitoring including vital signs
- 15. 15 ELK as a disruptive entrant Started as text search, now as a default for many orgs Sprint – 3bn events per day – Digital Transformation 4.0 Project – real-time data from Retail Management and Store Ops – 200 dashboards represent events from logs, databases, emails, syslogs, test messages, and internal and vendor application APIs. Uber, Facebook, Netflix
- 16. 16
- 17. 17
- 18. 18 Edge Filtering IoT Sampling and polling
- 19. 19 Tracing at Facebook 2014 Mystery Machine took 2 hours to compute a model from 1.3M traces 2017 Canopy generates and processes 1.3 billion traces per day (spanning end-user devices, web servers, and backend services, backs 129 performance datasets from high-level end-to-end metrics to specific custom use cases) Source https://blog.acolyer.org/2017/11/22/canopy-an-end-to-end-performance-tracing-and-analysis-system/
- 20. 20 Roll your own kind of sucks
- 21. 21 Using machine data for business process optimization Data isn’t on the balance sheet But data as product is Data optimized services and customer experiences Data culture Observability is the new hotness
- 22. 22 Cloud Native Tools Grafana – visualization graph and dashboards, Prometheus – monitoring and alerting, time series Fluentd – data collector and log aggregation Graphite – time series and graphs
Editor's Notes
- #7: In society we put a huge premium on convenience. Humans are fairly lazy when it comes to it. why did uber win, and why do so many Silicon Valley companies emerge that focus on convenience? Because convenience wins. Whatever you think of uber, it wins because of the way is packaged and serviced. It works and works well. I like black cabs but I hate messing around with receipts and cash.