Monitoring using Open source technologies
- 2. Utkarsh Bhatnagar • Senior Software Engineer @ Sony Interactive Entertainment (PlayStation). • An active contributor to Grafana. • Project initiator for wizzy – a user friendly CLI tool for GRAFANA GitHub - https://github.com/utkarshcmu Email – utkarsh.cmu@gmail.com GrafanaCon 2016 Speaker - https://www.youtube.com/watch?v=llRhdvV25rg
- 6. Requirements: • 50,000 unique metrics from one source • Data points every minute • Roughly about 72 million data points per day • Data retention 60 days • User friendly UI with possible customization
- 10. Mission accomplished! 1 metrics source 50,000 unique metrics 72 million data points per day
- 11. Team 1 Requirements: • 100,000 unique metrics • About 200 million data points per day Team 2 Requirements: • 400,000 unique metrics • About 600 million data points per day Team 3 Requirements: • 500,000 unique metrics • About 2 billion data points per day Team 4 Requirements: • 800,000 unique metrics • About 5 billion data points per day And more………
- 13. Should he continue with Graphite? Should he ask to reduce metrics or datapoints? How to dynamically scale Graphite? Does Grafana support other datasources? OpenTSDB / InfluxDB / KairosDB / Prometheus? Support scaling Infrastructure to support variable load of metrics? Challenges: • Multiple teams • Millions of unique metrics • Above 10 billion data points a day • Process 3 million logs every minute and generate metrics • Reprocessing of metrics and logs if needed • Provide real time monitoring for all of the above using GRAFANA!
- 14. Team 1 Requirements: • 100,000 unique metrics • About 200 million data points per day Team 2 Requirements: • 500,000 unique metrics • About 2 billion data points per day Team 3 Requirements: • 3 million logs a minute • Generate metrics in real time And more……… Team 1 Requirements: • 100,000 unique metrics • About 200 million data points per day
- 15. POC works for: 1 metrics source 50,000 unique metrics 72 million data points per day Team 1 requirements: 1 metrics source 100,000 unique metrics 200 million data points per day
- 17. Team 1 Requirements: • 100,000 unique metrics • About 200 million data points per day Team 2 Requirements: • 500,000 unique metrics • About 2 billion data points per day Team 3 Requirements: • 3 million logs a minute • Generate metrics in real time And more……… Team 2 Requirements: • 500,000 unique metrics • About 2 billion data points per day
- 18. Team 2 requirements: 1 metrics source 500,000 unique metrics 2 billion data points per day
- 20. Team 2 requirements: 1 metrics source 500,000 unique metrics 2 billion data points per day
- 22. Clustering Graphite CARBON RELAY CARBON CACHE + WHISPER + GRAPHITE WEB CARBON CACHE + WHISPER + GRAPHITE WEB CARBON CACHE + WHISPER + GRAPHITE WEB . . . GRAPHITE WEB GRAPHITE WEB LOAD BALANCER
- 23. Team 2 requirements: 1 metrics source 500,000 unique metrics 2 billion data points per day CR G G G. . . GW GW LB
- 25. Team 2 requirements: 1 metrics source 500,000 unique metrics 2 billion data points per day CR G G G. . . GW GW LB
- 27. Team 2 requirements: 1 metrics source 500,000 unique metrics 2 billion data points per day CR G G G. . . GW GW LB
- 29. Team 1 Requirements: • 100,000 unique metrics • About 200 million data points per day Team 2 Requirements: • 500,000 unique metrics • About 2 billion data points per day Team 3 Requirements: • 3 million logs a minute • Generate metrics in real time And more……… Team 3 Requirements: • 3 million logs a minute • Generate metrics in real time
- 31. Team 3 requirements: Over 5000 log sources 3 million logs per minute
- 34. Alerting
- 35. Graphite Stats - Apps using a stats library written by Alexander Filipchik Custom metrics - From other sources
- 40. (Subject to effort and time)
- 42. Alerting
- 43. • More than 3 million unique metrics supported - creation and deletion happens all the time • More than 11 billion data points written per day - across all TSDBs • Processing about 40 billion events per day - logs and metrics events in near real time (within 30 seconds) • More than 3000 requests per minute to Grafana dashboards - around 7000 requests in during outages
- 44. Alerting
- 52. • Move • Copy • Extract • Insert • Remove • Rows • Panels • Template varia • Dashboard tag
- 53. • Dashboards • Datasources • Orgs • Rows • Panels • Template variables • Dashboard tags Version Control
- 54. • Production • Staging • Testing • Development Grafana in multiple environments
- 55. • Last 24 hours • By a dashboard tag • Customized dashboa Generate GIFs of important dashbo
- 56. Generate GIFs of important dashbo
- 57. • Upload/Store/Download dashboards to/in/from AWS S3 respectively. • Search/Download community dashboards from Grafana.net External features
- 60. Utkarsh Bhatnagar • Senior Software Engineer @ Sony Interactive Entertainment (PlayStation). • An active contributor to Grafana. • Project initiator for wizzy – a user friendly CLI tool for GRAFANA GitHub - https://github.com/utkarshcmu Email – utkarsh.cmu@gmail.com GrafanaCon 2016 Speaker - https://www.youtube.com/watch?v=llRhdvV25rg