Talk @ GrafanaCon 2016
- 2. Utkarsh Bhatnagar • Senior Software Engineer @ Sony Interactive Entertainment (PlayStation). • An active contributor to Grafana. • Project initiator for wizzy – a user friendly CLI tool for GRAFANA GitHub - https://github.com/utkarshcmu Email – utkarsh.bhatnagar@sony.com
- 6. Requirements: • 50,000 unique metrics from one source • Data points every minute • Roughly about 72 million data points per day • Data retention 60 days • User friendly UI with possible customization
- 9. Mission accomplished! 1 metrics source 50,000 unique metrics 72 million data points per day
- 10. Team 1 Requirements: • 100,000 unique metrics • About 200 million data points per day Team 2 Requirements: • 400,000 unique metrics • About 600 million data points per day Team 3 Requirements: • 500,000 unique metrics • About 2 billion data points per day Team 4 Requirements: • 800,000 unique metrics • About 5 billion data points per day And more………
- 12. Should he continue with Graphite? Should he ask to reduce metrics or datapoints? How to dynamically scale Graphite? Does Grafana support other datasources? OpenTSDB / InfluxDB / KairosDB / Prometheus? Support scaling Infrastructure to support variable load of metrics? Challenges: • Multiple teams • Millions of unique metrics • Above 10 billion data points a day • Process 3 million logs every minute and generate metrics • Reprocessing of metrics and logs if needed • Provide real time monitoring for all of the above using GRAFANA!
- 13. Team 1 Requirements: • 100,000 unique metrics • About 200 million data points per day Team 2 Requirements: • 500,000 unique metrics • About 2 billion data points per day Team 3 Requirements: • 3 million logs a minute • Generate metrics in real time And more……… Team 1 Requirements: • 100,000 unique metrics • About 200 million data points per day
- 14. POC works for: 1 metrics source 50,000 unique metrics 72 million data points per day Team 1 requirements: 1 metrics source 100,000 unique metrics 200 million data points per day
- 16. Team 1 Requirements: • 100,000 unique metrics • About 200 million data points per day Team 2 Requirements: • 500,000 unique metrics • About 2 billion data points per day Team 3 Requirements: • 3 million logs a minute • Generate metrics in real time And more……… Team 2 Requirements: • 500,000 unique metrics • About 2 billion data points per day
- 17. Team 2 requirements: 1 metrics source 500,000 unique metrics 2 billion data points per day
- 19. Team 2 requirements: 1 metrics source 500,000 unique metrics 2 billion data points per day
- 21. Clustering Graphite CARBON RELAY CARBON CACHE + WHISPER + GRAPHITE WEB CARBON CACHE + WHISPER + GRAPHITE WEB CARBON CACHE + WHISPER + GRAPHITE WEB . . . GRAPHITE WEB GRAPHITE WEB LOAD BALANCER
- 22. Team 2 requirements: 1 metrics source 500,000 unique metrics 2 billion data points per day CR G G G. . . GW GW LB
- 24. Team 2 requirements: 1 metrics source 500,000 unique metrics 2 billion data points per day CR G G G. . . GW GW LB
- 26. Team 2 requirements: 1 metrics source 500,000 unique metrics 2 billion data points per day CR G G G. . . GW GW LB
- 28. Team 1 Requirements: • 100,000 unique metrics • About 200 million data points per day Team 2 Requirements: • 500,000 unique metrics • About 2 billion data points per day Team 3 Requirements: • 3 million logs a minute • Generate metrics in real time And more……… Team 3 Requirements: • 3 million logs a minute • Generate metrics in real time
- 30. Team 3 requirements: Over 5000 log sources 3 million logs per minute
- 33. Alerting
- 34. Application Metrics - Apps using a stats library written by Alexander Filipchik (Principal Engineer @ PlayStation) Custom metrics - From other sources
- 35. • More than 4 million unique metrics supported - creation and deletion happens all the time • More than 11 billion data points written per day - across all TSDBs • Processing about 40 billion events per day - logs and metrics events in near real time (within 30 seconds) • More than 3000 requests per minute to Grafana dashboards - around 7000 in during outages
- 40. (Subject to effort and time)
- 42. Alerting
- 46. Sep, 21st 2015 Nov, 17th 2016 Grafana Pull Requests: • Total - 144 • Accepted – 128 • Declined – 14 • Open - 2
- 48. • Prod , Stage and Dev installations of Grafana • Move/Copy rows, panels from one dashboard to another • Version control your dashboards • Manage Grafana entities like orgs, etc via CLI
- 50. Utkarsh Bhatnagar • Senior Software Engineer @ Sony Interactive Entertainment (PlayStation). • An active contributor to Grafana. • Project initiator for wizzy – a user friendly CLI tool for GRAFANA GitHub - https://github.com/utkarshcmu Email – utkarsh.bhatnagar@sony.com