Horacio Gonzalez - Monitoring OVH - Codemotion Amsterdam 2019
0 likes457 views
The document discusses OVH's metrics platform, highlighting its need for better decision-making through metrics in software development. It introduces the challenges of scalability and rich query capabilities faced with previous solutions like OpenTSDB and proposes Warp 10 as a more efficient alternative for handling time series data. The presentation also outlines various use cases for the metrics platform, including monitoring and billing efficiencies.
Horacio Gonzalez - Monitoring OVH - Codemotion Amsterdam 2019
- 1. @LostInBrittany Amsterdam | April 2 - 3, 2019 Monitoring OVH: 300k servers, 28 DCs... and one Metrics platform Horacio Gonzalez @LostInBrittany
- 2. @LostInBrittany Who are we? Introducing myself and introducing OVH
- 3. @LostInBrittany Horacio Gonzalez @LostInBrittany Spaniard lost in Brittany, developer, dreamer and all-around geek Flutter
- 4. @LostInBrittany OVH : Key Figures 1.3M Customers worldwide in 138Countries 1.5 Billions euros investment over five years 28 Datacenters (growing) 350k Dedicated Servers 200k Private cloud VMs running 650k Public cloud Instances created in a month 20TB bandwidth capacity 35 Points of presence 4TB Anti DDoS capacity Hosting capacity : 1.3M Physical Servers + 2 500 Employees in 19 countries 18Years of Innovation
- 5. @LostInBrittany OVH: A Global Leader on Cloud Own 15 Tbps Netwok with 35 PoPs > 1.3M Customers in 138 Countries Hosting capacity : 1.3M Physical Servers 360k Servers already deployed 2018 27 Datacenters 2020 50 Datacenters 1 Dedicated IaaS Europe 200k Private cloud VMs running
- 6. @LostInBrittany Ranking & Recognition 1st European Cloud Provider* 1st Hosting provider in Europe 1st Provider Microsoft Exchange Certified vCloud Datacenter Certified Kubernetes platform (CNCF) Vmware Global Service Provider 2013-2016 Veeam Best Cloud Partner of the year (2018) * Netcraft 2017 -
- 7. @LostInBrittany OVH: Our solutions Cloud Web Hosting ▪ Dedicated Server ▪ Data Storage ▪ Network and Security ▪ Licences Mobile Hosting Telecom VoIP SMS/Fax Virtual desktop Cloud HubiC Over theBox Containers Compute Database Object Storage Securities Messaging VPS Public Cloud Private Cloud Serveur dédié Cloud Desktop Hybrid Cloud Domain names Email CDN Web hosting MS Office MS solutions
- 8. @LostInBrittany Once upon a time... Because I love telling tales
- 9. @LostInBrittany This talk is about a tale... A true one nevertheless
- 10. @LostInBrittany And as in most tales It begins with a mission
- 11. @LostInBrittany And a band of heroes Engulfed into the adventure
- 12. @LostInBrittany They fight against mishaps And all kind of foes
- 13. @LostInBrittany They build mighty fortresses Pushing the limits of possible
- 14. @LostInBrittany And defend them day after day Against all odds
- 15. @LostInBrittany But we don't know yet the end Because this tale isn't finished yet
- 16. @LostInBrittany It begins with a mission Build a metrics platform for OVH
- 17. @LostInBrittany Why do we need metrics? To make better decisions by using numbers
- 18. @LostInBrittany Why do we need metrics? We want our code to add value
- 19. @LostInBrittany Why do we need metrics? We need to make better decisions about our code
- 20. @LostInBrittany Why do we need metrics? Code adds value when it runs not when we write it
- 21. @LostInBrittany Why do we need metrics? We need to know what our code does when it runs
- 22. @LostInBrittany Why do we need metrics? We can’t do this unless we measure it
- 23. @LostInBrittany Why do we need metrics? We have a mental model of what our code does
- 24. @LostInBrittany Why do we need metrics? This representation can be wrong
- 25. @LostInBrittany Why do we need metrics? We can’t know until we measure it
- 26. @LostInBrittany “The app is slow.” - User Find the bottleneck ‘’‘’
- 27. @LostInBrittany “The app is slow.” - User “The page takes 500ms!” - Ops Find the bottleneck ‘’
- 28. @LostInBrittany SQL Query? Template Rendering? Session Storage? ? Find the bottleneck
- 29. @LostInBrittany We don't know ? Find the bottleneck
- 30. @LostInBrittany With observability: SQL Query………………………....53ms Template Rendering……….1ms Session Storage………......315ms = Find the bottleneck
- 31. @LostInBrittany With observability: SQL Query………………………....53ms Template Rendering……….1ms Session Storage………......315ms = Find the bottleneck
- 32. @LostInBrittany Why do we need metrics? We improve our mental model by measuring what our code does
- 33. @LostInBrittany Why do we need metrics? We use our mental model to decide what to do
- 34. @LostInBrittany Why do we need metrics? A better mental model makes us better at deciding what to do
- 35. @LostInBrittany Why do we need metrics? Better decisions makes us better at generating value
- 36. @LostInBrittany Why do we need metrics? Measuring make your App better
- 37. @LostInBrittany It began with a mission Build a metrics platform for OVH
- 38. @LostInBrittany A metrics platform for OVH For all OVH
- 39. @LostInBrittany Building OVH Metrics One Platform to unify them all, One Platform to find them, One Platform to bring them all and in the Metrics monitor them
- 40. @LostInBrittany What is OVH Metrics? Managed Cloud Platform for Time Series
- 41. @LostInBrittany OVH monitoring story We had lots of partial solutions...
- 42. @LostInBrittany OVH monitoring story One Platform to unify them all What should we build it on?
- 43. @LostInBrittany OVH monitoring story Including a really big
- 44. @LostInBrittany OpenTSDB drawbacks OpenTSDB RowKey Design !
- 45. @LostInBrittany OpenTSDB Rowkey design flaws ● .*regex.* => full table scans ● High cardinality issues (Query latencies) We needed something able to manage hundreds of millions time series OpenTSBD didn't scale for us
- 46. @LostInBrittany OpenTSDB other flaws ● Compaction (or append writes) ● /api/query : 1 endpoint per function? ● Asynchronous ● Unauthenticated ● ...
- 48. @LostInBrittany Metrics needs First need: To be massively scalable
- 49. @LostInBrittany Analytics is the key to success Fetching data is only the tip of the iceberg
- 50. @LostInBrittany Analysing metrics data To be scalable, analysis must be done in the database, not in user's computer
- 51. @LostInBrittany Metrics needs Second need: To have rich query capabilities
- 52. @LostInBrittany Enter Warp 10... Open-source Time series Database
- 53. @LostInBrittany More than a Time Series DB Warp 10 is a software platform that ● Ingests and stores time series ● Manipulates and analyzes time series
- 54. @LostInBrittany Manipulating Time Series with Warp 10 A true Time Series analysis toolbox ○ Hundreds of functions ○ Manipulation frameworks ○ Analysis workflow
- 55. @LostInBrittany Manipulating Time Series with Warp 10 A Time Series manipulation language WarpScript
- 56. @LostInBrittany Did you say scalability? From the smallest to the largest...
- 57. @LostInBrittany More Warp 10 goodness ● Secured & multi tenant ● In memory Index ● No cardinality issues ● Lockfree ingestion ● WarpScript Query Language ● Support more data types ● Synchronous (transactions) ● Better Performance ● Better Scalability ● Versatile (standalone, distributed)
- 58. @LostInBrittany OVH Observability Metrics Platform
- 60. @LostInBrittany Building an ecosystem From Warp 10 to OVH Metrics
- 61. @LostInBrittany Multi-protocol Why to choose? We need them all!
- 62. @LostInBrittany Open source monitoring tools
- 63. @LostInBrittany Open source monitoring tools
- 64. @LostInBrittany Open source monitoring tools
- 65. @LostInBrittany Open source monitoring tools
- 66. @LostInBrittany Open source monitoring tools
- 67. @LostInBrittany Open source monitoring tools
- 68. @LostInBrittany Open source monitoring tools Why choose? Let’s support all of them!
- 71. @LostInBrittany Metrics Live In-memory, high-performance Metrics instances
- 72. @LostInBrittany In-memory: Metrics live +120 million of writes/s
- 75. @LostInBrittany Monitoring is only the beginning OVH Metrics answer to many other use cases
- 76. @LostInBrittany • Billing ……………………………………………..…....(e.g. bill on monthly max consumption) • Monitoring …..…………………………...(APM, infrastructure,appliances,...) • IoT …………………………………………….………………....(Manage devices, operator integration, ...) • Geo Location ….....………………...(Manage localized fleets) Use cases families
- 77. @LostInBrittany • DC Temperature/Elec/Cooling map • Pay as you go billing (PCI/IPLB) • GSCAN • Monitoring • ML Model scoring (Anti-Fraude) • Pattern Detection for medical applications Use cases
- 78. @LostInBrittany SREing Metrics With a great power comes a great responsibility
- 79. @LostInBrittany 432 000 000 000 datapoints / day Metrics' own metrics
- 80. @LostInBrittany 10 Tb / day Metrics' own metrics
- 81. @LostInBrittany 5 000 000 dp/s Metrics' own metrics
- 82. @LostInBrittany 500 000 000 series Metrics' own metrics
- 83. @LostInBrittany BHS: ● 30 nodes ● 400 TB ● 120 Mbps GRA: ● 150 nodes ● 2 PB ● 1.1 Gbps Our clusters size
- 85. @LostInBrittany Detecting errors 85 Before it's too late
- 86. @LostInBrittany Extract errors from logs
- 87. @LostInBrittany Forward logs and extract metrics! Tailor
- 90. @LostInBrittany JVM GC The good, the bad and the ugly
- 93. @LostInBrittany #java #jdk11 #zgc … and the ugly
- 95. @LostInBrittany Number of open regions
- 97. @LostInBrittany Number of read and write requests
- 101. @LostInBrittany Merging all data sources
- 108. @LostInBrittany Move region to another region server
- 109. @LostInBrittany Drain regions of the region server
- 110. @LostInBrittany Managing multiple hardware profiles