Scylla Summit 2018: Scaling your time series data with Newts
Download as PPTX, PDF1 like1,685 views
The document outlines Jesse White's presentation on scaling time series data using Newts, an open-source platform integrated with OpenNMS. It details the architecture and operational strategies employed to manage high-volume metrics with examples of metric aggregation and the performance of Newts versus Cassandra and ScyllaDB. Key operational considerations, such as estimating disk space and cluster management challenges, are also discussed.
Scylla Summit 2018: Scaling your time series data with Newts
- 1. SCALING YOUR TIME SERIES DATA WITH NEWTS Jesse White CTO, The OpenNMS Group Inc.
- 2. Presenter bio ▪ OpenNMS since 2012 ▪ Cassandra since 2014 ▪ ScyllaDB since 2017
- 3. OPENNMS
- 4. OPENNMS ▪ World's First ▪ Enterprise Grade ▪ Network management ▪ Application platform ▪ Open Source
- 6. TIME SERIES
- 7. RRDTOOL
- 8. TIME SERIES STRATEGIES ▪ rrdtool via JNI (1.0 - 2002) ▪ JRobin (1.1.4 – 2004) ▪ jrrd2 (17.0.0 - 2015) ▪ Newts (17.0.0 - 2015) ▪ Kafka (24.0.0 - 2018)
- 10. NEWTS
- 12. THE MODEL Resources are used to group related metrics. Metrics are used to identify specific values. Samples represent the value of a resource metric at some instance in time.
- 13. MODEL EXAMPLE Resource ID: 1:opennms-jvm:org_opennms_newts_name_repository_insert_time Metrics: NewtsInsert50, NewtsInsert75, NewtsInsert95, NewtsInsert98, …
- 14. SAMPLES
- 15. LATE AGGREGATION The target range is split into intervals, where we take weighted averages of the corresponding samples. The heartbeat defines up to how long a sample is considered to be valid.
- 16. TERMS
- 17. SEARCH env:uat AND (dc:dc1 OR dc:dc2)
- 18. NEWTS + OPENNMS
- 19. GRAPHING
- 20. SCALE UP AND OUT ▪ 1004234.38 samples/s ▪ Cassandra: • 5 * i3.4xlarge ▪ Load: • 3 * m5.4xlarge
- 21. ScyllaDB ▪ Cassandra (3.11) • 1004234.38 samples/s ▪ ScyllaDB (2.3.1) • 1823485.12 samples/s • 1.8x improvement on write heavy workload
- 22. OPERATIONAL ISSUES ▪ Estimating disk space usage ▪ Cluster management ▪ Data mangling • Deleting a point • Renaming a series
- 23. NEWTS.IO
- 24. Thank You Any Questions ? Please stay in touch jesse@opennms.ca @jesse_white_