The Data Mullet: From all SQL to No SQL back to Some SQL
1 like1,496 views
This document discusses Datadog's data architecture, which uses a combination of SQL and NoSQL databases. It initially used all SQL (Postgres) but found it did not scale well. It added Cassandra for durable storage and Redis for in-memory storage to improve performance and scalability. While Cassandra provided large-scale durable storage, it had issues with I/O latency on EC2. The document examines different database choices and how Datadog addressed scaling and latency issues through a hybrid "data mullet" approach using different databases for their strengths.
![Alexis Lê-Quôc @alq
Riak
• Prototyped out of spite for Cassandra 0.7[0123]
• We ♡ Erlang
• Great folks
• But Cassandra pain subsided, priorities shifted.
• git merge datadog/riak did not happen](https://image.slidesharecdn.com/thedatamullet-fromallsql-141023163242-conversion-gate01/85/The-Data-Mullet-From-all-SQL-to-No-SQL-back-to-Some-SQL-47-320.jpg)
The Data Mullet: From all SQL to No SQL back to Some SQL
- 1. The Data Mullet From All SQL to No SQL back to Some SQL Alexis Lê-Quôc @alq
- 2. The Data Mullet From All SQL to No SQL back to Some SQL Alexis Lê-Quôc @alq
- 3. Alexis Lê-Quôc @alq This Talk • A (mostly) DIRTy Architecture for... • A new application (datadoghq.com) on a limited budget • Running on a public cloud • Focussing on data stores.
- 4. Some context
- 5. Servers Monitoring IaaS, PaaS Usage Analytics Perf. Management Apps Hosting CDNs Asset Management SDLC Ops team Dev team
- 7. Dev & Ops “collaborate” Alexis Lê-Quôc @alq
- 8. Concretely, what does Datadog do?
- 9. Alexis Lê-Quôc @alq etc.
- 10. Watching real time feeds Looking for patterns Alexis Lê-Quôc @alq Constant telemetry Real-time Bursty batches Share
- 11. Alexis Lê-Quôc @alq Data Taxonomy Metrics Unique visitors Load Transaction duration ... Events Conversations Alerts Build & Deploys ...
- 12. Alexis Lê-Quôc @alq Unit of scale • 1 source, typically a server • 100 metrics • Every 15 s • 24,000 points per hour • ~3 bytes per point • 100 KB/hour, 850 MB/year • Events • whenever they occur • Highest resolution: 1s • Small payload + metadata
- 13. Alexis Lê-Quôc @alq ACID, BASE & DIRT • ACID • http://en.wikipedia.org/wiki/ACID • BASE • http://en.wikipedia.org/wiki/Eventual_consistency • DIRT (Bryan Cantrill at Surge 2010) • http://dtrace.org/resources/bmc/DIRT.pdf
- 14. Let’s dig some DIRT
- 15. DI-RealTime
- 16. Alexis Lê-Quôc @alq The Consequences of DIRT? Latency • Data consumed by people (and machines) • Low end-to-end latency (5-15s) • Psycho-physiological Factor • Same order of magnitude as email/SMS* http://citeseerx.ist.psu.edu/viewdoc/* download?doi=10.1.1.76.2465&rep=rep1&type=pdf
- 17. Alexis Lê-Quôc @alq The Consequences of DIRT? Concurrency • Concurrent events & data points show up in sync • Access Patterns? • All recent data, e.g. last 24 hours
- 18. Alexis Lê-Quôc @alq The Consequences of DIRT? Tolerance to noise • Not a System of Record • “Real-time” decisions • Drop (some) individual data points rather be late • Applies to metrics, not events
- 19. Noise but no Latency Latency but no Noise Alexis Lê-Quôc @alq Cross here? Or here?
- 20. DataIntensive-RT
- 21. Alexis Lê-Quôc @alq The Consequences of DIRT? Storage • Business Cycles • Retention Policy > Business Cycle • E.g. retail, education 12 months • Elastic Storage • !CAPEX
- 22. Alexis Lê-Quôc @alq The Consequences of DIRT? Latency • Datadog, a data exploration app for people • Looking for patterns • Ideal: 300 ms round-trip • Access patterns for long-term data? • Storage trade-off: precompute oft-used properties • Run-time Trade-off: want longer timespan, get lower resolution • != RRD
- 24. Alexis Lê-Quôc @alq Aggregate Constant data influx Large data sets
- 25. Alexis Lê-Quôc @alq Aggregate Constant data influx Large data sets Watch & Share Real-time updates On-the-fly data analysis
- 26. Watch & Share Real-time updates On-the-fly data analysis Alexis Lê-Quôc @alq Aggregate Constant data influx Large data sets Look for Patterns On-demand visualization Background data analysis
- 27. Watch & Share Real-time updates On-the-fly data analysis Alexis Lê-Quôc @alq Aggregate Constant BASE data DIRT influx Large data sets Look for Patterns On-demand visualization Background data analysis
- 28. Watch Real-time DIRT & Share updates On-the-fly data analysis Alexis Lê-Quôc @alq Aggregate Constant BASE data DIRT influx Large data sets Look for Patterns On-demand visualization Background data analysis
- 29. Watch Real-time DIRT & Share updates On-the-fly data analysis Alexis Lê-Quôc @alq Aggregate Constant BASE data DIRT influx Large data sets Look for On-demand BASE Patterns visualization Background data analysis
- 30. Watch Real-time DIRT & Share updates On-the-fly data analysis Alexis Lê-Quôc @alq Aggregate Constant BASE data DIRT influx Large data sets Look for On-demand BASE Patterns visualization Background data analysis Datadog = DIRT + BASE + a tiny bit of ACID
- 31. Alexis Lê-Quôc @alq How It All Fits Together
- 32. Alexis Lê-Quôc @alq The Mullet All SQL in front, NoSQL party in the back
- 33. Alexis Lê-Quôc @alq Actual Stack
- 34. Alexis Lê-Quôc @alq Choices, choices • 5 axes • Volume of Data • Latency • Ops: wake-up-in-the-middle-of-the-night factor • Dev: community & tools • Cost as in “a function of X”
- 36. Alexis Lê-Quôc @alq Durable, Large-Scale Storage • Postgres • Mongo • Cassandra • (Riak) • SciDB
- 37. Alexis Lê-Quôc @alq Durable, Large-Scale Storage • Postgres • Itemized data points in a time series are useless • BLOB management not fun • Mongo • Cassandra • (Riak) • SciDB
- 38. Alexis Lê-Quôc @alq Durable, Large-Scale Storage • Postgres • Mongo • SciDB • Cassandra • (Riak)
- 39. Alexis Lê-Quôc @alq Durable, Large-Scale Storage • Postgres • Mongo • Durability in question in 2010 • SciDB • Cassandra • (Riak)
- 40. Alexis Lê-Quôc @alq Durable, Large-Scale Storage • Postgres • Mongo • SciDB • Very very early • Cassandra • (Riak)
- 41. Alexis Lê-Quôc @alq Durable, Large-Scale Storage • Postgres • Mongo • SciDB • Our pick: Cassandra • (Riak)
- 42. Alexis Lê-Quôc @alq Cassandra: Volume of Data • 100s of hosts, 150TB at FB in 2010 • Easy to distribute data, durable quorum writes
- 43. Alexis Lê-Quôc @alq Cassandra: Latency • < 10ms on writes • reads more variable (on EC2)* * More on this in a bit
- 44. Alexis Lê-Quôc @alq Cassandra: Ops • Release Engineering too aggressive • ~10 releases since 1/2011 on 0.7 branch • Good resilience to node loss in the later 0.7 versions • Annoying idiosyncrasies (cassandra.yaml, predictability of disk use)
- 45. Alexis Lê-Quôc @alq Cassandra: Dev • Bizarre nomenclature (rows, columns... families?) • Cumbersome data access • Limited Semantics when used to SQL • Good libraries
- 46. Alexis Lê-Quôc @alq Cassandra: Cost • Ops time • I/O limits raised by increasing number of nodes • Thereby increasing costs,
- 47. Alexis Lê-Quôc @alq Riak • Prototyped out of spite for Cassandra 0.7[0123] • We ♡ Erlang • Great folks • But Cassandra pain subsided, priorities shifted. • git merge datadog/riak did not happen
- 48. Choosing In-Mem
- 49. Alexis Lê-Quôc @alq In-memory DB • We started with Redis • Then we stopped looking :)
- 50. Alexis Lê-Quôc @alq Redis • Volume of Data • Limited by available RAM, easy partitioning in our case • Latency • << 5 ms, dominated by network • Ops • Low-maintenance, stable, predictable, replicated, boringly rock-solid • Dev • Brilliant, clear docs, simple protocol, oft-used native data structures • Cost • ~ cost of RAM on EC2
- 51. Choosing a SQL Data Store
- 52. Alexis Lê-Quôc @alq General-purpose data store • We ♡ SQL • Oracle • Postgres
- 53. Alexis Lê-Quôc @alq Oracle in numbers • base license 47.5 • clustered db 23 • replication 10 • partitioning 11.5 • analytics 23 • in-mem cache 23 • total: $138,000
- 54. Alexis Lê-Quôc @alq Oracle in numbers • base license 47.5 • clustered db 23 • replication 10 • partitioning 11.5 • analytics 23 • in-mem cache 23 • total: $138,000 • for 2 cores • + 22% annual support • Just in licenses...
- 55. Alexis Lê-Quôc @alq Oracle in numbers • base license 47.5 • clustered db 23 • replication 10 • partitioning 11.5 • analytics 23 • in-mem cache 23 • total: $138,000 • for 2 cores • + 22% annual support • Just in licenses...
- 56. Alexis Lê-Quôc @alq General-purpose data store • Oracle • Postgres
- 57. Alexis Lê-Quôc @alq Postgres • Volume of Data • High GBs, Low TBs • Latency • 10-100 ms after EXPLAIN ANALYZE • Ops • Low-maintenance, stable, predictable, replicated, boringly rock-solid • Dev • Well understood by (a certain class of) engineers • Cost, a function of storage latency
- 58. Alexis Lê-Quôc @alq Not forgetting... • VoltDB • RAM-based, potentially a match for our DIRTy parts • Stored procedures, an acquired taste • Home-grow data stores (soon) • Rainbird • ...
- 59. Alexis Lê-Quôc @alq The Data Mullet • All open-source, good if you’re ready to dive in code • $0 CAPEX • All OPEX on EC2
- 60. Alexis Lê-Quôc @alq The Data Mullet on EC2 Structural Weakness: I/O latency at moderate throughputs
- 61. One “bad” cassandra query Alexis Lê-Quôc @alq
- 62. Clogging the I/O pipes on EC2 Alexis Lê-Quôc @alq Maximum Average Wait: up to 670 ms Maximum Service Time: up to 5 ms While writing 100 MB/s and reading 30 MB/s
- 63. Alexis Lê-Quôc @alq Average wait in ms Transfer per seconds Consumer HD: ~75 tps SSD: 1-30 Ktps DEV tps rd_sec/s wr_sec/s avgrq-sz avgqu-sz await svctm %util 03:35:02 PM dev8-80 380 24000 5.7 62 47 130 1.3 47 03:35:02 PM dev8-96 370 24000 5.6 63 46 120 1.2 45 03:35:02 PM dev8-112 380 24000 5.5 63 46 120 1.2 46 03:35:02 PM dev8-128 380 24000 7.2 63 56 150 1.3 50 Average service time in ms Read throughput in sector/s Total: 46 MB/s Another “Bad” Query
- 64. Mitigation of I/O issues?
- 65. Alexis Lê-Quôc @alq Cassandra: I/O Mitigation • More nodes, more RAM, more partitions, more parallelism • $$$
- 66. Alexis Lê-Quôc @alq Postgres: I/O Mitigation • Scale up to a point • Replicate • Move to bare Metal => $$$ • A well-trodden but difficult path
- 67. Alexis Lê-Quôc @alq Better yet... • Less dependency on low-latency, durable storage • Move more data to RAM (Redis) • Archive immutable data • S3/Cloudfront?
- 68. Alexis Lê-Quôc @alq A digression: Your Very Own Chaos Monkey • Instances go bye-bye • https://bugs.launchpad.net/ubuntu/+source/linux-ec2/+bug/741224 • Instances go bye-bye, take 2 (high load) • https://bugs.launchpad.net/ubuntu/+source/linux-ec2/+bug/708920
- 69. Alexis Lê-Quôc @alq Takeaway • By mixing and matching open-source SQL (PG) and NoSQL (Redis, Cassandra) Datadog has been able to quickly & simply get up-and-running with “$0” down payment on infrastructure.
- 70. http://datadoghq.com @datadoghq Alexis Lê-Quôc @alq