Exploring Phantom Traffic Jams in Your Data Flows
0 likes394 views
This document discusses phantom traffic jams that can occur in data flows even without true hardware bottlenecks. It explains how producer-consumer systems with jitter at the producer, consumer, or dispatcher can lead to self-limiting behavior and effective dispatch rates below capacity. The document advocates for good metrics and analyzing bottlenecks with an interposer that can measure queue lengths rather than assuming hardware limitations. It provides a simulator experiment to demonstrate these effects and conclusions about checking bottlenecks throughout systems instead of only at hardware.
Exploring Phantom Traffic Jams in Your Data Flows
- 1. Exploring Phantom Traffic Jams in Your Data Flows Pavel Emelyanov, Principal Engineer @ScyllaDB
- 2. Poll Where are you in your NoSQL adoption?
- 3. About myself 3 Pavel Emelyanov ● Co-maintainer of Seastar & ScyllaDB ● Ex Linux kernel hacker
- 4. + Infoworld 2020 Technology of the Year + Founded by designers of KVM Hypervisor The Database Built for Gamechangers 4 “ScyllaDB stands apart...It’s the rare product that exceeds my expectations.” – Martin Heller, InfoWorld contributing editor and reviewer “For 99.9% of applications, ScyllaDB delivers all the power a customer will ever need, on workloads that other databases can’t touch – and at a fraction of the cost of an in-memory solution.” – Adrian Bridgewater, Forbes senior contributor + Resolves challenges of legacy NoSQL databases + >5x higher throughput + >20x lower latency + >75% TCO savings + DBaaS/Cloud, Enterprise and Open Source solutions + Proven globally at scale
- 5. 5 +400 Gamechangers Leverage ScyllaDB Seamless experiences across content + devices Fast computation of flight pricing Corporate fleet management Real-time analytics 2,000,000 SKU -commerce management Real-time location tracking for friends/family Video recommendation management IoT for industrial machines Synchronize browser properties for millions Threat intelligence service using JanusGraph Real time fraud detection across 6M transactions/day Uber scale, mission critical chat & messaging app Network security threat detection Power ~50M X1 DVRs with billions of reqs/day Precision healthcare via Edison AI Inventory hub for retail operations Property listings and updates Unified ML feature store across the business Cryptocurrency exchange app Geography-based recommendations Distributed storage for distributed ledger tech Global operations- Avon, Body Shop + more Predictable performance for on sale surges GPS-based exercise tracking
- 6. 6 + Written in C++, built over a Seastar library + Share nothing + All asynchronous About Seastar
- 7. 7 What Kind of Jams?
- 8. 8 + Hardware bottlenecks + Self-limiting + Phantom jams Why Is … So Slow?
- 9. 9 + IO-bound + CPU-bound + Memory-bound + Networking bound Hardware Bottlenecks
- 10. 10 + Throttling and Rate-limiting + https://linuxfoundation.org/webinars/understanding-storage-i-o-under-load/ + https://www.scylladb.com/2022/08/03/implementing-a-new-io-scheduler-algorithm-for-mixed-read -write-workloads/ + Throttle for guarantees + https://wiki.openvz.org/Containers/Guarantees_for_resources + Linux cgroup controllers Self-Limiting
- 11. 11 Phantom Jams
- 12. 12 Producer-Consumer Producer Consumer Message P messages / sec C messages / sec
- 13. 13 + P ≤ C + Producer is process/thread/fiber (pool) doing IO / talking over network + Consumer is disk / NIC / server Producer-Consumer (Cont.)
- 14. 14 And Dispatcher Producer Consumer P messages / sec C messages / sec Dispatcher Message D wake-ups / sec d messages / wake-up
- 15. 15 + Dispatcher examples + IO scheduler + Traffic shaper + API gateway + Dispatcher can impose + Fair-scheduling + Access policy + Buffering + Routing + d ≥ C / D And Dispatcher (Cont.)
- 16. 16 + https://github.com/xemul/queue-dispatch + Seastar-centric (simplified seastar) + Virtual time-axis + Producer + Generates empty messages at given rate + Dispatcher + wakes up every 0.5 ms + submits 1.5 ✕ C / D messages + Consumer + 200k messages / sec by default The Experiment
- 17. 17 + Ideal simulator shows 0.5 ms latency (expected) + Real “components” should show P, D and C on average + Poisson point process + Pause between delays is exponentially distributed Make Them “Real”
- 22. 22 Effective Dispatch Rate Producer Consumer Dispatcher X messages / sec
- 24. 24 + Interposer between producer and consumer + Queue in interposer + Possible jitter (a.k.a. cooperative preemption) + Seastar + Coroutines / Goroutines + Unikernels In RL of <your system>
- 25. 25 + Bottleneck might not be in “hardware” + Good metrics is a must + https://www.scylladb.com/2022/04/19/exploring-phantom-jams-in-your-data-flow/ Conclusions
- 26. Poll How much data do you under management of your transactional database?
- 27. Q&A WANT TO KEEP LEARNING? Join ScyllaDB University for Free: university.scylladb.com SCYLLADB VIRTUAL WORKSHOP Getting Started with ScyllaDB 19 January, 2023, 1PM GMT | 8 AM ET | 6:30 PM IST
- 28. Thank you for joining us today. @scylladb scylladb/ slack.scylladb.com @scylladb company/scylladb/ scylladb/