Scylla Summit 2022: Learning Rust the Hard Way for a Production Kafka+ScyllaDB Pipeline
0 likes1,005 views
The document discusses the transition from Python to Rust for building a scalable Kafka and ScyllaDB pipeline at Numberly, detailing the advantages and challenges of Rust in terms of efficiency, reliability, and overall performance. Alexys Jacob highlights the paradigms that support reliable software development provided by Rust, as well as practical insights into optimizing latency and handling data serialization with Confluent Kafka and Avro. The transition resulted in a significant reduction of application pods and improved latency metrics, showcasing the success of rewriting Python applications into a single, efficient Rust application.
Scylla Summit 2022: Learning Rust the Hard Way for a Production Kafka+ScyllaDB Pipeline
- 1. Learning Rust the Hard Way for a Production Kafka+ScyllaDB Pipeline Alexys Jacob CTO
- 2. Alexys Jacob ■ ScyllaDB awarded Open Source & University contributor ■ Open Source author & contributor • Apache Avro, Apache Airflow, MongoDB, MkDocs… ■ Tech speaker & writer ■ Gentoo Linux developer ■ Python Software Foundation contributing member CTO, Numberly @ultrabug
- 3. Your next 20 minutes ■ The thought process to move from Python to Rust • Context, promises, arguments and decision ■ Learning Rust the hard way • All the stack components I had to work with in Rust • Tips, Open Source contributions and code samples ■ What is worth it? • Graphs, production numbers • Personal notes
- 4. Choosing Rust over Python
- 5. At Numberly, we move and process (a lot of) data using Kafka streams and pipelines that are enriched using ScyllaDB. processor app processor app Project context at Numberly Scylla processor app raw data enriched data enriched data enriched data client app partner API business app
- 6. processor app processor app Pipeline reliability = latency + resilience Scylla processor app raw data enriched data enriched data enriched data client app partner API business app If a processor or ScyllaDB is slow or fails, our business, partners & clients are at risk.
- 7. A major change in our pipeline processors had to be undertaken, giving us the opportunity to redesign them. The (rusted) opportunity Scylla processor app raw data enriched data enriched data enriched data client app partner API business app
- 8. “Hey, why not rewrite those 3 Python processor apps into 1 Rust app?”
- 9. A language empowering everyone to build reliable and efficient software. ■ Secure • Memory and thread safety as first class citizens • No runtime or garbage collector ■ Easy to deploy • Compiled binaries are self-sufficient ■ No compromises • Strongly and statically typed • Exhaustivity is mandatory • Built-in error management syntax and primitives ■ Plays well with Python • PyO3 can be used to run Rust from Python (or the contrary) The (never tried before) Rust promises
- 10. Efficient software != Faster software ■ “Fast” meanings vary depending on your objectives. • Fast to develop? • Fast to maintain? • Fast to prototype? • Fast to process data? • Fast to cover all failure cases? “ Selecting a programming language can be a form of premature optimization
- 11. Efficient software != Faster software ■ “Fast” meanings vary depending on your objectives. • Fast to develop? Python is way faster, did that for 15 years • Fast to maintain? Nobody at Numberly does Rust yet • Fast to prototype? No, code must be complete to compile and run • Fast to process data? Sure: to prove it, measure it • Fast to cover all failure cases? Definitely: mandatory exhaustivity + error handling primitives “ I did not choose Rust to be “faster”. Our Python code was fast enough to deliver their pipeline processing.
- 12. Innovation cannot exist if you don’t accept to lose time. The question is to know when and on what project.
- 13. The Reliable software paradigms ■ What makes me slow will make me stronger. • Low level paradigms (ownership, borrowing, lifetimes). • Strong type safety. • Compilation (debug, release). • Dependency management. • Exhaustive pattern matching. • Error management primitives (Result).
- 14. The Reliable software paradigms ■ What makes me slow will make me stronger. • Low level paradigms (ownership, borrowing, lifetimes). If it compiles, it’s safe • Strong type safety. Predictable, readable, maintainable • Compilation (debug, release). Compiler is very helpful vs a random Python exception • Dependency management. Finally something looking sane vs Python mess • Exhaustive pattern matching. Confidence that you’re not forgetting something • Error management primitives (Result). Handle failure right from the language syntax “ I chose Rust because it provided me with the programming paradigms at the right abstraction level that I needed to finally understand and better explain the reliability and performance of an application.
- 15. Learning Rust the hard way
- 16. Production is not a Hello World ■ Learning the syntax and handling errors everywhere ■ Confluent Kafka + Schema Registry + Avro ■ Asynchronous latency-optimized design ■ ScyllaDB multi-datacenter ■ MongoDB ■ Kubernetes deployment ■ Prometheus exporter ■ Grafana dashboarding ■ Sentry Scylla processor app Confluent Kafka
- 17. Confluent Kafka Schema Registry ■ Confluent Schema Registry breaks vanilla Apache Avro deserialization. • Gerard Klijs’ schema_registry_converter crate helps • I discovered performance problems which we worked on and are being addressed! ■ Latency-overhead-free manual approach:
- 18. Apache Avro Rust was broken! ■ Avro Rust crate given to Apache Avro without an appointed committer. • Deserialization of complex schemas was broken... • I contributed fixes to Apache Avro (AVRO-3232+3240) • Now merged thanks to Martin Grigorov! ■ Rust compiler optimizations give a hell of a boost (once Avro is fixed) • Deserializing Avro is faster than JSON!
- 19. green thread / msg Asynchronous patterns to optimize latency ■ Tricks to make your Kafka consumer strategy more efficient. • Deserialize your consumer messages on the consumer loop, not on green-threads • Spawning a green-thread has a performance cost • Control your green-thread parallelism • Defer to green-threads when I/O starts to be required Kafka consumer + avro deserializer raw data green thread / msg green thread / msg green thread / msg green thread / msg Scylla enriched data
- 20. Absorbing tail latency spikes with parallelism x16 x2 parallelism load
- 21. Scylla Rust (shard-aware) driver ■ The scylla-rust-driver crate is mature enough for production. • Use a CachingSession to automatically cache your prepared statements • Beware: prepared queries are NOT paged, use paged queries with execute_iter() instead!
- 22. Exporting metrics properly for Prometheus ■ Effectively measuring latencies down to microseconds. • Fine tune your histogram buckets to match your expected latencies! ...
- 23. Grafana dashboarding ■ Graph your precious metrics right! • ScyllaDB prepared statement cache size • Query and throughput rates • Kafka commits occurrence • Errors by type • Kubernetes pod memory • ... ■ Visualizing Prom Histograms max by (environment)(histogram_quantile(0.50, processing_latency_seconds_bucket{...}))
- 24. Was it worth it?
- 25. Did I really lose time because of Rust? ■ I spent more time analyzing the latency impacts of code patterns and drivers’ options than struggling with Rust syntax. ■ Key figures for this application: • Kafka consumer max throughput with processing? 200K msg/s on 20 partitions • Avro deserialization P50 latency? 75µs • Scylla SELECT P50 latency on 1.5B+ rows tables? 250µs • Scylla INSERT P50 latency on 1.5B+ rows tables? 2ms
- 26. It went better than expected ■ Rust crates ecosystem is mature, similar to Python Package Index. ■ The scylla-rust-driver is stable and efficient! ■ It took me a while to accept that Apache Avro was broken, not me. ■ 3 Python apps totalling 54 pods replaced by 1 Rust app totalling 20 pods ■ This feels like the most reliable and efficient software I ever wrote!
- 27. Thank you! Stay in touch Alexys Jacob - Join us: ultrabug alexys@numberly.com