Build Low-Latency Applications in Rust on ScyllaDB
1 like589 views
The document outlines a virtual developer workshop on building low-latency applications in Rust using ScyllaDB, featuring presenters Piotr Sarna and Felipe Cardeneti Mendes. It covers topics such as the ScyllaDB Rust driver, an application overview, performance profiling, and offers guidance on setting up an IoT Rust application. The workshop aims to highlight the advantages of Rust and the ScyllaDB database for scalable and efficient application development.
![tokio-console is a very cool project which brings perf/top capabilities to async Rust.
Prerequisites:
Installation:
$ cargo install tokio-console
Cargo.toml:
[dependencies]
console-subscriber = "0.1"
Code:
console_subscriber::init();
Compilation flags:
RUSTFLAGS="--cfg tokio_unstable" cargo run <your-project>
Profiling: tokio-console](https://image.slidesharecdn.com/scyllarustworkshop2022-march3piotrfelipe-220303185258/85/Build-Low-Latency-Applications-in-Rust-on-ScyllaDB-15-320.jpg)
![On Linux, flamegraph scripts use perf underneath. Perf is a well-known and very powerful
profiling tool with direct support in the kernel. Rust programs, since they're compiled via LLVM,
are perfectly capable candidates for profiling with perf.
Prerequisites:
Cargo.toml (the root one in case you use multiple workspaces):
[profile.release]
lto = false
debug = true
System (Linux-specific):
$ echo 0 | sudo tee /proc/sys/kernel/perf_event_paranoid
$ echo 0 | sudo tee /proc/sys/kernel/kptr_restrict
Profiling: perf](https://image.slidesharecdn.com/scyllarustworkshop2022-march3piotrfelipe-220303185258/85/Build-Low-Latency-Applications-in-Rust-on-ScyllaDB-19-320.jpg)
Build Low-Latency Applications in Rust on ScyllaDB
- 1. Piotr Sarna - Principal Software Engineer, ScyllaDB Felipe Cardeneti Mendes - Solutions Architect, ScyllaDB Andrew Brown - Director of Product Growth, ScyllaDB Virtual Developer Workshop Build Low-Latency Applications in Rust on ScyllaDB
- 2. Presenters 2 Piotr Sarna Piotr is a software engineer very keen on open-source projects, C++ and Rust. Felipe Cardeneti Mendes Felipe Mendes is an IT Specialist with years of experience with Linux. In ScyllaDB, he works as a Solutions Architect Andrew Brown Andrew has been working with distributed systems in roles across product development and product marketing.
- 3. Agenda ● Background: Why Rust, why ScyllaDB? ● The ScyllaDB Rust driver ● The Rust sample app walkthrough ● Profiling Rust performance
- 4. IOT Rust Application Setup $ git clone https://github.com/fee-mendes/rust-driver-example/ $ cd rust-driver-example/docker-compose $ docker-compose up -d $ docker exec -it rust-app /bin/bash Minimum requirements: - Linux or OSX x86 - Docker installed and setup - Quadcore CPU - 2 GB of memory available - Windows is not supported Join the #rust-driver channel on ScyllaDB Slack to discuss any issues or questions you might have.
- 5. Low latency, close to hardware schedulers Perfect horizontal & Vertical scale 5 1000 Nodes Cluster 2000 Cluster K8S Deployment 60TB per Node 256 Cores per Node 1B Operations per Second About ScyllaDB: Fast and Scalable
- 6. 6 Poll: How proficient are you with the Rust language?
- 7. + Asynchronous, non-blocking runtimes + Tokio: Most widely used Rust runtime + Seastar: C++ runtime for ScyllaDB + Fast, flexible, and reliable + Scalable, allows high concurrency and low latency + Green and sustainable Why Rust? Why Tokio? Why ScyllaDB? Rust Driver ScyllaDB
- 8. The effort started during a 2020 internal hackathon at ScyllaDB. The existing CQL drivers (cassandra-cpp, cdrs) in the Rust ecosystem were not good enough for us: ● Not fully async ● Unsatisfactory performance ● Not scalable enough ● Known, long-standing bugs ● No support for ScyllaDB-specific optimizations ○ e.g., Shard-per-core ● etc. Some history
- 9. Our design goals: ● fully asynchronous ● token-aware, shard-aware ● ease of use ● maximizing performance ● minimizing overheads ● extensibility ● an extremely cool logo Based on the principles above, we created the ScyllaDB Rust Driver. Some history
- 10. ● Most popular async runtime, which should translate to best adoption ● It would be tempting to write the driver in a runtime-agnostic way, but it's hard: ○ Not all API's are well defined yet ○ Tokio offers quite complete support for TCP communication, timeouts and other useful abstractions ● Very actively developed Why Tokio?
- 11. Let’s Code 11
- 12. IOT Application Overview Metric Collector Metric Reader UUID Finder Write to ScyllaDB in parallel Deploy schema (ks/tables) Generate data: 100 devices 3 days Every 5 minutes Device metrics aggregator Analytics sample Split token-ring in small parts: Efficient full table scan token() function usage BYPASS CACHE Single device queries Real-time sample Partition scan: MAX(), AVG(), MIN() Range queries Date/Time handling
- 13. Poll: What databases do you use (or are planning to use) for Rust Apps?: 13
- 14. Profiling 14
- 15. tokio-console is a very cool project which brings perf/top capabilities to async Rust. Prerequisites: Installation: $ cargo install tokio-console Cargo.toml: [dependencies] console-subscriber = "0.1" Code: console_subscriber::init(); Compilation flags: RUSTFLAGS="--cfg tokio_unstable" cargo run <your-project> Profiling: tokio-console
- 17. Rust ecosystem has a very convenient way of generating flamegraphs: cargo flamegraph Usage: $ cargo install flamegraph $ cargo flamegraph your-project Profiling: cargo flamegraph
- 19. On Linux, flamegraph scripts use perf underneath. Perf is a well-known and very powerful profiling tool with direct support in the kernel. Rust programs, since they're compiled via LLVM, are perfectly capable candidates for profiling with perf. Prerequisites: Cargo.toml (the root one in case you use multiple workspaces): [profile.release] lto = false debug = true System (Linux-specific): $ echo 0 | sudo tee /proc/sys/kernel/perf_event_paranoid $ echo 0 | sudo tee /proc/sys/kernel/kptr_restrict Profiling: perf
- 20. By the way, when run on Linux, cargo flamegraph leaves a side effect - a perf.data file, which is a result format from calling perf record. Here's another example: $ perf record -g ./target/release/metric-collector $ perf report -g Profiling: perf
- 21. Felipe Cardeneti Mendes felipemendes@scylladb.com Piotr Sarna sarna@scylladb.com Q&A Andrew Brown andrew.brown@scylladb.com
- 22. United States 2445 Faber St, Suite #200 Palo Alto, CA USA 94303 Israel Maskit 4 Herzliya, Israel 4673304 www.scylladb.com @scylladb Thank You!