Introduction to Fluvio Data Engineer.pdf
- 1. Introduction to Fluvio Data engineers lunch 3/14/2022
- 2. Fluvio is the engine for Real-time Economy Real-time data stack v Low Latency v Reliability v Flexibility v Scale
- 3. Modern Data Stack: Is this enough?
- 4. Problem: Data in Motion â Pipeline of Tools is not abstraction â SQL is not good abstraction for data in motion â Time and History â As developer, I want API
- 5. Stream: Abstraction for data in motion Stream is an unbounded sequence of data
- 6. Fluvio: Intelligent Data Streaming Platform â Open source project â Built on Rust. â Stream as first class citizen. â Cloud-native data streaming architecture. â Intelligent stream processing using WebAssembly.
- 7. Why Rust? â High Performance with Low Latency â Compiled language â No Garbage collection â Zero cost abstraction â Safety and Reliability â No Null Pointer â High level typed language (Similar to Haskell and Scalar) â If it compiles, it works â Batteries built-in â Package Manager â Well established ecosystem â Outstanding IDE support â Stream as first class citizen. â Inter-operate with other languages â C / C++ â Java â Python â JavaScript âAbout 70% of all CVEs at Microsoft are memory safety issuesâ Microsoft
- 8. Rust Performance Memory Latency p99.999 Kafka 1GB 215 ms Fluvio 20MB 50 ms
- 9. Data Stream as first class citizen Rust Stream Storage Durable Stream Smart Stream Networking I/O Server Networking Python Client Rust Stream Python Stream Networking Rust Stream Node.js Stream Node.js Client
- 10. Cloud Native Architecture Streaming Controller Metadata Stream Processing Unit Producer Consumer
- 11. Immutable Durable Stream Temperature Time 10 Old New 20 3 Consumer 1 6 11 5 9 Consumer 2 Consumer 3 Producer 1 Producer 2
- 12. Partitioning and Replication of Durable Stre Stream Processing Unit Stream Processing Unit Stream Processing Unit p1 p2 Stock Transactions p0 Web Clicks ML Model P 0 p0 p0
- 13. Producer and Consumer Client Rust Async runtime OS (Mac, Window, Linux, Browser) Fluvio Rust API & C Wrapper CLI Node API Python API Java API Go API
- 15. SmartModule based transformation Temperature 10 20 3 Consumer 6 11 5 9 SmartModule Network connection
- 16. WebAssembly Technology â Memory Isolation â Efficient â Support multiple Languages â Rust â C++/C â AssemblyScript â C# â Python â Etc.
- 19. Demo