Scylla Summit 2022: Overcoming the Performance Cost of Streaming Transactions
0 likes587 views
The document discusses Redpanda, a modern streaming platform designed for high-performance streaming transactions with Kafka protocol support. It highlights its unique features like append-only writes, transactional semantics, and optimizations for consistent performance during transactions. The author, Denis Rystsov, also provides insights into the architecture and capabilities of Redpanda in executing cross-ledger transactions.
Scylla Summit 2022: Overcoming the Performance Cost of Streaming Transactions
- 1. Overcoming the Performance Cost of Streaming Transactions Denis Rystsov Staff Engineer
- 2. Denis Rystsov ■ Invented CASPaxos ■ Designed Redpanda transactions ■ Tested consistency of Cosmos DB ■ Love surfing in cold water Staff Engineer
- 4. What is Redpanda? ■ Modern streaming platform supporting Kafka protocol ■ Written in C++ ■ Uses thread per core ■ Based on Seastar
- 5. What is Redpanda? ■ Specialized distributed replicated transactional storage • append only writes • sequential reads • persistent cursors • primitives for consumers cooperation
- 8. Transactions ■ All or nothing semantics for • Write to multiple topics • Consume-transform-produce loop
- 10. ■ its own topic (Redpanda) ■ its own table (Scylla) Each service has
- 11. ■ check balance ■ move money between a user’s own accounts
- 12. ■ read a personalized ledger and update balance ■ check that an account have enough money before scheduling a transfer to an another user ■ execute cross ledger transactions (Redpanda transactions)
- 13. ■ NMSI: please just don’t conflict ■ SR: all updates are ordered ■ SI: all reads are ordered
- 15. Money transfer between two random (out of 64) partitions Total threads (num. clients x thread per client) Redpanda
- 16. Money transfer between two random (out of 64) partitions Total threads (num. clients x thread per client) Redpanda Kafka
- 17. Money transfer between two random (out of 64) partitions Total threads (num. clients x thread per client) Redpanda Kafka
- 18. ■ beginTransaction(); ■ send("account1",...); ■ send("account2",...); ■ commitTransaction();
- 19. Add Topic to Txn
- 20. Add Topic to Txn
- 21. Add Topic to Txn
- 22. Add Topic to Txn
- 23. ■ beginTransaction(); ■ send("account1",...); ■ send("account2",...); ■ commitTransaction();
- 24. Send
- 25. Send
- 26. Send
- 27. Send
- 28. ■ beginTransaction(); ■ send("account1",...); ■ send("account2",...); ■ commitTransaction();
- 29. Commit
- 30. Commit
- 31. Commit
- 32. Commit
- 33. Redpanda optimization ■ Lets send() be eventually consistent ■ Validate that it’s fully replicated during commit ■ Parallel commits ■ RAM over disk
- 34. Add Topic to Txn
- 35. Add Topic to Txn
- 36. Add Topic to Txn
- 37. Send
- 38. Send
- 39. Commit
- 40. Commit
- 41. Commit
- 42. Commit
- 43. Commit
- 44. Commit
- 45. Money transfer between two random (out of 64) partitions Total threads (num. clients x thread per client) Redpanda Kafka
- 46. Thank you! Stay in touch @rystsov denis@vectorized.io