An update from the RabbitMQ team - Michael Klishin
3 likes674 views
Michael Klishin from Pivotal gave an update on RabbitMQ. He discussed the state of the 3.7.x releases which focused on deployment automation, operator friendliness, and stability. Future plans include improved scalability, reliability, and simplifying upgrades. Key projects include implementing quorum queues using Raft for improved mirroring, adding OAuth 2.0 support, developing a new schema storage system, and allowing mixed version clusters. The 3.8 release will include quorum queues, OAuth 2.0, and initial mixed version cluster support.
![Raft: an oversimplified explanation
Raft
Follower
Leader
Follower
Commit index = 0
Enqueue
Append entry
Append entry
Log = []
Log = []
Log = []](https://image.slidesharecdn.com/michael-klishin-slides-181212143819/85/An-update-from-the-RabbitMQ-team-Michael-Klishin-20-320.jpg)
![Raft: an oversimplified explanation
Raft
Follower
Leader
Follower
Commit index = 1
Success
Success
Success
Log = [{1, Enqueue}]
Log = [{1, Enqueue}]
Log = [{1, Enqueue}]](https://image.slidesharecdn.com/michael-klishin-slides-181212143819/85/An-update-from-the-RabbitMQ-team-Michael-Klishin-21-320.jpg)
![Quorum queues
The State of Quorum Queues
! Reasonably polished
! Throughput beats mirrored queues
! Passes a [slightly modified] Jepsen test
! Available in RabbitMQ 3.8.0-beta.1 today](https://image.slidesharecdn.com/michael-klishin-slides-181212143819/85/An-update-from-the-RabbitMQ-team-Michael-Klishin-22-320.jpg)
![Next gen schema storage
The State of Mnevis
! Currently an area of active research
! Mnesia will be used in a node-local way
! Mnesia is quite extensible
! Raft is not necessarily a great fit for transaction log propagation due to [relatively] high
latency](https://image.slidesharecdn.com/michael-klishin-slides-181212143819/85/An-update-from-the-RabbitMQ-team-Michael-Klishin-34-320.jpg)
An update from the RabbitMQ team - Michael Klishin
- 2. © Copyright 2018 Pivotal Software, Inc. All rights Reserved. Version 1.0 November 2018 An update from the RabbitMQ team
- 3. Who am I? Michael Klishin ! Senior Engineer at Pivotal ! RabbitMQ contributor since 2010 ! @michaelklishin on GitHub and most other places
- 4. Cover w/ Image Agenda ■ The state of RabbitMQ 3.7.x ■ Future directions ■ What’s coming in 3.8 and beyond
- 5. The state of 3.7.x An update from the RabbitMQ team
- 6. RabbitMQ 3.7.0 3.7.0 release themes ! Deployment automation friendliness ! Operator friendliness ! Stability ! Setting up stage for larger changes
- 7. RabbitMQ 3.7.0 Operator friendliness ! Distributed stats storage ! Per-vhost limits ! Operator policies ! New logging subsystem ! Proxy protocol Deployment automation friendliness ! Peer discovery subsystem ! New, easy to generate configuration format ! Extensible command line tools
- 8. RabbitMQ 3.7.x What have we been up to in 3.7.x? ! Best practices ! Documentation ! Non-breaking refinements ! Ecosystem improvements
- 9. RabbitMQ 3.7.x Documentation ! Upgrades guide ! Monitoring guide ! Production checklist updates ! Runtime tuning guide Guidance & best practices ! Blue/green deployment upgrades ! Workload-specific runtime tuning
- 10. RabbitMQ 3.7.x Non-breaking refinements ! Erlang 21 support ! OAuth 2 support ! New metrics ! New CLI commands ! Greater adoption of the new config format in plugins ! Usability improvements ! Kubernetes deployment example ! Improved default runtime flags* ! LDAP Ecosystem Improvements ! Client libraries (primarily Java, .NET, Python) ! Erlang packaging
- 11. Future directions An update from the RabbitMQ team
- 12. Future directions General directions ! Improved reliability and operator friendliness ! Improved scalability ! Simplify upgrades ! Continue improving and expanding the ecosystem ! Move away from home grown distsys algorithms 🤦 " ! Improved correctness ! Repay technical debt
- 13. What problems are we addressing? Scalability ! Mirroring uses a ring topology which is linear ! Excessive bandwidth usage ! No DC awareness ! The core is not protocol-agnostic which leads to overhead Reliability ! Recovery from failures is not always predictable (or easy to reason about) ! Fails to pass certain resiliency tests in some scenarios ! Queue sync after failure can take too long and cause a thundering herd ! Schema data store (Mnesia) is overly opinionated and isn’t a great fit ! Lack of unified WAL/oplog makes backups hard
- 14. Future directions Currently in progress ! Quorum queues (mirrored queues 2.0) based on Raft ! OAuth 2.0 support ! Mixed version clusters ! Mnevis: a new schema data store ! Protocol-agnostic core
- 15. Quorum queues An update from the RabbitMQ team
- 16. How does Raft-based replication compare? Improved scalability ! No ring topology, parallel replication ! More reasonable bandwidth usage* ! Opens the door to cross-DC replication Improved reliability ! Recovery from failures is predictable and well defined ! Queue sync after failure transfers as little data as possible ! Passes more resiliency tests ! Opens the door to a unified WAL/oplog to simplify backups
- 17. What is Raft? An update from the RabbitMQ team
- 18. What is Raft? ! A group of algorithms for reaching consensus in a distributed system ! Implementer-focused ! Proven ! Multiple implementations ! Industry use ! TLA+ specification
- 19. Raft: an oversimplified explanation Raft Follower Leader Follower
- 20. Raft: an oversimplified explanation Raft Follower Leader Follower Commit index = 0 Enqueue Append entry Append entry Log = [] Log = [] Log = []
- 21. Raft: an oversimplified explanation Raft Follower Leader Follower Commit index = 1 Success Success Success Log = [{1, Enqueue}] Log = [{1, Enqueue}] Log = [{1, Enqueue}]
- 22. Quorum queues The State of Quorum Queues ! Reasonably polished ! Throughput beats mirrored queues ! Passes a [slightly modified] Jepsen test ! Available in RabbitMQ 3.8.0-beta.1 today
- 23. Quorum queues Implementing Quorum Queues ! Powered by our own from scratch Raft implementation ! github.com/rabbitmq/ra ! Heavily geared towards but not specific to RabbitMQ ! Deviates from Raft a little bit to reduce network and disk I/O ! Does not sacrifice correctness ! Includes a peer unavailability detection library ! Optimized for throughput, adapts I/O for latency or throughput based on load
- 25. Quorum queues The State of Quorum Queues ! Has limitations ! Doesn’t support some features, e.g. TTL won’t be supported ! Memory management is still a hard problem ! As any new major feature, will take time to mature (you can help by giving it a try!)
- 26. Queue types An update from the RabbitMQ team
- 27. One queue type to rule them all? What are the chances that all these features can work efficiently or be easy to reason about when they are combined? ! Durable ! Auto-delete ! Mirrored ! Lazy ! With TTL ! With message TTL ! With length limit ! With dead-lettering ! With failure recovery settings ! With exclusive consumers ! Affected by plugins ! …
- 28. One queue type per workload? Or is there a better way? ! Durable queues with consensus replication ! Transient queues, auto-delete with TTL and/or length limit ! “Infinite queues” (durable, aggressive paging to disk) ! Lower latency in-memory queues
- 29. OAuth 2.0 Support An update from the RabbitMQ team
- 30. OAuth 2.0 support How does it work? ! Implemented as a plugin, rabbitmq_auth_backend_oauth2 ! OAuth 2.0/JWT token scopes that follow naming conventions are translated to RabbitMQ permissions ! Clients can use any OAuth 2.0 authorization code flow ! Management UI will use the authorization code flow ! Officially supported clients will simplify token renewal
- 31. OAuth 2.0 support The State of OAuth 2.0 Support ! Plugin is done, open source and currently targets RabbitMQ 3.8 ! Will ship in 3.8.0 Milestone 2 ! Management UI needs work ! Targets UAA and AD
- 32. Next gen schema storage An update from the RabbitMQ team
- 33. How does next gen schema storage compare? Improved scalability ! Mnesia does reasonably well here (for schema storage) ! Opens the door to DC awareness ! No long-entrenched Mnesia limitations ! Higher rate of development iteration Improved reliability ! Recovery from failures that works for RabbitMQ users (and maintainers) ! Nodes no longer have to erase themselves in order to re-sync ! Can integrate with the unified WAL/oplog
- 34. Next gen schema storage The State of Mnevis ! Currently an area of active research ! Mnesia will be used in a node-local way ! Mnesia is quite extensible ! Raft is not necessarily a great fit for transaction log propagation due to [relatively] high latency
- 35. Mixed version clusters An update from the RabbitMQ team
- 36. Mixed version clusters How do we get there? ! Relaxing overly conservative restrictions ! Capability testing instead of version testing ! Feature flags ! Extensive internal refactoring ! Somewhat funny APIs to work around Erlang record limitations
- 37. Mixed version clusters The State of Mixed Version Clusters ! Feature flag experiment with promising results so far ! Meant to simplify upgrades ! Long running mixed clusters is not a goal ! Cannot guarantee safety for every possible breaking change ! Needs more feedback
- 38. RabbitMQ 3.8 An update from the RabbitMQ team
- 39. RabbitMQ 3.8 What’s coming in 3.8? ! Quorum queues ! OAuth 2.0 ! Initial mixed version cluster support ! Minor usability improvements
- 40. RabbitMQ 3.9 An update from the RabbitMQ team
- 41. RabbitMQ 3.9 What’s coming in 3.9? ! Protocol-agnostic core ! 🤔
- 42. RabbitMQ 4.0 An update from the RabbitMQ team
- 43. RabbitMQ 4.0 What’s coming in 4.0? ! Next generation schema databas ! Transport beam? Food replicators? Tricorders? 🤔 ! All the breaking changes
- 44. Thank you! An update from the RabbitMQ team
- 45. Transforming How The World Builds Software © Copyright 2018 Pivotal Software, Inc. All rights Reserved.