RabbitMQ Operations
15 likes5,290 views
The document provides insights and best practices for operating RabbitMQ, focusing on aspects such as provisioning, OS resources tuning, memory and disk usage, and cluster formation. It highlights the importance of configuring server settings for optimal performance, managing memory effectively, and handling network partitions. Additionally, it discusses upcoming features in RabbitMQ versions 3.6 and later, including pluggable cluster formation and improved management tools.
![[{registered_name,rabbit_mgmt_db},
{current_function,{erlang,hibernate,3}},
{initial_call,{proc_lib,init_p,5}},
{status,waiting},
{message_queue_len,0},
{messages,[]},
{links,[<5477.358.0>]},
{dictionary,[{'$ancestors',[<5477.358.0>,rabbit_mgmt_sup,rabbit_mgmt_sup_sup,
<5477.338.0>]},
{'$initial_call',{gen,init_it,7}}]},
{trap_exit,false},
{error_handler,error_handler},
{priority,high},
{group_leader,<5477.337.0>},
{total_heap_size,167},
{heap_size,167},
{stack_size,0},
{reductions,318},
{garbage_collection,[{min_bin_vheap_size,46422},
{min_heap_size,233},
{fullsweep_after,65535},
{minor_gcs,0}]},
{suspending,[]}]](https://image.slidesharecdn.com/rabbitmqoperations-151027051046-lva1-app6891/85/RabbitMQ-Operations-57-320.jpg)
![rabbitmqctl rename_cluster_node [old name] [new name]](https://image.slidesharecdn.com/rabbitmqoperations-151027051046-lva1-app6891/85/RabbitMQ-Operations-73-320.jpg)
RabbitMQ Operations
- 2. About me
- 3. About me • RabbitMQ staff engineer at Pivotal
- 4. About me • RabbitMQ staff engineer at Pivotal • @michaelklishin just about everywhere
- 7. About this talk • Brain dump from years of answering questions
- 8. About this talk • Brain dump from years of answering questions • Focusses on the most recent release (3.5.6)
- 10. Provisioning
- 11. Provisioning • Be aware of mirrors: GitHub, Bintray, …
- 12. Provisioning • Be aware of mirrors: GitHub, Bintray, … • Looking into community-hosted mirrors
- 13. Provisioning • Be aware of mirrors: GitHub, Bintray, … • Looking into community-hosted mirrors • Use packages + Chef/Puppet/…
- 15. OS resources
- 16. OS resources • Modern Linux defaults are absolutely inadequate for servers
- 17. ulimit -n default: 1024
- 19. Set ulimit -n and fs.file-max to 500K and forget about it
- 20. TCP keepalive timeout: from 11 minutes to over 2 hours by default
- 21. net.ipv4.tcp_keepalive_time = 6 net.ipv4.tcp_keepalive_intvl = 3 net.ipv4.tcp_keepalive_probes = 3
- 22. enable client heartbeats, e.g. with an interval of 6-12 seconds
- 23. OS resources • Modern Linux defaults are absolutely inadequate for servers • Tuning for throughput vs. high number of concurrent connections
- 24. Throughput: larger TCP buffers
- 25. net.core.rmem_max = 16777216 net.core.wmem_max = 16777216
- 26. rabbit.hipe_compile = true (only on Erlang 17.x or 18.x)
- 27. Concurrent connections: smaller TCP buffers, low tcp_fin_timeout, tcp_tw_reuse = 1, …
- 28. rabbit.tcp_listen_options.sndbuf rabbit.tcp_listen_options.recbuf rabbit.tcp_listen_options.backlog
- 29. Reduce per connection RAM use by 10x rabbit.tcp_listen_options.sndbuf = 16384 rabbit.tcp_listen_options.recbuf = 16384
- 31. Reduce per connection RAM use by 10x Throughput drops by a comparable amount
- 34. Careful with tcp_tw_reuse behind NAT* * http://vincent.bernat.im/en/blog/2014-tcp-time-wait-state-linux.html
- 38. Disk space
- 39. Disk space • Pay attention to what partition /var/lib ends up on
- 40. Disk space • Pay attention to what partition /var/lib ends up on • Transient messages can be paged to disk
- 41. Disk space • Pay attention to what partition /var/lib ends up on • Transient messages can be paged to disk • RabbitMQ’s disk monitor isn’t supported on all platforms
- 43. RAM usage
- 45. RAM usage • rabbit.vm_memory_high_watermark • rabbit.vm_memory_high_watermark_paging_ratio
- 47. RAM usage • rabbit.vm_memory_high_watermark • rabbit.vm_memory_high_watermark_paging_ratio • Significant paging efficiency improvements in 3.5.5-3.5.6
- 48. RAM usage • rabbit.vm_memory_high_watermark • rabbit.vm_memory_high_watermark_paging_ratio • Significant paging efficiency improvements in 3.5.5-3.5.6 • Disable rabbit.fhc_read_buffering (3.5.6+)
- 50. recon
- 51. Ability to set VM RAM watermark as absolute value is coming in 3.6
- 53. Stats collector falls behind
- 54. Stats collector falls behind • Management DB stats collector can get overwhelmed
- 55. Stats collector falls behind • Management DB stats collector can get overwhelmed • Key symptom: disproportionally higher RAM use on the node that hosts management DB
- 56. rabbitmqctl eval 'P = whereis(rabbit_mgmt_db), erlang:process_info(P).'
- 60. rabbitmqctl eval 'P = whereis(rabbit_mgmt_db), erlang:exit(P, please_crash).'
- 61. Parallel stats collector is coming in 3.7
- 64. Cluster formation • Node restart order dependency
- 65. Cluster formation • Node restart order dependency • github.com/rabbitmq/rabbitmq-clusterer
- 66. Cluster formation • Node restart order dependency • github.com/rabbitmq/rabbitmq-clusterer • github.com/aweber/rabbitmq-autocluster
- 68. Backups
- 69. How do I back up? • cp $RABBITMQ_MNESIA_DIR + tar
- 70. How do I back up? • cp $RABBITMQ_MNESIA_DIR + tar • Replicate everything off-site with exchange federation + set message TTL via a policy
- 72. Hostname changes
- 73. rabbitmqctl rename_cluster_node [old name] [new name]
- 76. Network partition handling • When in doubt, use “autoheal”
- 77. Network partition handling • When in doubt, use “autoheal” • “Merge” is coming but has very real downsides, too
- 79. Misc
- 80. Misc • Don’t use default vhost and/or credentials
- 81. Misc • Don’t use default vhost and/or credentials • Don’t use 32-bit Erlang
- 82. Misc • Don’t use default vhost and/or credentials • Don’t use 32-bit Erlang • Use reasonably up-to-date releases
- 83. Misc • Don’t use default vhost and/or credentials • Don’t use 32-bit Erlang • Use reasonably up-to-date releases • Participate in rabbitmq-users
- 84. Misc • OCF resource template from Fuel (by Mirantis)
- 85. Misc • OCF resource template from Fuel (by Mirantis) • Use TLS
- 87. Coming in 3.6
- 88. Coming in 3.6 • In process file buffering disabled by default
- 89. Coming in 3.6 • In process file buffering disabled by default • Queue master to node distribution strategies
- 90. Coming in 3.6 • In process file buffering disabled by default • Queue master to node distribution strategies • SHA-256 (or 512) for password hashing
- 91. Coming in 3.6 • In process file buffering disabled by default • Queue master to node distribution strategies • SHA-256 (or 512) for password hashing • More responsive management UI with pagination
- 92. Coming in 3.6 • In process file buffering disabled by default • Queue master to node distribution strategies • SHA-256 (or 512) for password hashing • More responsive management UI with pagination • Streaming rabbitmqctl
- 93. Coming past 3.6
- 94. Coming past 3.6 • Pluggable cluster formation (à la ElasticSearch)
- 95. Coming past 3.6 • Pluggable cluster formation (à la ElasticSearch) • On disk data recovery tools
- 96. Coming past 3.6 • Pluggable cluster formation (à la ElasticSearch) • On disk data recovery tools • Better CLI tools
- 97. Coming past 3.6 • Pluggable cluster formation (à la ElasticSearch) • On disk data recovery tools • Better CLI tools • Easier off-site replication
- 98. Coming past 3.6 • Pluggable cluster formation (à la ElasticSearch) • On disk data recovery tools • Better CLI tools • Easier off-site replication • “Merge” partition handling strategy (no earlier than 3.8)
- 100. Thank you
- 101. Thank you • @michaelklishin • github.com/michaelklishin • rabbitmq-users • Our team is hiring!