Troubleshooting RabbitMQ and Microservices That Use It
3 likes1,022 views
The document provides troubleshooting strategies for RabbitMQ and its associated services, focusing on issues faced by both publishers and consumers. It emphasizes the importance of collecting logs, metrics, and utilizing various tools for data analysis, including RabbitMQ's built-in commands and the HTTP API. The author advocates for a structured approach to debugging distributed systems to improve the problem-solving process.
Troubleshooting RabbitMQ and Microservices That Use It
- 2. Who am I? • Staff Engineer, RabbitMQ @ Pivotal
- 3. Who am I? • Staff Engineer, RabbitMQ @ Pivotal • @michaelklishin, github.com/michaelklishin
- 22. Troubleshooting publishers • I/O exceptions (shutdown handlers)
- 23. Troubleshooting publishers • I/O exceptions (shutdown handlers) • Publisher confirms
- 25. When in doubt, borrow ideas from TCP
- 26. Troubleshooting publishers • I/O exceptions (shutdown handlers) • Publisher confirms • Returned message handlers
- 27. Troubleshooting publishers • I/O exceptions (shutdown handlers) • Publisher confirms • Returned message handlers • Invalid payload (e.g. fails to deserialize or decrypt)
- 28. Troubleshooting publishers • I/O exceptions (shutdown handlers) • Publisher confirms • Returned message handlers • Invalid payload (e.g. fails to deserialize or decrypt) • Identifying publisher instances
- 29. Troubleshooting publishers • identifying blocked (throttled) publishers
- 30. Client-provided connection names in RabbitMQ 3.6.3+
- 31. Troubleshooting publishers • identifying blocked (throttled) publishers • retries
- 32. Troubleshooting publishers • spring-amqp can cover all of the above
- 36. Troubleshooting consumers • I/O exceptions
- 37. Troubleshooting consumers • I/O exceptions • Inadequate delivery QoS
- 38. Troubleshooting consumers • I/O exceptions • Inadequate delivery QoS • Lack of confirmations; double-confirming
- 40. Troubleshooting consumers • I/O exceptions • Inadequate delivery QoS • Lack of confirmations; double-confirming
- 41. Troubleshooting consumers • I/O exceptions • Inadequate delivery QoS • Lack of confirmations; double-confirming • Redelivery metrics
- 42. Troubleshooting consumers • I/O exceptions • Inadequate delivery QoS • Lack of confirmations; double-confirming • Redelivery metrics • Identifying consumer instances
- 43. Troubleshooting consumers • Consumer utilization (reported by HTTP API)
- 44. Troubleshooting consumers • spring-amqp can help with some of the above
- 46. — W. Edwards Deming “In God we trust, all others must bring data…”
- 47. — W. Edwards Deming “In God we trust, all others must bring data…”
- 48. — What do you do for a living?
- 49. — What do you do for a living? — Tell people to read the logs.
- 50. Sources of data useful for debugging
- 51. Sources of data useful for debugging • Metrics
- 52. Sources of data useful for debugging • Metrics • Your logs
- 53. Sources of data useful for debugging • Metrics • Your logs • Someone else's logs
- 54. Sources of data useful for debugging • Metrics • Your logs • Someone else's logs • Tracing data
- 55. Sources of data useful for debugging • Metrics • Your logs • Someone else's logs • Tracing data • Wireshark (tcpdump, libpcap)
- 59. Collecting data from RabbitMQ • Logs
- 60. Collecting data from RabbitMQ • Logs • rabbitmqctl status
- 61. Collecting data from RabbitMQ • Logs • rabbitmqctl status • rabbitmqctl environment
- 62. Collecting data from RabbitMQ • Logs • rabbitmqctl status • rabbitmqctl environment • rabbitmq-top (ships with RabbitMQ as of 3.6.3)
- 63. Collecting data from RabbitMQ • Logs • rabbitmqctl status • rabbitmqctl environment • rabbitmq-top (ships with RabbitMQ as of 3.6.3) • HTTP API (lots of metrics)
- 65. curl -u guest:guest http://127.0.0.1:15672/api/overview | python -m json.tool curl -u guest:guest http://127.0.0.1:15672/api/nodes/{node} | python -m json.tool curl -u guest:guest http://127.0.0.1:15672/api/queues | python -m json.tool
- 66. Collecting data from RabbitMQ • Logs • rabbitmqctl status • rabbitmqctl environment • rabbitmq-top (ships with RabbitMQ as of 3.6.3) • HTTP API (lots of metrics) • Message tracing ("firehose")
- 67. Collecting data from RabbitMQ • HTTP API (lots of metrics) • Message tracing ("firehose") • Infrastructure metrics
- 69. Common theme?
- 70. Common theme? • Collect logs system-wide
- 71. Common theme? • Collect logs system-wide • Collect metrics system-wide
- 72. Common theme? • Collect logs system-wide • Collect metrics system-wide • Collect exceptions system-wide
- 73. Common theme? • Collect logs system-wide • Collect metrics system-wide • Collect exceptions system-wide • Trace requests (e.g. with Zipkin)
- 74. Common theme? • Collect logs system-wide • Collect metrics system-wide • Collect exceptions system-wide • Trace requests (e.g. with Zipkin) • Analyze
- 76. Common theme? • Collect logs system-wide • Collect metrics system-wide • Collect exceptions system-wide • Trace requests (e.g. with Zipkin) • Analyze • Sounds like something a structured platform can help with!
- 77. Distributed system debugging is a problem far from being solved.
- 79. Thank you
- 81. Thank you • @michaelklishin • github.com/michaelklishin
- 82. Thank you • @michaelklishin • github.com/michaelklishin • mklishin@pivotal.io