How to choose the right messaging service for your workload
0 likes91 views
The document provides guidance on selecting the appropriate AWS messaging service based on workload requirements, focusing on event-driven architecture and the various service options like Kinesis, EventBridge, SNS, and SQS. It highlights key considerations such as scaling constraints, cost efficiency, error handling, and observability in messaging systems. Additionally, it outlines the differences between task and event-driven messaging and the importance of understanding trade-offs in service capabilities.
How to choose the right messaging service for your workload
- 1. How to choose the right messaging service for your workload
- 6. Yan Cui http://theburningmonk.com @theburningmonk Independent Consultant advise training delivery
- 8. AWS services are the new data structures. Understanding their trade-offs is the new big O notation.
- 13. EventBridge
- 17. event
- 19. event FIFO Archiving Replays Schema Discovery Scalability Cost Performance Debugging Error Handling
- 21. picking the right AWS service is a valuable skill
- 23. Scaling
- 25. What are the scaling constraint for the messaging service?
- 26. Google “{service name} quotas”
- 30. How does Lambda’s concurrency scale with throughput?
- 38. more concurrency is not always better…
- 39. if you want… maximum throughput precise control over throughput SNS EventBridge Kinesis Provisioned
- 40. if you want… maximum throughput precise control over throughput SNS EventBridge Kinesis Provisioned
- 41. Downstream System SNS Lambda use Reserved Concurrency to limit concurrency
- 42. Reserved Concurrency are taken out of the available regional Lambda concurrency
- 43. managing Reserved Concurrency for many functions is dif fi cult and error prone, easy to create more problems than it solves
- 44. Costs
- 45. always factor scale into the equation
- 46. $10.836 1 msg/s for a month, 1KB per msg 1 x 60s x 60m x 24hr x 30days @ $0.014 per mil + 24hrs x 30days @ $0.015 per shard per hr $2.592 SNS SQS EventBridge Kinesis 1 x 60s x 60m x 24hr x 30days @ $1.00 per mil 1 x 60s x 60m x 24hr x 30days @ $0.40 per mil 1 x 60s x 60m x 24hr x 30days @ $0.50 per mil $1.037 $1.296
- 47. Kinesis on-demand mode pricing
- 48. $10.836 1 msg/s for a month, 1KB per msg 1 x 60s x 60m x 24hr x 30days @ $0.014 per mil + 24hrs x 30days @ $0.015 per shard per hr $2.592 SNS SQS EventBridge Kinesis Provisioned 1 x 60s x 60m x 24hr x 30days @ $1.00 per mil 1 x 60s x 60m x 24hr x 30days @ $0.40 per mil 1 x 60s x 60m x 24hr x 30days @ $0.50 per mil $1.037 $1.296 Kinesis On-Demand 1kb x 60s x 60m x 24hr x 30days @ $0.08 per GB ingested + 24hrs x 30days @ $0.04 per stream per hr $28.998
- 49. $47.088 1,000 msg/s for a month, 1KB per msg 1000 x 60s x 60m x 24hr x 30days @ $0.014 per mil + 24hrs x 30days @ $0.015 per shard per hr $2592.00 SNS SQS EventBridge Kinesis Provisioned 1000 x 60s x 60m x 24hr x 30days @ $1.00 per mil 1000 x 60s x 60m x 24hr x 30days @ $0.40 per mil 1000 x 60s x 60m x 24hr x 30days @ $0.50 per mil $1036.80 $1296.00 Kinesis On-Demand 1000kb x 60s x 60m x 24hr x 30days @ $0.08 per GB ingested + 24hrs x 30days @ $0.04 per stream per hr $226.55
- 50. services that charge by uptime are order(s) of magnitude cheaper when running at scale
- 51. National broadcaster of Finland
- 52. National broadcaster of Finland 500+ millions events per day, peaks at 600K+ messages/min
- 53. National broadcaster of Finland 500+ millions events per day, peaks at 600K+ messages/min Anahit Pogosova
- 54. National broadcaster of Finland 500+ millions events per day, peaks at 600K+ messages/min Anahit Pogosova decisions driven by cost ef fi ciency
- 55. National broadcaster of Finland 500+ millions events per day, peaks at 600K+ messages/min Anahit Pogosova
- 56. National broadcaster of Finland 500+ millions events per day, peaks at 600K+ messages/min Anahit Pogosova fallback in case Kinesis is down
- 57. Error Handling
- 59. 0, 1, 2
- 60. DLQs only capture the failed event
- 61. DLQ
- 68. prefer Lambda destination over DLQs (you can use both together, but there’s no clear reason for doing so)
- 72. DLQ
- 73. Observability
- 74. A measure of how well the internal state of an application can be inferred from its external outputs
- 75. X-Ray
- 78. X-Ray doesn’t trace through many popular messaging services.
- 79. X-Ray doesn’t trace through many popular messaging services. Need separate solution (e.g. correlation IDs) for Lambda logs.
- 81. all the indirect invocations are accounted for
- 83. all the Lambda logs from the transaction in chronological order
- 86. Step 1. Step 2.
- 87. No manual instrument. Better support for messaging services. Supports containers and Lambda.
- 88. “which messaging service should I use?”
- 90. task vs event
- 91. Task Event “Something happened” “Go do a thing”
- 92. Task Event “Something happened” “Go do a thing” Intended for a target receiver.
- 93. Task Event “Something happened” “Go do a thing” Intended for a target receiver. Often expects an answer.
- 94. Task Event “Something happened” “Go do a thing” Intended for a target receiver. Publishers are obvious to subscribers. Often expects an answer.
- 101. EventBridge SNS FIFO Do Stuff!
- 102. EventBridge SNS FIFO Do Stuff! order is lost
- 103. EventBridge SNS FIFO Do Stuff! order is lost
- 104. EventBridge SNS FIFO SQS FIFO Do Stuff (in order)! Other subscribers (not my problem!)
- 106. Kinesis Lambda
- 107. EventBridge Kinesis Lambda Other subscribers (not my problem!) send domain events
- 112. https://theburningmonk.com/hire-me Advise Training Delivery “Fundamentally, Yan has improved our team by increasing our ability to derive value from AWS and Lambda in particular.” Nick Blair Tech Lead