AWS SQS for better architecture
11 likes21,831 views
The document discusses AWS SQS (Simple Queue Service) and its role in solving problems related to job scheduling and resource management in applications. It explains how SQS facilitates a producer-consumer model, provides architectural patterns for utilizing SQS, and highlights common mistakes in its implementation. Additionally, it presents example architectures and usage guidance for effectively leveraging SQS in cloud-based applications.
AWS SQS for better architecture
- 1. AWS SQS for better architecture 23rd Dec'14 Saurabh S Bangad
- 2. Contents -How did SQS solve a problem? -Usage guidance -Architectural patterns -Example architectures -Mistakes -Example cases -References
- 3. How did SQS solve a problem? “My app has several jobs running based on a cron setting. That is, some run every 15 minutes, some once an hour, some once a day. These are jobs that affect our RDS data and our S3 resources, which are shared amongst all our ec2 instances such that each job only needs to be run by one machine. Running the same job multiple times is at best wasteful, and at worst could send out multiple emails to the same person. So I want to be able to auto-scale to multiple ec2 instances, yet still have these jobs only run by one machine. But because each auto-scaled instance must use the same ami and run the same application, it is not easy to configure which particular machine runs which jobs.”
- 4. How did SQS solve a problem? "My app has several jobs running based on a cron setting." => ???? "That is, some run every 15 minutes, some once an hour, some once a day." => Randomness "These are jobs that affect our RDS data and our S3 resources, : => Transactional
- 5. How did SQS solve a problem? "These are jobs that affect our RDS data and our S3 resources, which are shared amongst all our ec2 instances such that each job only needs to be run by one machine." => ????
- 6. How did SQS solve a problem? "Running the same job multiple times is at best wasteful, and at worst could send out multiple emails to the same person." => Of course! Just make use of an EC2 instance and you are all set.
- 7. How did SQS solve a problem? "So I want to be able to auto- scale to multiple ec2 instances" => AutoScaling! I like that, but instances don't talk to each other. "yet still have these jobs only run by one machine" => Sorry ?
- 8. How did SQS solve a problem? "But because each auto-scaled instance must use the same ami and run the same application, it is not easy to configure which particular machine runs which jobs." => I know..
- 9. How did SQS solve a problem? -This is an SQS session, so obviously the answer is SQS, but how would you solve this problem otherwise ? -Problem : This is not a client-server model -How would the instances synchronize ? -How about a protocol for decision making in a distributed fashion ? Too much of research!
- 11. Usage Guidance "My app has several jobs running based on a cron setting." => The Producer "These are jobs that affect our RDS data and our S3 resources, which are shared amongst all our ec2 instances such that each job only needs to be run by one machine." => The Consumer *Do think about it after the session.*
- 12. Usage Guidance Processing of a message in SQS P: Producer m:Message C: Consumer Q:Queue Story/Flow : 1)P1 sends m1 to Q1 & P2 sends m2,m3 to the Q1 2)Q1={m1,m2,m3} //unordered set 3)C1 requests a message 4)Q1 returns one of the messages 5)m2 is received by the C1 6)C1 processes m2 7)C1 deletes m2 from the queue 8)Q1={m1,m3} 9)C2 requests a message 10)...story continues...
- 13. Usage Guidance What will happen if the two consumers request at the same time ? => The Visibility timeout
- 14. Usage Guidance What will happen when the consumer that polled the message dies/forgets ? => Expired visibility timeout
- 15. Usage Guidance What if there are no messages in the queue for a longer duration? It would waste resources looking up continuously and getting empty responses. This does affect the billing as well. =>sleep on the consumer side =>long polling while receiving The wait time
- 16. Architectural patterns Characteristics of SQS : -Buffer reliably -Scalable -Managed/Queue-aaS -Secured -Highly available -SIMPLE-QS HA in real sense : (Queue==Que==Q)LOL
- 17. Architectural patterns Where can it be used ? -Asynchronous functioning -Loosely coupled components -Stateless system -Failure resilient system
- 18. Example architecture Chained processing e.g. Intermediate code generation between multiple teams and/or components of the system
- 19. Example architecture Priority queuing with SQS e.g. meeting SLAs based on the type of a customer Another approach : With a single queue?
- 20. Example architecture File sharing Possible approaches : 1)Producer AutoScaling group can upload the files to S3 and share the URLs over the queue. 2)S3 events can be pushed directly to SQS 3)If the file size is upto 256KB, you can just share it as a message.
- 21. Architectural patterns/example Publisher-Subscriber model e.g. independent teams using part of the message TIMESTAMP Order Shipping-Type 03/03/14:10:10GMT HeadphonesX Two Days Paid 03/03/14:10:10GMT ProductY Three Days Free A team uses timestamp+shipping, the other ignores that
- 22. Example architecture Automation by signaling e.g. Polling SQS for Job Status / Event driven when Elastic Transcoder publishes status messages to SNS, they ultimately get buffered to the Queue. When a consumer poll the message, it can resume the pending operations
- 23. Example architecture Email confirmation e.g. Sign up of new user where user would be sent an email to confirm which doesn't happen interactively
- 24. Example architecture SES bounces and complains
- 25. Example architecture Users talking directly to the SQS which is backed by DynamoDB Mobile app
- 26. Mistakes -Can SQS send message to .. ? NO! -Is SQS FIFO? NO! -Not deleting the messages -visibility timeout=0 -visibility timeout=too high -Using standard/short polling -Not polling enough -Retention period=too low -Synchronous/interactive jobs -Producer=consumer(same machine)
- 27. Example Cases -How do I deal with the at least once delivery? -How do I deploy the new code to prod with the changes in the message format? -Our website will start splittting the traffic between two ELBs, one for actual content and other for logging, we need to get the ELB prewarmed to support logging. -Where did that message go?
- 29. Thanks a lot! Any Questions?