Lessons Learned from Building a Serverless Notifications System.pdf
- 1. @SrushithR @SrushithR Srushith Repakula Head of Engineering, KonfHub AWS Serverless Hero Lessons Learned from Building a Serverless Notifications System
- 2. @SrushithR @SrushithR Srushith Repakula AWS Serverless Hero Head of Engineering at KonfHub Serverless Consultant AWS UG Tirupati Lead Co-organiser of Serverless Group Host of ServerlessSaturdays
- 3. @SrushithR Agenda 1. What did we try to solve? 2. How did we solve it? 3. Why Serverless? 4. Problems, challenges encountered and how we conquered it 5. Unconquered challenges
- 4. @SrushithR ● A highly scalable notification system that can deliver notifications to hundreds of thousands of recipients ● Should work for multiple channels - Email, SMS and WhatsApp ● Should support instant launch and scheduled campaigns ● Handle duplicates ● Real time analytics on deliverability, open, and click rates What did we try to solve?
- 5. @SrushithR How did we solve it? Amazon API Gateway Amazon Simple Queue Service (Amazon SQS) AWS Lambda Amazon EventBridge Amazon Simple Email Service (Amazon SES) Amazon Simple Notification Service (Amazon SNS) AWS Lambda Amazon Cognito Amazon DynamoDB Amazon Aurora Webhooks
- 6. @SrushithR AWS Services used ● Amazon API Gateway ● Amazon Cognito ● AWS Lambda ● Amazon DynamoDB ● Amazon SQS ● Amazon Aurora ● Amazon EventBridge ● Amazon SES ● Amazon SNS
- 7. @SrushithR
- 8. @SrushithR Problems and Challenges ● Deliverability of emails and domain reputation ● SQS - FIFO queue vs standard queue ● Throttling limits of the notification service providers. For example, number of emails sent per second and per day ● Scalability, cost and time to finish the campaigns ● Lambda function’s max timeout of 15 minutes ● Scheduling of campaigns every minute
- 9. @SrushithR Deliverability of emails and domain reputation Dedicated IP Each consumer uses a dedicated IP address set for sending emails. For example, one dedicated IP for transactional, one for marketing, etc. Shared IP Multiple consumers use the same IP set for sending emails Pic credit:https://postmarkapp.com/guides/dedicated-vs-shared-ips-for-email-when-to-use-each
- 10. @SrushithR Dedicated IP Addresses for Amazon SES Standard Refers to dedicated IP addresses that you manually set up and manage, including the option to manually warm them up and scale them out, and to manually move them in and out of IP pools Managed Refers to dedicated IP addresses that are automatically set up. SES automatically warm up for each ISP individually and auto-scale based on your sending volume Bring your own IP addresses Makes it possible to use your own IP addresses to send email through SES Helpful when there is a positive IP reputation built using an in-house email sending system, but want to migrate to Amazon SES
- 11. @SrushithR SQS - FIFO queue vs standard queue Pic credit:https://www.bitslovers.com/sqs-fifo/
- 12. @SrushithR SQS - FIFO queue vs standard queue: The case for FIFO ● Handling of duplicates and order ● Easier to understand the status of a campaign ● Unbatched throughput - 300 ● Batched transactions per second FIFO - 3000 ● Batched transactions per second for high throughput FIFO - 70,000* ● In flight messages per queue - 20,000
- 13. @SrushithR ● In flight messages per queue - 120,000 ● Faster to process since there is no order to be maintained ● Cheaper in comparison to FIFO SQS - FIFO queue vs standard queue: The case for Standard
- 14. @SrushithR Although Standard queue increase application layer complexity like ● Implementation of idempotency ● Extra processing for fetching the campaign status etc, Standard was the choice because: ● FIFO has only a 5-minute deduplication interval ● Slower when hundreds of messages are published within the same group SQS - FIFO queue vs standard queue: Standard wins!
- 15. @SrushithR Throttling limits are everywhere! ● Daily sending quota - the maximum number of emails that you can send in a 24-hour period ● Maximum send rate - the maximum number of emails that Amazon SES can accept from the account each second. One can exceed this quota for short bursts, but not for sustained periods of time. Throttling limits of the notification service providers
- 16. @SrushithR Conquering the throttling limits - Reserved Concurrency ● The concurrency limit is reserved just for the lambda function ensuring that there are execution environments available and not shared with the account level settings ● The maximum number of execution environment instances ● Throttled when the reserved concurrency limit is reached
- 17. @SrushithR Conquering the throttling limits - Maximum concurrency for SQS
- 18. @SrushithR Scheduling of campaigns every minute The Challenges: ● Campaigns can be scheduled to the minute ● Scalability of the scheduler service - hundreds and thousands of campaigns can be scheduled
- 19. @SrushithR Scheduling of campaigns every minute The Challenges: ● Campaigns can be scheduled to the minute ● Scalability of the scheduler service - hundreds and thousands of campaigns can be scheduled Common solution - run a cron job every minute to pick up campaigns that are scheduled Problems with this solution: ● Unnecessary costs involved in empty Lambda function invocations ● Increases application complexity
- 20. @SrushithR Scheduling of campaigns every minute
- 21. @SrushithR Unconquered Challenges ● Automated end-to-end testing of the entire architecture ● Observability and monitoring ● Automatic handling of service provider limits for “Bring your own provider” use case