The good, the bad, the ugly side of step functions
- 1. The Good, The Bad and The Ugly side of Step Functions Mohsiur Rahman, Software Engineer at Comcast
- 2. Serverless on Amazon Web Services (AWS) ● Compute Technologies - Lambda, Lambda@Edge, Fargate ● Databases - DynamoDB, Aurora Serverless ● API Proxy - API Gateway ● Interprocess Messaging - Simple Notification Service (SNS), Simple Queue Service (SQS) ● Analytics - Kinesis ● Orchestration - Step Functions
- 3. What is a Step Function? ● Workflows that are made up of a series of steps, with the output of one step acting as input into the next ● Allows to build and update apps quickly by coordinating multiple AWS services into serverless workflows ● Translates workflow into a state machine diagram that is easy to understand, easy to explain to others, and easy to change. ● Identify and fix problems promptly, by monitoring each execution step, Source: DAZN Tech
- 4. Different States in Step Functions ● Wait State ● Parallel State ● Retry State ● Catch Failure State ● Choice State
- 5. When should you use Step Functions and Why?
- 6. When N is a known number ● Automation of Tasks run daily ● Batch processing jobs ○ Required 24/7 monitoring. ○ Resulted in costs increasing exponentially. ○ Bugs in AWS Simple Workflow (SWF) that could not be solved by AWS Support Engineers. ○ Had over 1000 instances running at its peak. ○ Loss of documents being processed in a given SLA i.e impacting business. ● Modernize a monolithic code base as a series of microservice steps.
- 7. By the end of December costs were at ~$4000 with predicted costs for January being lower.
- 8. Impact of using Step Functions ● Costs went down by 70% within two months. ● Conversion of existing architecture was done easily with minimal changes to codebase. ● Allowed new features to be added during conversion. ● Time spent by developers monitoring decreased to less than 10 hours. ● Maintaining stateless codebase. The benefits of serverless
- 9. Why not to use Step Functions?
- 10. When N is unknown/large Where n is the number of invocations ● Step Functions charges per transition, plus Lambda costs. ● Example shows 8 state transitions ● For 1,000,000 invocations the cost would be 1000 * 8 * 0.0250 = $200 + Lambda costs ● At comcast our case involved receiving over a million invocations per minute, resulting in an ~$1400 bill in 2 weeks.
- 11. How to still be serverless without using Step Functions ● Lambdas can call another lambda asynchronously. ● Thus mitigating costs to only Lambda pricing model. ● Lambda costs equalled to ~500$ per month compared to Step Functions running for 2 weeks at ~$1400. ● Lack of visualization is given up in this process on where it is failing. ● Lambdas are stateful, and requires frequent code changes if new feature is added.
- 12. Q&A Social Media LinkedIn : linkedin.com/in/mohsiur/ GitHub : github.com/mohsiur Medium : medium.com/@mohsiurr