Serverless Workflows on AWS - A Journey from SWF to Step Functions
Download as PPTX, PDF8 likes3,476 views
The document discusses the limitations of AWS Lambda functions, which run for a maximum of five minutes, and the challenges of deploying enterprise applications with complex workflows. It explores the use of Amazon SWF and, eventually, AWS Step Functions as solutions for orchestration and workflow management while emphasizing the advantages and disadvantages of each method. The author encourages leveraging AWS Step Functions for building efficient workflows in serverless architectures.
Serverless Workflows on AWS - A Journey from SWF to Step Functions
- 1. AWS Lambda functions can only run for a maximum of five minutes. This must be distinctly understood, or nothing wonderful can come of the story you are about to hear.
- 2. FORREST BRAZEAL Cloud Services team lead at Infor AWS automation tooling Blog: forrestbrazeal.com Twitter: @forrestbrazeal
- 3. SERVERLESS WORKFLOWS IN AWS A JOURNEY FROM SWF TO STEP FUNCTIONS FORREST BRAZEAL
- 4. THE PROBLEM (MAY 2016) Enterprise apps with many moving parts Deployment orchestration becomes tricky Need a workflow system
- 5. WHAT DOES A WORKFLOW SYSTEM DO? Coordinates worker processes Manages state Responds to interrupts Is highly available, elastic, and all that good stuff
- 6. WHY SERVERLESS FOR THIS USE CASE? Lots of downtime Need to scale wide Internal tooling, need low cost/maintenance It’s cool! (Famous last words)
- 7. How do you manage a long-running process using short-lived, stateless functions? THE BIG QUESTION
- 8. IDEA 1: PURE LAMBDA
- 9. GHASTLY!
- 10. BUT WAIT A MINUTE AWS has had a workflow service, Amazon SWF, for several years. Could we use it?
- 11. WE NEED TO TALK ABOUT SWF (not so) “Simple Workflow Service” You write a decider program that orchestrates your workflow You write activity worker programs to handle tasks SWF is the glue
- 13. IDEA 2: SWF + LAMBDA Put the “decider” program on Lambda Find some way to invoke it repeatedly over the life of the workflow Advantages • SWF manages the workflow • More maintainable than Possibility 1 • Still technically “serverless”! Disadvantages • Let’s see …
- 19. THIS (WAS) MY ARCHITECTURE AWS even made a video about it!
- 20. SERVERLESS SWF PAIN POINTS ☹ Latency ☹ Cost ☹ Retries ☹ Debugging ☹ State
- 23. AWS STEP FUNCTIONS (SFN) Announced at re:Invent 2016 A “serverless-native” workflow solution State machine as a service
- 24. THE FIRST RULE OF STEP FUNCTIONS CLUB Your Lambda functions don’t talk about Step Functions Club! They just accept input and output – SFN does the orchestration “Amazon States Language” template defines the state machine in JSON
- 25. SFN IN ACTION: DYNAMIC BACKOFF You pay here Not here!
- 26. DYNAMIC BACKOFF
- 27. DYNAMIC BACKOFF In Lambda function for “BackoffTaskState”
- 28. From this …
- 29. …to this!
- 30. STEP FUNCTIONS WISHLIST CloudFormation support ✔ > 32kb state size Updates Signals Child state machines Dynamic parallel states
- 34. IN CONCLUSION Recognize when your serverless reach exceeds your grasp Your use cases make the tooling better Go build an awesome workflow with AWS Step Functions!
- 35. RESOURCES More details about our SFN use case and results: https://forrestbrazeal.com/2016/12/29/serverless-workflows-on-aws-my-journey- from-swf-to-step-functions/ AWS “This Is My Architecture” video about the SWF/Lambda solution: https://youtu.be/rKeS3RpMEOw Infor’s re:Invent session about deployment automation: https://www.youtube.com/watch?v=Epx_32c3c6s
Editor's Notes
- #21: Latency. CloudWatch rules cannot run more frequently than once a minute, and SWF scheduling delays made it advisable to invoke the decider even less frequently than that. This situation created significant latency between workflow actions, leading to inflated workflow times that were especially noticeable for workflows involving lots of short tasks. Cost. Having to run the decider on a two-minute loop throughout the life of the workflow somewhat negated the cost advantages we hoped to get from using Lambda instead of EC2 in the first place, especially as our number of workflows scaled up. Runtime State. Every time the Lambda decider function ran, it had to figure out where it was in the workflow process. SWF is supposed to make stateless execution easy, and it provides the complete deployment workflow history as a JSON blob when handing tasks to the decider, but the blob quickly becomes unmanageably large and filled with superfluous data. To keep track of what was going on in the workflows, we resorted to maintaining ephemeral state in a DynamoDB table, adding more latency and cost. Retries/Error Handling. SWF, I regret to say, has bugs. Sometimes SWF completely fails to schedule a task, or does it so late that the workflow’s task timeout expires. Finding and catching these errors required even more state maintained outside of SWF. Debugging. The SWF console’s workflow event views are difficult to read, oddly paginated and don’t provide much information, leading to a rabbit trail of log searches anytime something went wrong. Code Maintainability. The combination of multiple state sources and ramifying failure scenarios, not to mention the central “hack” of running the SWF decider in a loop, led to a mess of one-off fixes and hacky workarounds in our codebase.