Autoscaled Github Runners using StepFunctions
- 1. GitLab Runners auto-scaled with StepFunctions AWS User Group Stockholm 2022-05-31
- 2. GitLab Runners auto-scaled with StepFunctions AWS User Group Stockholm 2022-05-31
- 3. Hello, I’m JIMMY DAHLQVIST Head of AWS @ Sigma Technology Cloud AWS Ambassador & Community Builder
- 4. @jimmydahlqvist Why self hosted runners?
- 5. @jimmydahlqvist Reasons for ◦ Long lived IAM Credentials ▫ Oauth is now supported ◦ Limits in CI/CD minutes ◦ Network requirements
- 6. @jimmydahlqvist Reasons against ◦ Operational Overhead ◦ Runner bottlenecks ▫ All runners are busy
- 10. @jimmydahlqvist How it works ◦ Webhooks add message to EventBridge ◦ EventBridge invoke StepFunction
- 12. @jimmydahlqvist Why StepFunction ◦ Several tasks too perform ◦ Need to be coordinated
- 14. @jimmydahlqvist
- 15. @jimmydahlqvist StepFunction Logic ◦ Check Cooldown ◦ Determine needed scaling action ◦ Perform Scaling ▫ Add / Remove runners
- 16. @jimmydahlqvist
- 19. @jimmydahlqvist Definition ◦ Single Unit of work ◦ All work is done by tasks
- 20. @jimmydahlqvist
- 21. @jimmydahlqvist Service / SDK Integration ◦ 15+ service integrations ◦ 200+ SDK integrations ◦ Specified in Resource
- 22. @jimmydahlqvist Lambda Integration ◦ Invokes Lambda function directly ◦ Specify the function name ◦ arn:aws:lambda:region:account:function: function_name
- 23. @jimmydahlqvist Service Integration ◦ Calls service directly ◦ Specify the Service and API ◦ arn:aws:states:region:account:servicenam e:APIname
- 24. @jimmydahlqvist SDK Integration ◦ Calls the service using SDK ◦ Specify Service and apiAction ◦ arn:aws:states:::aws-sdk: serviceName:apiAction
- 25. @jimmydahlqvist Retry ◦ Powerful error handling ◦ Supported in Task and Parallel ◦ Specify backoff, max attempts, intervals, and specific errors
- 27. @jimmydahlqvist Catch ◦ Powerful error handling ◦ Supported in Task, Map, and Parallel ◦ Specify an error handling task
- 29. @jimmydahlqvist Input and Output Processing ◦ Manipulate the input and output ◦ Select only what you need
- 30. @jimmydahlqvist ResultPath ◦ We can: ▫ Replace input with Result ▫ Discard Result ▫ Include Result with Input ▫ Update a Node in Input
- 31. @jimmydahlqvist Replace input with Result
- 33. @jimmydahlqvist Include Result with Input
- 34. @jimmydahlqvist Update a Node in Input
- 36. @jimmydahlqvist Overview ◦ Controls the flow in the state machine ◦ Powerful comparison operations ◦ Support catch all else (Default) path
- 37. @jimmydahlqvist
- 38. @jimmydahlqvist String comparison ◦ StringEquals ◦ StringGreaterThan ◦ StringMatches ◦ IsString ◦ Etc
- 39. @jimmydahlqvist Extended logic ◦ Support AND, OR, NOT operators
- 40. @jimmydahlqvist Variable comparison ◦ Compare with a variable in the event
- 41. @jimmydahlqvist String matches ◦ Support wildcards
- 42. @jimmydahlqvist Boolean and Numeric ◦ Supports same logic both Boolean and Numeric
- 44. @jimmydahlqvist Pass ◦ State that will just be passed through ◦ Use when END is not directly supported
- 45. @jimmydahlqvist
- 46. @jimmydahlqvist Wait ◦ Specified number of seconds ◦ Timestamp has passed ◦ Can be a variable in the event
- 48. @jimmydahlqvist Run it in parallel ◦ Runs several Tasks at the same time ◦ Collect data in parallel
- 50. @jimmydahlqvist
- 51. @jimmydahlqvist
- 54. Thank You!! @jimmydahlqvist https://www.linkedin.com/in/dahlqvistjimmy/ https://jimmydqv.com | https://dev.to/jimmydqv
Editor's Notes
- #25: Super powerful! Before you had to use a lambda to do a simple SDK call!!
- #26: USE IT!!!! Backoff is a very good practice! Erros like: TaskFailed, Runtime, HeartbeatTimeout, BranchFailed and many more!
- #28: USE IT!!!! Handle the errors that got away….
- #29: Combination of retry and catch DESCRIBE WHAT HAPPENS IN DIFFERENT SCENARIOS!!!!!
- #32: THIS IS DEFAULT AS WELL!!!
- #55: Thank you! You can follow me on twitter or connect on linked in.