Building a serverless integration platform
- 1. Building aServerless Integration Platform Robert O Grady Cloud Platform Lead
- 2. Where did Yellow’sCloud Journey Begin. CONTAINER PLATFORM STABLE WHY CHANGE ? $$ OPERATIONAL OVERHEAD INVESTIGATEA NEW SOLUTION
- 3. Serverless is .. It means different things to different people Finance – Cost saving (pay as you go model) Marketing –Speed to market, POC’s, Fail Fast. Technology – Foster Innovation in a quickly evolving space
- 4. Objective Create an Integration platform to process incoming messages and events from external sources, process them, transform them, and enrich other internal and external systems.
- 5. Initial Challenges Company Buy-In to Serverless Culture Change - break down Silo’s NewWay of working –Traditional roles would change Upskilling people with serverless Time/Effort (Project timelines do not change, technologies do)
- 7. How do we build this ?
- 8. Cloudformation Infrastructure as code – Resources as YAML Single point of control/track changes. Deploy with ansible modules Build and destroy* stacks at ease Custom Resources (lambda) Roll out additional environments (e.g DR) Gain an in dept knowledge of the AWS Service and API
- 9. APIGateway Core of all things serverless in events inYellow. Secure Entry point intoYellow’s Environment’s from our third parties Centralized Logging Call Step Function Directly not Lambda functions VTL templates (DataTransformation) – Can be used as a form of security Define API’s with the Swagger framework. Build your API through the console and export it.
- 11. Step Functions Coordinate AWS Services into workflows. Visual – Easy to see failure in a string of services Logs the state of each step with Input/Out Put and Exception fields
- 12. Step Functions
- 13. Step Functions
- 14. Function as a Service (Lambda) Small unit of code - Many reads one write Idempotency – Ability to rerun failures Stateless – DynamoDB/S3 Scalable – concurrently, memory X-Ray –End to End DistributedTracing System
- 15. Step Functions/ Lambda/ APIGateway Powerful Combination Retry Lambda Function for third party integrations in the event of failure Unique execution ID Ability to add other services and call them directly from the Step Function. (DynamoDB, Elastic Search, Glue, ECS and more)
- 16. Cloudwatch Events Communication Mechanism for Events in AWS –The unsung hero Importance of event Structure and consistency – Information in the event gives Lambda/SF context Two different event categories – Something that has happened (immutable) and something that should happen Event Recorder to Dynamo DB and steamed to ES Code to call Specific Events
- 19. Demo Create an Integration platform to process incoming messages and events from external sources, process them, transform them, and enrich other internal and external systems.
- 20. Deployments Concept of Domains (Service) Unit of deployment Small code changes CI/CD – Codebuild/Codepipeline Docker
- 21. What Next
- 22. AppSync GraphQL for efficient data query and manipulation RealTime – Self Service Portal Authentication - Cognito VTL Language Data sources Support/Pipeline Resolver Query/Mutations/Subscriptions
- 23. Operational Challenges Managing failure Monitoring a distributed system (Single pane of Glass is a challenge) Development teams more Ops focused Cloudwatch logs can be difficult to use.
- 25. Yellow Learnings
- 26. Thanks Yellow Are Currently Hiring and looking to expand Questions
Editor's Notes
- #5: Yellow is going through a digital transformation at the moment and it was decided that we should start to consume other external services from third parties to deliver functionality instead of trying to create everything ourselves. We had spent years building our own systems and spend a lot of money trying to maintain them.When something falls over, everyone is pulled from whatever project they are working on to solve operation issues. Double edged sword. In order for Yellow to move quickly, we decided we would stop doing this and leverage third parties that already had what we needed, we just needed access to them. It was decided that we should build an integration platform that would manage the interaction between yellow and third parties to manage that information flow. It needed to be secure and reliable .. So what would it look like ?