Individual Serverless Development Environments for AWS
0 likes102 views
The document details a presentation on implementing individual serverless development environments using AWS services at an IoT Systems meetup. It covers the challenges faced during development, the architecture used, and the solutions implemented, such as the use of CloudFormation for resource management and Travis CI for continuous integration. The session emphasizes the importance of security and the structured methodologies adopted to ensure streamlined and secure development practices.
Individual Serverless Development Environments for AWS
- 1. Individual Serverless Development Environments Copenhagen AWS User Group Meetup Søren Peter Nielsen & George Niculae, IoT Systems October 3rd, 2017
- 2. Agenda • Introduction − What does IoT Systems do? − Software Focus − Architecture − Built on AWS • Individual Serverless Development Environments − AWS Services used − Build process − The challenge − Considered aspects − Solution • Developer view • Infrastructure view − Q&A
- 3. • IoT Systems manufactures automation equipment for consumer electronics • Motion control software & Automation systems software(orchestrator) • Our cloud software solution “SenseQ” was originally built to support cloud connected automation
- 5. Our Team Takes IoT Security Seriously • Security By Design Security is not an after thought • Defense in Depth Layered security mechanisms assure that one security failure does not provide an attacker access to the full solution • Detailed Audit Trail Enables an Auditor to verify that the application is operated in a secure manner • Best practice security organization Making sure nothing falls “through the cracks” - All necessary Policies, Processes and Procedures are documented in our Information Management Security System https://iotsyst.com/security/
- 6. Individual Serverless Development Environments George Niculae – Head of DevOps • Agenda − AWS Services used − Build process − The challenge − Considered aspects − Solution • Developer view • Infrastructure view − Q&A
- 7. AWS Services - CloudFront - S3 - API Gateway - Lambda functions - DynamoDB - EC2 - Elasticsearch Service - AWS IoT - AWS Greengrass
- 8. Powered by IoT Systems Dashboard - CI - Code hosted in GitHub, private repository - Travis CI build on commit - Travis CI deploy dashboard application in AWS through serverless framework
- 9. Problem - Developers working on multiple branches at the same time with need to test integrated, closer to production setup
- 10. Problem
- 11. Considered aspects - Development branches could be or could not be compatible with each other (e.g. different database schemas, different payload of messages) - CI process is needed to be performed for each development branch - Transparent configuration of AWS resources for developers (for both setting up / tearing down development environment) - Minimum to no specific configuration should be performed by developers to set up their own AWS based environment - Reusing AWS account services and keep AWS resources at minimum usage (reuse where possible) is a goal
- 12. Solution
- 13. Solution - from developer point of view - Create feature branch and instruct Travis to build branch - Push branch upstream, Travis start building - Slack notification containing URL to access is received when build ends - Subsequent development cycles and tests take place - Delete git branch from github to tear down allocated resources
- 14. Powered by Solution - from developer point of view
- 15. Solution - Splitting CloudFront and S3 buckets - For each development branch a separate CloudFront distribution is created (using CloudFormation stack, through Travis) - For each development branch an S3 bucket named using iot- app.dashboard.{branch-name} pattern is created - CloudFront distribution is set up with S3 Origin to proper bucket
- 16. Powered by Solution - Splitting CloudFront and S3 buckets
- 17. Powered by Solution - Splitting API Gateways and Lambdas - For each development branch a separate API Gateway suffixed with branch name is created (using CloudFormation stack, through Travis) - Each branch has its own set of Lambda functions, suffixed with branch name (using serverless customName feature) - API Gateways are set up to forward request to proper Lambda functions - CloudFront distribution is set up with Custom Origin to API Gateway
- 18. Powered by Solution - Splitting API Gateways and Lambdas
- 19. Powered by Solution - Lambdas and Database Integration - For each development branch a separate set of DynamoDB tables suffixed with branch name is created (using CloudFormation stack, through Travis) - AWS IoT rules are configured with actions to invoke proper Lamda functions
- 20. Powered by Solution - Lambdas and Database Integration
- 21. Solution - Multiple Inventory Services - Inventory services share the same EC2 instance running Docker service - Travis builds a Docker image is created and properly labeled for each branch - Travis deploys Docker container and expose different port on host - Each branch gets its own API Gateway suffixed with branch name created
- 22. Powered by Solution - Multiple Inventory Services
- 23. Powered by Solution - Dashboard Overview
- 24. Powered by Solution - Inventory Service Overview
- 25. Powered by Solution - Tearing down resources - GitHub repository is set up with webhook to an API Gateway - API Gateway / Lambda function receives event from GitHub and tears down all allocated resources (programmatically and by deleting proper CloudFormation stack)
- 26. Experiences • Benefits − Releasing / merging in master branch each week only features that were tested close to production • ability to put on hold non finalized features until development ready − It was not a huge effort to develop this support − We now have several actively developed branches at the same time; − Easily adopted by developers − Each environment is easy to debug, • separated CloudWatch log streams for each environment / lambda function • CloudFront distribution taking long time to set up (about 20 minutes) − We increased default Travis CI time-out, Travis CI is configured to wait for CloudFront distribution to be set up (only for the first time environment is built) • We are continuously working to improve the implementation: − Looking into serverless-aws-alias project for using lambda aliases as an alternative to create new lambda functions − We have a mix of serverless deployment (through CloudFormation stacks) and scripting, looking into refine that – where it makes sense
- 27. Q&A