Drone CI - Container native continuous Integration / Delivery
- 1. Drone 13.08.2018 Patrick Jahns Container native continuous Integration / Delivery
- 2. 2 • Software Engineer • ❤ Clean Code // DevOps // SRE • Solutions Architect at ownCloud patrick_jahns patrickjahns mail@patrickjahns.de
- 3. 3
- 4. 4 Drone CI
- 5. 5 Drone CI • Open Source (https://github.com/drone) - started in 2014 • Easy to install & maintain (docker pull drone/drone) • Simple YAML Configuration (superset of docker-compose.yml) • Container native CI/CD platform (everything runs within containers) • Isolated builds ( workspace / network ) • Integrates with several VCS Providers ( Github, Gitlab, Bitbucket, Gitea … ) • Rich set of official plugins (any container can be a plugin) • Management via cli / REST Api ( but also has a WebUI ) • Execute locally with “drone exec” • Provides prometheus metrics
- 7. 7 Drone - Concepts • Pipeline – Steps to be executed • Workspace – Ephemeral – Shared between individual pipeline steps • Services – Containers that are available during the pipeline execution – Yaml key == hostname of service • Conditions – Ability to specify conditions when a pipeline/step will be executed Basic Concepts Examples: https://github.com/patrickjahns/drone-examples
- 8. 8 Drone - Concepts Groups • Shared workspace • Run on the same Agent Matrix Builds • Completely seperated builds • Can be scheduled on different Agents Parallel builts // Concurrency Examples: https://github.com/patrickjahns/drone-examples
- 9. 9 Drone - Concepts • Secrets are injected as environment variables • Ability to limit secret per build event ( push / pull-request / tag ) • Ability to limit secrets to specific images/plugins • Secrets can be stored in various backends ( database / “global secrets file”* / vault* ) • For public repositories –> gated builds (changes to .drone.yml need approval ) Secrets *) requires enterprise subscription Examples: https://github.com/patrickjahns/drone-examples
- 10. 10 Drone - Concepts • Plugins are “just” containers => complete freedom and language agnostic • Large ecosystem of readily available plugins => http://plugins.drone.io/ • Anything missing => easy to build your own plugin – Example from ownCloud Plugins
- 11. 11 Drone - Concepts • Run your CI pipeline locally • Ability to debug // introspect if anything fails on the remote server • Example: Drone Exec
- 12. Drone@AWS
- 13. 13 Drone@AWS • Easy Deployment – Instance where drone server runs ( EC2 / ECS / Kubernetes ) – Database Server ( RDS ? ) – Instance(s) where drone Agents run • Various Plugins for direct interaction with AWS services available – Example: AWS Cloudformation, AWS ECR, AWS ECS, AWS Elastic Beanstalk, AWS Lambda more at http://plugins.drone.io/ – Most plugins require secret and access_key => currently not possible to use IAM Profiles
- 14. 14 Drone@AWS Publishing Container to ECR • plugins/ecr can publish to ECR via a ACCESS_KEY & Secret Using Images from ECR in drone pipeline • Global Registry File *) • Periodic Job Running on Agent Host Instance ( aws ecr get-login --no-include-email ) • Pipeline step with mounted docker-socket that logs into aws ecr and pulls the image *) requires enterprise subscription http://docs.drone.io/setup-global-registry-credentials/#elastic-container-registry-ecr Working with AWS ECR
- 15. Scaling DroneCI
- 16. 16 Scaling Drone - Autoscaler • CI/CD volumes are highly dependant on ”daily-business” • Fast feedback cycles are required for developer velocity • CI/CD requires compute power -> easily scaleable with cloud providers
- 17. 17 Scaling Drone - Autoscaler • Support for AWS, DigitalOcean, Google, HetznerCloud • Planned to support Azure, Packet.net, Scaleway • Simple service connected to Drone server • Hooked into Drone CLI, e.g. “drone server create” • Checks the Drone queue in a loop • Launch servers based on a cloud-init config • Start Drone agent via remote Docker connection (secured by TLS) • Unregister Drone agent if not needed anymore • Destroy server instance after a minimal amount of time • Provides Prometheus metrics
- 18. 18 Server Autoscaler Agent Agent Agent Agent Scaling Drone - Autoscaler
- 19. 19 Scaling Drone – Autoscaler@AWS • Requirements: – VPC, Subnet, Security Group – EC2 / ECS where autoscaler can be additionally started • Caveats – Agents either need to have a public-ip associate // or a NAT needs to be defined – It is currently limited to On Demand instances – Pull Request for Spot Instances will follow – Currently not yet possible to attach instance roles
- 20. 20
- 21. 21 0 5000 10000 15000 20000 25000 30000 35000 Nov 6-Nov 11-Nov 16-Nov 21-Nov 26-Nov Dec 5-Dec 12-Dec 17-Dec 22-Dec 27-Dec Jan 5-Jan 11-Jan 16-Jan 21-Jan 26-Jan 31-Jan 4-Feb 9-Feb 14-Feb 19-Feb 24-Feb Mar 5-Mar 10-Mar 15-Mar 20-Mar 25-Mar 30-Mar 3-Apr 8-Apr 13-Apr 18-Apr 26-Apr 1-May 7-May 12-May 17-May 22-May 28-May 1-Jun 6-Jun 11-Jun 16-Jun commulated runtime time to finish time to finish (including queue wait)
- 22. DroneCI Outlook
- 23. 23 Drone - Outlook • Official support for Arm / Arm64 • Experimental support for windows containers • Support for multi-machine fan-in / fan-out • Support for multi-machine, multi-architecture • Hooks (http/REST) to override default behavior i.e. secrets, registry credentials, approval workflows – Example for AWS – Trigger a Lambda Function to fetch secrets from AWS SecretsManager / ParameterStore RoadMap / Topics for drone 0.9
- 24. 24 Links Slides & SourceCode • https://github.com/patrickjahns/awsmeetup-20180813-drone • https://github.com/patrickjahns/drone-examples Drone • Drone Github – https://github.com/drone • Drone documentation - http://docs.drone.io/ • Autoscaler - https://github.com/drone/autoscaler • Autoscaler documentation - https://autoscale.drone.io/ • Official Drone plugins - http://plugins.drone.io/ • Drone community - https://discourse.drone.io/
- 25. Thank You!