Heroku to Kubernetes & Gihub to Gitlab success story
- 1. Heroku / Github / CircleCI to Kubernetes and Gitlab (using Helm & Vault) Jérémy Wimsingues @jwimsingues March 2019 1
- 2. ● Context of the migration and overall infrastructure setup before migration ● GitHub → Gitlab ● Heroku → Kubernetes ● DevOps Improvements What will we talk about? 2
- 3. Marketplace connecting shippers with truckers Jérémy Wimsingues DevOps & Data engineer 2 years of Cloud Experience @jwimsingues Who are we? 3
- 4. Everoad’s infrastructure @September 2017 Frontend Backend Heroku MongoDB (atlas) Projection databases + Event store Elastic (cloud) RabbitMQ cloudamqp Pricing Google cloud platform LogEntries 4
- 5. GitHub + CircleCI to Gitlab, gitlab-ci & Kubernetes runners ● Easy for dev ● Efficiency ● Wait for a pipeline to be run ● Pay for each dev joining the team ● Switch between several tools ● Pain for dev to put in production on Kubernetes ● CI/CD integration with Kubernetes ● On demand test environment ● Focus the dev to what they do the best And… I had some experience with gitlab so… That was easy to sell to the CTO :-D 5
- 6. ● Create the gitlab server ○ Easy to do with a Bitnami deployment ○ A lot of operational stuff (https, create teams, …) ○ Enable runners to Kubernetes ● Create the first pipeline using .gitlab-ci.yml with a test app ○ Create a prepare stage (check if it compiles) ○ Create a test stage ○ Create a build (and push) stage (create the docker image and push it) ○ Deploy the code to Heroku staging ○ Force the pipeline promotion from staging to production ● Move all the repositories Roadmap 6
- 7. ● Basically, just follow the documentation… ● As an information my first gitlab server took me 10 minutes to run. It then took 2 days to setup the whole operational stuff (DNS domain, https, setup the google auth, create the team, pull the repos, setup the MR process, slack integration…) How to setup the server + runners inside Kubernetes 7 ● The easiest and fastest way is to use Bitnami release: GitLab CE Certified by Bitnami
- 8. ● Helm is the BEST PRACTICE you MUST FOLLOW when you want to use Kubernetes in production ● Helm is a release manager for Kubernetes. It “merges” all the Kubernetes object into a “release”. It “links” your Kubernetes object so you can versionize them and do one command upgrade and rollback Who knows Helm in the audience? /! NEVER USE KUBERNETES WITHOUT HELM /! 8
- 9. Setup runners inside Kubernetes 9
- 10. Setup runners inside Kubernetes 10
- 14. Everoad’s infrastructure @November 2017 Frontend Backend Heroku MongoDB (atlas) Projection databases + Event store Elastic (cloud) RabbitMQ cloudamqp Pricing Google cloud platform LogEntries 14
- 15. Heroku to Kubernetes ● Easy for dev to put in staging/production ● Configuration of env variables ● Review app ● Scale the infrastructure with $$$ ● Pass X hours to setup the new git for the new microservice ● Have splitted metrics ● Manage our infrastructure ● Manage our costs ● Scale with microservices ● On demand test environment 15
- 16. 1. Create a Kubernetes cluster 2. Install Helm ;-) 3. Setup Prometheus + Grafana to monitor the infrastructure 4. Setup a reverse proxy to spend less money (~$20/route for GCP w/o high traffic) ○ nginx-ingress-controller did the job for us 5. Prepare a Kubernetes directory with some yaml templates which are going to be present on each of your microservices: ○ One deployment ○ One service (which contains annotation to allow Prometheus to scrape the metrics) ○ If required, an ingress (which contains annotations to use nginx/other) ○ Configmap if required (we will talk/debate about secrets later) Roadmap 16
- 17. 1. Create a Kubernetes cluster 2. Install Helm ;-) 3. Setup Prometheus + Grafana to monitor the infrastructure 4. Setup a reverse proxy to spend less money (something like $15/20/route for GCP) ○ nginx-ingress-controller did the job for us 5. Prepare a Kubernetes directory with a few yaml template which are going to be present on each of your microservices: ○ One deployment ○ One service (which contains annotation to allow Prometheus to scrape the metrics) ○ If required, an ingress (which contains annotations to use nginx/other) ○ Configmap if required (we will talk/debate about secrets later) Roadmap 17
- 18. Everoad’s infrastructure @January 2018 (& live one!) 18 MongoDB (atlas) Projection databases + Event store Elastic (cloud) RabbitMQ cloudamqp Google cloud platform
- 19. Everoad Kubernetes infrastructure @January 2018 (& live one!) Belgique - europe-west1-b Belgique - europe-west1-c Belgique - europe-west1-d services storage réseau sur la zone storage réseau sur la zone storage réseau sur la zone Google cloud platform 19
- 20. Cluster Proxy Proxy Proxy Proxy Risoli Backend Frontend Backend Proxy Global infrastructure Namespaces: kube-system risoli frontend backend Users (clients) Admin kubectl Services External Load Balancer 20 Frontend Backend Risoli
- 21. A to Z user request 21 ingresses ... User service nginx DNS GLB app-service pod Kubernetes cluster - of the six nodes
- 22. Kubernetes staging cluster Continuous integration 22 homemade scripts Developer Gitlab - merge => staging daemon runner local tiler pod upgrade release <project>
- 23. Kubernetes staging cluster Continuous deployment 23 Developer homemade scripts Gitlab - job => production daemon runner get prod credentials Kubernetes production cluster local tiler pod upgrade release <project> configmap.. secret.. service.. ingress..
- 24. ● All the secrets are under a different path: secret/<ENV>/<SERVICE_NAME>/<LIST_OF_CONFIGURATION> ● The CI of the specific microservice is requesting all the secrets of this directory and create a Kubernetes secret ● If we want to rollback we will rollback secret with the release (useful) ● If we want to change the configuration, we change the secret and run another production pipeline (new helm release created) ● Done in bash (could have been done using your favourite language, there are just API calls on it DevOps improvements - Vault as a config manager 24
- 25. ● All the secrets are under a different path: secret/<ENV>/<SERVICE_NAME>/<LIST_OF_CONFIGURATION> ● The CI of the specific microservice is requesting all the secrets of this directory and create a Kubernetes secret ● If we want to rollback we will rollback secret with the release (useful) ● If we want to change the configuration, we change the secret and run another production pipeline (new helm release created) ● Done in bash (could have been done using your favourite language, there are just API calls on it DevOps improvements - Vault as a config manager 25
- 26. ● All the secrets are under a different path: secret/<ENV>/<SERVICE_NAME>/<LIST_OF_CONFIGURATION> ● The CI of the specific microservice is requesting all the secrets of this directory and create a Kubernetes secret ● If we want to rollback we will rollback secret with the release (useful) ● If we want to change the configuration, we change the secret and run another production pipeline (new helm release created) ● Done in bash (could have been done using your favourite language, there are just API calls on it DevOps improvements - Vault as a config manager 26
- 27. ● Based on Helm, we use the CI to deploy an ephemeral environment with all the micro services ● Automatic request a subdomain to add a new DNS entry (in fact we use a sub subdomain trick) ● Available port forward for the devs to see the data inside the dabases ● Replay events from the production to have a usable set of data ● Useful for all these kinds of use cases: ○ Bench a new feature during user tests on a dedicated environment (before huge release!) ○ Bench a new version of a core component before migrating the compotant (mongo, rabbit…) ○ Create environment with a subset of events to reproduce production issues DevOps improvements - On demand test environment 27
- 28. ● Based on Helm, we use the CI to deploy an ephemeral environment with all the micro services ○ A frontend ○ A backend ○ Some microservices (pricing, emails …) ○ Ephemeral databases (postgresql, elastic, mongo) ● Automatic request a sub domain to add a new DNS entry (in fact we use a sub subdomain trick) ● Replay events from the production to have a usable set of data ● Useful to: ○ Bench a new feature during user tests on a dedicated env ○ Bench a new version of a core component before migrate the compotant ○ Create env with a subset of events to reproduce production issues DevOps improvements - On demand test environment 28
- 29. ● Based on Helm, we use the CI to deploy an ephemeral environment with all the micro services ○ A frontend ○ A backend ○ Some microservices (pricing, emails …) ○ Ephemeral databases (postgresql, elastic, mongo) ● Automatic request a sub domain to add a new DNS entry (in fact we use a sub subdomain trick) ● Replay events from the production to have a usable set of data ● Useful to: ○ Bench a new feature during user tests on a dedicated env ○ Bench a new version of a core component before migrate the compotant ○ Create env with a subset of events to reproduce production issues DevOps improvements - On demand test environment 29
- 30. ● Based on Helm, we use the CI to deploy an ephemeral environment with all the micro services ○ A frontend ○ A backend ○ Some microservices (pricing, emails …) ○ Ephemeral databases (postgresql, elastic, mongo) ● Automatic request a sub domain to add a new DNS entry (in fact we use a sub subdomain trick) ● Replay events from the production to have a usable set of data ● Useful to: ○ Bench a new feature during user tests on a dedicated env ○ Bench a new version of a core component before migrate the compotant ○ Create env with a subset of events to reproduce production issues DevOps improvements - On demand test environment 30
- 31. ● Based on Helm, we use the CI to deploy an ephemeral environment with all the micro services ○ A frontend ○ A backend ○ Some microservices (pricing, emails …) ○ Ephemeral databases (postgresql, elastic, mongo) ● Automatic request a sub domain to add a new DNS entry (in fact we use a sub subdomain trick) ● Replay events from the production to have a usable set of data ● Useful to: ○ Bench a new feature during user tests on a dedicated env ○ Bench a new version of a core component before migrate the compotant ○ Create env with a subset of events to reproduce production issues DevOps improvements - On demand test environment 31
- 32. 32 Thanks for listening, available for questions (with a beer!) @jwimsingues