How to scale with Terraform
Download as PPTX, PDF2 likes337 views
The document summarizes an Equipe Meetup event for the HashiCorp User Group Lyon. It includes: - Two technical talks on using Terraform to scale infrastructure and deploy Prometheus and Thanos on Kubernetes. - Details on the HashiCorp User Group Lyon which has been running since June 2019, has 120 members, and holds bimonthly meetups in Lyon as well as other cities in France. - A presentation by Mehdi Laruelle on using Terraform at scale which covers introducing Terraform and community, strategies for using Terraform across multiple accounts and regions, best practices for modularizing infrastructure with Terraform modules, and considerations for when to use Ter
How to scale with Terraform
- 2. Equipe Meetup Julien Bichon Community ambassador @jubichon + Talk 1 - How to scale with Terraform - Mehdi Laruelle + Talk 2 - Déployer Prometheus + Thanos avec Terraform dans kubernetes - Thierry Sallé
- 3. HashiCorp User Group LYON + Depuis juin 2019 + 120 membres + 2 meetups + 5 HUG en France + Paris + Toulouse + Nice + Nantes + Lyon
- 4. Thank you
- 5. Mehdi LARUELLE @D2SI Hashicorp Terraform: How to scale
- 6. D2SI Me Mehdi LARUELLE Consultant Cloud & Automation @mehdilaruelle Whoami ?
- 7. Christophe Gasmi Consultant Cloud & Kubernetes chez D2SI @RekCaH_fr The force will be always with you
- 8. PROGRAMME 1 Terraform ? Community ? 2 One for all 3 All for one 4 What next ?
- 11. Opensource
- 12. Providers
- 13. HCL
- 14. Multi providers usage in HCL
- 15. Plugins & more
- 16. // One for all2
- 17. Stack App Organization AppApp App App BackendApp Backend App Backend DEV RE7PROD App Frontend App FrontendApp Frontend
- 18. Isolation Folder Variable maps Tfvars Tfvars + maps Divergence Complexific ation Solution ? Best for multi region
- 19. Starting point Root Base Infra Apps As an Ops, I need to update my EC2 instance As a Dev, I need to update my app
- 20. Example with one region
- 21. Example with multiple regions
- 22. Example with multiple regions
- 23. Modularity ? Functional usage Documentation Test Reusable and standard
- 24. Module module name : kms variables.tfmain.tf module name : kms
- 25. Module: best practices 1. No big module 2. No module-inception 3. Have a nice variables naming and description 4. Define a minimal provider & Terraform version a. terraform { required_version = "< 0.12.0" } b. terraform { required_providers { aws = ">= 2.6.0" } } 5. Have an output defined by a ressource Ex: aws_vpc.default.id => vpc_id aws_subnet.private.*.id => private_subnets_id aws_subnet.private.*.id => private_subnets
- 26. Module around the world Documentation ● Automation: ○ terraform-docs ○ pre-commit-terraform ○ CI ● Examples: ○ Make multiple cases ○ Test it with CI (terraform fmt, check, validate, terraform-kitchen)
- 27. To Terraform or not to Terraform Strongly coupled: Advantage : Reusable resources & less static values Disadvantage : ● Not always have access read only to the tfstate by another team ● Can be difficult to reuse Terraform output from another tool Loosely coupled: Advantage : No Terraform dependencies Disadvantage : ● A resource do not have always a data source ● Infrastructure state can shift without knowing
- 28. Strongly coupled local computer webapp vpc terraform.tfstate terraform.tfstate main.tf webapp vpc main.tf my-bucket-tfstate
- 30. Loosely coupled local computer webapp vpc terraform.tfstate terraform.tfstate main.tf webapp vpc main.tf my-bucket-tfstate
- 31. // All for one3
- 32. All for the best in the best of all worlds Ops
- 33. In real world Ops1 Ops2 Problems: ● No lock ● Local state ● Terraform file can be variant
- 34. With CI/CD 1. Push IaC 2. CI/CD process use Terraform Ops Tfstate Lock
- 35. CI/CD example (Christophe G.) 1.1 Create branch from develop 1.2. Hook pre- commit 1.3. Commit code 1.4. Push code 3. Terraform apply in Feature Env Ops Reviewers 4. Infrastructure/Apps Tests 2. Open Merge Request automatically 5. Merge Request Review
- 36. CI/CD example (Christophe G.) 9a. Terraform plan in Dev Env Reviewers 9b. Terraform apply in Dev Env (Manual) 8. Open Merge Request automatically 10. Merge Request Review 7. Pipeline run in develop branch
- 37. CI/CD example (Christophe G.) 14. Terraform plan in Staging Env Ops & Reviewers 15. Terraform apply in Staging Env (Manual) 16. Tests Infra/Apps 12. Pipeline run in master branch 17. Auto Tag Branch master 18. Terraform plan in Prod Env 19. Terraform apply in Prod Env (Manual) ● Analyse terraform plan ● Start the terraform apply job 20a. Tests Infra/Apps20b. Rollback if fail
- 38. What about the flow ? Master (prod) Develop Feature V0.1 V1.0
- 39. // To infinity and // beyond 4
- 40. The last but not the least ● Always return an empty list in your output at least resource "aws_subnets" "public" { count = "${var.subnets_public_enabled ? length(var.subnets_public) : 0}" vpc_id = aws_vpc.main.id cidr_block = element(var.subnets_public, count.index) # ... arguments omitted } output "public_subnets" { description = "The subnet IDs for public network" value = "${concat(aws_subnets.public.*.id, list(""))}" }
- 41. The last but not the least Define a commun separator Write in lowercase resource "aws_eip" "public" {} resource "aws_eip" "public_eip" {} resource "aws_eip" "public_aws_eip" {} resource "aws_eip" "gitlab_public" {} resource "aws_eip" "gitlab-public" {} Or
- 42. Best Practice CI/CD The last but not the least Make your own TF image (providers included) Module as artefact or cache (each steps) TF_IN_AUTOMATION Use variables and datasource Tagging: ● Projet ● Env ● Namespace ● terraform_state_bucket ● terraform_state_key ● terraform_git_repo Keep It Simple Stupid Module everywhere Never apply yourself, let the CI do it Documentat ion can be obfuscated Force your providers & TF versions Update frequently providers & TF
- 43. Questions ?