Terraform AWS modules and some best-practices - May 2019
11 likes1,552 views
This document is a comprehensive guide on using Terraform for AWS infrastructure, emphasizing best practices, module management, and orchestration strategies. It highlights the advantages of Terraform over other tools, offers solutions for managing and organizing configurations, and provides insights on handling secrets and collaboration through various tools. The author, Anton Babenko, shares resources for open-source modules, visual diagram generators, and emphasizes the importance of clean code and documentation.
![- [ ] How to write modules?
- [ ] How to call modules?
- [ ] How to work with the code?
@antonbabenko](https://image.slidesharecdn.com/terraform-meetup-may-2019-190514150956/85/Terraform-AWS-modules-and-some-best-practices-May-2019-48-320.jpg)
![- [x] How to write modules?
- [x] Do not write, if you can
- [x] Avoid providers in modules, provisioners
- [ ] How to call modules?
- [ ] How to work with the code?
@antonbabenko](https://image.slidesharecdn.com/terraform-meetup-may-2019-190514150956/85/Terraform-AWS-modules-and-some-best-practices-May-2019-66-320.jpg)
![Orchestration = Terragrunt
@antonbabenko
subnet_id = ["subnet-1a2b3c4d"] # tfvars:terragrunt_output.vpc.public_subnets](https://image.slidesharecdn.com/terraform-meetup-may-2019-190514150956/85/Terraform-AWS-modules-and-some-best-practices-May-2019-90-320.jpg)
![- [x] How to write modules?
- [x] How to call modules?
- [x] 1-in-1 works beter over time
- [x] Orchestration = Terragrunt
- [x] Dynamic values in tfvars
- [ ] How to work with the code?
@antonbabenko](https://image.slidesharecdn.com/terraform-meetup-may-2019-190514150956/85/Terraform-AWS-modules-and-some-best-practices-May-2019-91-320.jpg)
![- [x] How to write modules?
- [x] How to call modules?
- [x] How to work with the code?
- [x] Lists in Terraform 0.11 can be painful
- [x] Perceive Terraform easier
@antonbabenko](https://image.slidesharecdn.com/terraform-meetup-may-2019-190514150956/85/Terraform-AWS-modules-and-some-best-practices-May-2019-105-320.jpg)
Terraform AWS modules and some best-practices - May 2019
- 1. Terraform AWS modules and some best- practices Anton Babenko @antonbabenko May 2019
- 2. Anton Babenko Terraform AWS fanatic since 2015 Organiser of HashiCorp UG, AWS UG, DevOps Norway, DevOpsDays Oslo I 💚 open-source: terraform-community-modules + terraform-aws-modules antonbabenko/pre-commit-terraform — clean code and documentation antonbabenko/tfvars-annotations — update terraform.tfvars using annotations antonbabenko/modules.tf-lambda — generate Terraform code from visual diagrams antonbabenko/terragrunt-reference-architecture — Terragrunt reference architecture www.terraform-best-practices.com medium.com/@anton.babenko @antonbabenko — Twitter, GitHub, Linkedin
- 3. What do I do? All-things Terraform + AWS + DevOps Consulting Workshops Trainings Mentorship My email: anton@antonbabenko.com LinkedIn: https://www.linkedin.com/in/antonbabenko
- 4. Collection of open-source Terraform AWS modules supported by the community. More than 2 mil. downloads since September 2017. (VPC, Autoscaling, RDS, Security Groups, ELB, ALB, Redshift, SNS, SQS, IAM, EKS, ECS…) github.com/terraform-aws-modules registry.terraform.io/modules/terraform-aws-modules @antonbabenko
- 5. Cloudcraft.co — the best way to draw AWS diagrams @antonbabenko
- 6. cloudcraft.co features • Manage components in browser (EC2 instances, autoscaling groups, RDS, etc) • Connect components • Import live AWS infrastructure • Calculate the budget • Share link to a blueprint • Export as image • Embed drawing to wiki, Confluence, etc @antonbabenko
- 7. Infrastructure as code makes DevOps possible Key benefits: • Treat infrastructure like application code • Always know what changed • Validate infrastructure before deployment https://dzone.com/articles/infrastructure-as-code-the-benefits @antonbabenko
- 8. Tool for building, changing and versioning infrastructure safely and efficiently. www.terraform.io @antonbabenko
- 10. @antonbabenko
- 11. @antonbabenko
- 12. @antonbabenko
- 13. @antonbabenko
- 15. Google Cloud Deployment Manager Azure Resource Manager @antonbabenko
- 16. @antonbabenko
- 18. Why Terraform and not AWS CloudFormation, Azure ARM, Google Cloud Deployment Manager? Terraform manages 100+ providers, has easier syntax (HCL), has native support for modules and remote states, has teamwork related features, is an open-source project. Provides a high-level abstraction of infrastructure (IaC) Allows for composition and combination Orchestration, not merely configuration Supports parallel management of resources (graph, fast) Separates planning from execution (dry-run) @antonbabenko
- 19. Terraform — universal tool for everything with an API Google G Suite Dropbox files and access New Relic metrics Datadog users and metrics Jira issues Minecraft, or even order Domino’s pizza All Terraform providers — https://www.terraform.io/docs/providers/index.html @antonbabenko
- 20. Let’s begin — everything fits in main.tf @antonbabenko
- 21. @antonbabenko
- 22. Project grows — main.tf: 20+ resources and data sources @antonbabenko
- 24. Resources, regions, providers, … @antonbabenko
- 25. @antonbabenko
- 26. @antonbabenko
- 27. @antonbabenko
- 28. @antonbabenko
- 29. @antonbabenko
- 31. Emerging issues Code size is increasing Dependencies between resources become complicated @antonbabenko
- 32. Solution — Terraform modules @antonbabenko
- 33. Modules in Terraform are self-contained packages of Terraform configurations that are managed as a group. @antonbabenko
- 34. Resource modules Create resources in a very flexible configuration Open-source @antonbabenko
- 39. Would you use Terraform module to manage AWS EC2 security group? @antonbabenko
- 40. @antonbabenko
- 41. Would you use Terraform module to manage AWS EC2 security group? Yes :) @antonbabenko
- 42. Infrastructure modules Consist of resource modules Enforce tags and company standards Use preprocessors, jsonnet, cookiecutter @antonbabenko
- 46. @antonbabenko
- 47. Types of Terraform modules Resource modules (github.com/terraform-aws-modules , for eg) Infrastructure modules @antonbabenko
- 49. Tip №0 Check Terraform Registry before writing resource modules @antonbabenko
- 51. @antonbabenko
- 52. @antonbabenko
- 55. Things to avoid in modules @antonbabenko
- 56. Exception: logical providers (template, random, local, http, external) Providers in modules — evil @antonbabenko
- 57. @antonbabenko
- 58. Provisioner — evil Avoid provisioner in all resources @antonbabenko
- 59. Provisioner — evil Avoid provisioner in all resources @antonbabenko
- 60. Provisioner — evil Avoid provisioner even in EC2 resources @antonbabenko
- 61. Provisioner — evil Avoid provisioner even in EC2 resources @antonbabenko
- 62. @antonbabenko
- 63. @antonbabenko
- 64. null_resource provisioner — good @antonbabenko
- 65. Traits of good Terraform modules Documentation and examples Feature rich Sane defaults Clean code Tests Read more: http://bit.ly/common-traits-in-terraform-modules @antonbabenko
- 67. How to structure Terraform configurations? How to call them? @antonbabenko
- 68. Call Terraform modules Use Terraform modules, because amount of resources and code is increasing How to organize Terraform configurations and invoke them? How to orchestrate modules? @antonbabenko
- 69. All-in-one Good: Declare variables and outputs in fewer places Bad: Large blast radius Everything is blocked at once Impossible to specify dependencies between modules (depends_on) @antonbabenko
- 70. 1-in-1 Good: Smaller blast radius Possible to join invocation Easier and faster to work with Bad: Declare variables and outputs in more places @antonbabenko
- 71. Which way do you group your code? All-in-one or 1-in-1? @antonbabenko
- 73. Correct MFA (Most Frequent Answer): Somewhere in between @antonbabenko
- 74. All-in-one Undefined project scope Fast prototyping and initial development phase Small number of resources & developers Tightly connected resources 1-in-1 @antonbabenko Defined project scope Different types of developers involved * Code reuse is encouraged (across organization and environments) Use Terragrunt
- 75. What about Terraform workspaces? @antonbabenko
- 76. Problems with Terraform workspaces Terraform Workspaces aren’t infrastructure-as-code friendly. You can’t answer straight from the code: "How many workspaces do you have?" "What infrastructure has been deployed in workspaceX?" "What is the difference between workspaceX and workspaceY?" Introducing complexity almost in all cases. @antonbabenko
- 77. Solution — use re-usable modules instead of workspaces @antonbabenko
- 78. What kind of orchestration method do you use? -target Makefile … @antonbabenko
- 80. No really, do not try this at home! @antonbabenko
- 81. Orchestration = Terragrunt https://github.com/gruntwork-io/terragrunt/ @antonbabenko
- 86. tfvars can’t contain dynamic values :( Orchestration = Terragrunt @antonbabenko subnet_id = "???"
- 87. Orchestration = Terragrunt tfvars can’t contain dynamic values, so I have fixed it :) @antonbabenko subnet_id = "???"
- 88. before_hook + shell script Go binary https://github.com/antonbabenko/tfvars-annotations — Update values in terraform.tfvars using annotations (WIP) or take a look at modules.tf @antonbabenko
- 89. Orchestration = Terragrunt @antonbabenko subnet_id = "" # tfvars:terragrunt_output.vpc.public_subnets
- 90. Orchestration = Terragrunt @antonbabenko subnet_id = ["subnet-1a2b3c4d"] # tfvars:terragrunt_output.vpc.public_subnets
- 92. Work with stateless lists @antonbabenko
- 93. @antonbabenko Work with stateless lists
- 94. @antonbabenko Work with stateless lists
- 95. https://jsonnet.org/ Work with stateful lists @antonbabenko
- 96. Work with stateful lists @antonbabenko
- 97. Work with stateful lists @antonbabenko
- 98. Work with stateful lists @antonbabenko
- 99. Work with stateful lists @antonbabenko
- 103. Edge cases Different AWS regions (version of S3 signature, EC2 ClassicLink, IPv6) Date of creation of AWS account Limits on resources in AWS Services and features availability @antonbabenko
- 104. Avoid in Terraform Not secret arguments should not be specified as command line arguments => put them in tfvars Reduce usage of "-target" and "-parallelism" "Terraform workspaces" evil in=> separate by directories Dependency hell in modules @antonbabenko
- 106. Summary Write less and simpler (Terraform 0.12 won’t fix your code for you!) Use existing modules and utilities @antonbabenko
- 107. How to handle secrets in Terraform? • Can you accept secrets to be saved in state file in plaintext? Probably not. • AWS IAM password & access secret keys — use PGP as keybase.io • AWS RDS — set dummy password and change after DB is created • AWS RDS — use iam_database_authentication_enabled = true • EC2 instance user-data + AWS KMS • EC2 instance user-data + AWS System Manager’s Parameter Store • AWS Secrets Manager • https://github.com/opencredo/terrahelp • Other options: • Secure remote state location (S3 bucket policy, KMS key) @antonbabenko
- 108. What are the tools/solutions out there? • Terraform Registry (https://registry.terraform.io/) — collection of public Terraform modules for common infrastructure configurations for any provider. • Terraform linter to detect errors that can not be detected by `terraform plan` — https://github.com/wata727/tflint • Terraform version manager — https://github.com/kamatama41/tfenv • A web dashboard to inspect Terraform States — https://github.com/camptocamp/ terraboard • Jsonnet — The data templating language — http://jsonnet.org @antonbabenko
- 109. Atlantis — Start working on Terraform as a team A unified workflow for collaborating on Terraform through GitHub, GitLab and Bitbucket https://www.runatlantis.io @antonbabenko
- 110. Bonus
- 115. ✓ cloudcraft.co — design, plan and visualize ✓ terraform-aws-modules — building blocks of AWS infrastructure ✓ Terraform — infrastructure as code
- 116. Infrastructure as code generator — from visual diagrams to Terraform https://github.com/antonbabenko/modules.tf-lambda Demo video: https://www.youtube.com/watch?v=F1Ax1zfZbiY
- 117. 1. Go to cloudcraft.co 2. Sign up, sign in (free account) 3. Draw your AWS infrastructure 4. Click "Export" 5. Click "Terraform code export" Try it yourself!
- 118. modules.tf — generated code ✓ Potentially ready-to-use Terraform configurations ✓ Suits best for bootstrapping ✓ Enforces Terraform best-practices ✓ Batteries included (terraform-aws-modules, terragrunt, tfvars- annotations, pre-commit) ✓ 100% free and open-source (https://github.com/antonbabenko/ modules.tf-lambda) ✓ Released under MIT license