leboncoin DataEngineering / Terraform - beginner to advanced
1 like292 views
The document is a comprehensive guide on using Terraform for managing AWS infrastructure, covering basics, collaboration, automation, and advanced features. It discusses core concepts such as providers, resources, and state management while providing practical examples and best practices. The conclusion emphasizes Terraform's effectiveness in infrastructure as code, safety in operations, and ease of sharing and reusing configuration.
![Terraform 101
Concepts:
● Providers
● Resources
● Data sources
A specific “dynamic” data you want
● External source
● Like dynamic variables
# Find the latest available AMI that is tagged with Component = web
data "aws_ami" "web" {
filter {
name = "state"
values = ["available"]
}
filter {
name = "tag:Component"
values = ["web"]
}
most_recent = true
}](https://image.slidesharecdn.com/lbcde-tf-beginnertoadvanced-190104104737/85/leboncoin-DataEngineering-Terraform-beginner-to-advanced-16-320.jpg)
![Terraform 101
Concepts:
● Providers
● Resources
● Data sources
● Variables
Parameters of our code
● Have to be declared specifically
● Different types (String, boolean, maps, list, …)
● Can have defaults
variable "key" { type = "string" }
variable "images" {
type = "map"
default = {
us-east-1 = "image-1234"
us-west-2 = "image-4567"
}
}
variable "zones" {
default = ["us-east-1a", "us-east-1b"]
}](https://image.slidesharecdn.com/lbcde-tf-beginnertoadvanced-190104104737/85/leboncoin-DataEngineering-Terraform-beginner-to-advanced-17-320.jpg)
![Terraform 103
Remote state access
{
version: 3,
terraform_version: "0.11.3",
serial: 43,
lineage: "c188d838-a1a0-419a-b04d-31ccb92b6e2c",
modules: [
{
path: [
"root"
],
outputs: {
spark_master_dns: {
sensitive: false,
type: "list",
value: [
"spark-master-qa-0.data.mydomain.io"
]
},
spark_master_ips: {
sensitive: false,
type: "list",
value: [
"172.17.32.207"
]
},
spark_role: {
sensitive: false,
type: "string",
value: "spark-s3rw-qa"
},
spark_sg: {
sensitive: false,
type: "string",
value: "sg-xxxxxxxx"
}
},](https://image.slidesharecdn.com/lbcde-tf-beginnertoadvanced-190104104737/85/leboncoin-DataEngineering-Terraform-beginner-to-advanced-52-320.jpg)
leboncoin DataEngineering / Terraform - beginner to advanced
- 1. Demystifying Terraform to manage AWS @lbcde 2018-10-26
- 2. Xavier Krantz - Site Reliability Engineer @Leboncoin Previously: ● Criteo ● Viadeo ● Smile (OSS integrator) https://github.com/xakraz https://speakerdeck.com/xakraz https://fr.linkedin.com/in/xavierkrantz/en About Me
- 3. Introduction ● Terraform 101 - Bases ● Terraform 102 - Working together ● Terraform 103 - Easier, Better, Stronger ● Terraform 104 - Automation & Tooling Conclusion Agenda
- 4. Introduction AWS “management” today @lbcde ● Web console ● Python boto scripts (some)
- 5. Introduction Needs ● A way to work as a team ● A way to document our work ● History
- 6. Introduction Existing tools ● Code libraries ● Config management ● AWS Service / Other SaaS https://www.terraform.io/intro/vs/index.html
- 7. Introduction Existing tools ● Code libraries ● Config management ● AWS Service / Other SaaS https://www.terraform.io/intro/vs/index.html
- 8. Introduction Existing tools ● Code libraries ● Config management ● AWS Service / Other SaaS https://www.terraform.io/intro/vs/index.html
- 11. Terraform 101 Overview Terraform is a tool for building, changing and versioning infrastructure safely and efficiently. Terraform can manage existing and popular service providers as well as custom in-house solutions. terraform.io/intro
- 12. Terraform 101 Overview What is Terraform ● Infrastructure as code ● Execution plan ● Resource graph ● Change automation tool https://www.terraform.io/intro/index.html
- 15. Terraform 101 Concepts: ● Providers ● Resources A TANGIBLE component of you infrastructures ● Provider specific ● What you want to manage resource "aws_db_instance" "timeout_example" { allocated_storage = 10 engine = "mysql" engine_version = "5.6.17" instance_class = "db.t1.micro" name = "mydb" # ... timeouts { create = "60m" delete = "2h" } }
- 16. Terraform 101 Concepts: ● Providers ● Resources ● Data sources A specific “dynamic” data you want ● External source ● Like dynamic variables # Find the latest available AMI that is tagged with Component = web data "aws_ami" "web" { filter { name = "state" values = ["available"] } filter { name = "tag:Component" values = ["web"] } most_recent = true }
- 17. Terraform 101 Concepts: ● Providers ● Resources ● Data sources ● Variables Parameters of our code ● Have to be declared specifically ● Different types (String, boolean, maps, list, …) ● Can have defaults variable "key" { type = "string" } variable "images" { type = "map" default = { us-east-1 = "image-1234" us-west-2 = "image-4567" } } variable "zones" { default = ["us-east-1a", "us-east-1b"] }
- 18. Terraform 101 Concepts: ● Providers ● Resources ● Data sources ● Variables ● Outputs Outputs = Informations we want to get after Terraform has run ● Can be queried via CLI ● Will be shared across modules and resources output "address" { value = "${aws_instance.db.public_dns}" }
- 20. Terraform 101 Files ● *.tf ● *.tfvars *.auto.tfvars terraform.tfvars ● terraform.tfstate https://www.terraform.io/intro/getting-started/install.html Basics: ● Files
- 21. Terraform 101 4 Main commands ● terraform init ● terraform plan ● terraform apply ● terraform destroy https://www.terraform.io/intro/getting-started/install.html Basics: ● Files ● Commands
- 22. Terraform 101 Other capabilities ● Templates / Files ● Provisioner ● Built-in “functions” ● Basic conditionals https://www.terraform.io/intro/getting-started/provision.html https://www.terraform.io/docs/configuration/interpolation.html Basics: ● Files ● Commands ● Others
- 24. Terraform 102 Working together TF internals Remote state State locking
- 26. Terraform 102 Internals 1 - Pre-Compiles Check syntax, types … Validate resources 5 - Applies Makes the API call to apply the changes described in the plan 2 - Refresh the state Call the providers APIs to get an updated view 4 - Plan Computes the plan to match the desired state 3 - Compiles 2 Runs DataSources Instantiates the resources -> Gets desired state Terraform internals 6 - Applies Updates the final state file
- 27. Terraform 102 Internals 1 - Pre-Compiles Check syntax, types … Validate resources 5 - Applies Makes the API call to apply the changes described in the plan 2 - Refresh the state Call the providers APIs to get an updated view 4 - Plan Computes the plan to match the desired state 3 - Compiles 2 Runs DataSources Instantiates the resources -> Gets desired state Terraform internals 6 - Applies Updates the final state file
- 28. Terraform 102 - Internals
- 29. Terraform 102 - Internals ? ?
- 31. Terraform Remote state “Backend”: principles Terraform 102 Remote state
- 32. Terraform Remote state “Backend”: types Terraform 102 Remote state
- 33. Terraform Remote state “Backend”: example Terraform 102 Remote state backend.tf terraform { backend "s3" { bucket = "mybucket_name" key = "path/to/my/key" } }
- 35. Terraform 102 - State locking
- 36. Terraform 102 - State locking
- 37. Terraform 102 - State locking ?
- 38. Terraform Remote state “Backend”: ● S3 ● + DynamoDB Terraform 102 State locking backend.tf terraform { backend "s3" { bucket = "my_bucket_name" encrypt = "true" dynamodb_table = "my_ddb_table)name" region = "eu-west-1" role_arn = "arn:aws:iam::xxxxxxxxxxxx:role/AssumeRole" } }
- 39. Terraform 102 - State locking
- 40. Terraform 102 - State locking
- 41. Terraform 103 Better, easier, stronger
- 42. Terraform 103 Better, easier, stronger Modules Remote state access Workspaces
- 44. Terraform 103 Modules Terraform Modules ● Reusable set of “pre” defined / packaged resources ● Helps to model the architecture Features: ● Versioned ● Various sources: ○ HTTP ○ SCM (git, svn, hg, …) ○ Local file system https://registry.terraform.io/ https://www.terraform.io/docs/modules/index.html
- 45. Terraform 103 Modules Terraform Modules privacy-access.tf module "privacy-access" { source = "modules/privacy-access" instance_count = "${var.access_instance_count}" instance_type = "${var.access_instance_type}" … }
- 46. Terraform 103 Modules Terraform Modules data-privacy/ | ├── code/ ├── modules/ ├── shared/ └── README.md
- 47. Terraform 103 Modules Terraform Modules data-privacy/ | ├── code/ ├── modules/ ├── shared/ └── README.md data-privacy/ └── modules/ └── privacy-access/ ├── alb.tf ├── ec2.tf ├── iam.tf ├── rds.tf ├── s3.tf ├── sg.tf | ├── outputs.tf └── input.tf
- 48. Terraform 103 Modules Terraform Modules data-privacy/ | ├── code/ ├── modules/ ├── shared/ └── README.md data-privacy/ └── modules/ └── privacy-access/ ├── alb.tf ├── ec2.tf ├── iam.tf ├── rds.tf ├── s3.tf ├── sg.tf | ├── outputs.tf └── input.tf data-privacy/ └── code/ ├── modules -> ../modules/ ├── vars/ │ ├── aws-account/ │ │ ├── datadev.tfvars -> │ │ └── dataprod.tfvars -> │ │ │ └── env/ │ ├── prod.tfvars │ ├── qa0.tfvars │ └── qa2.tfvars │ ├── backend.conf ├── backend.tf -> ../shared/backend.tf ├── shared-variables.tf -> │ ├── privacy-access.tf ├── privacy-request.tf │ ├── route53.tf ├── security_groups.tf │ ├── tf-config.tf ├── data-sources.tf ├── outputs.tf └── variables.tf
- 49. Terraform 103 Remote state access
- 50. Terraform Remote state “data source” Terraform 103 Remote state access data-privacy/scripts/provision/terraform/code/data-sources.tf data "terraform_remote_state" "spark" { backend = "s3" config{ bucket = "my_bucket_name" region = "${var.region}" key = "env:/${var.env_type}/spark/main.tfstate" } }
- 51. Terraform Remote state “data source” Terraform 103 Remote state access data-privacy/scripts/provision/terraform/code/data-sources.tf data "terraform_remote_state" "spark" { backend = "s3" config{ bucket = "data-engineering.infrastructure.leboncoin.io-tfstates" region = "${var.region}" key = "env:/${var.env_type}/spark/main.tfstate" } } privacy-access.tf module "privacy-access" { source = "modules/privacy-access" # Spark shared cluster spark_role = "${data.terraform_remote_state.spark.spark_role}" spark_security_group_id = "${data.terraform_remote_state.spark.spark_sg}" instance_count = "${var.access_instance_count}" instance_type = "${var.access_instance_type}" ... }
- 52. Terraform 103 Remote state access { version: 3, terraform_version: "0.11.3", serial: 43, lineage: "c188d838-a1a0-419a-b04d-31ccb92b6e2c", modules: [ { path: [ "root" ], outputs: { spark_master_dns: { sensitive: false, type: "list", value: [ "spark-master-qa-0.data.mydomain.io" ] }, spark_master_ips: { sensitive: false, type: "list", value: [ "172.17.32.207" ] }, spark_role: { sensitive: false, type: "string", value: "spark-s3rw-qa" }, spark_sg: { sensitive: false, type: "string", value: "sg-xxxxxxxx" } },
- 54. Terraform 103 Workspaces Terraform States “workspaces” ● 1st the Monolith main.tf terraform.tfvars
- 55. Terraform 103 Workspaces Terraform States “workspaces” ● 2nd the split backend.tf main.tf ec2.tf route53.tf security-groups.tf terraform.tfvars
- 56. Terraform 103 Workspaces Terraform States “workspaces” ● 3rd ENVs segregation
- 57. Terraform 103 Workspaces Terraform States “workspaces” ● 3rd ENVs segregation
- 58. Terraform 103 Workspaces Terraform States “workspaces” ● 3rd ENVs segregation
- 59. Terraform 103 Workspaces Terraform States “workspaces” ● 3rd ENVs segregation WHY ? →Use a variable in Backend config ?
- 60. Terraform 103 Workspaces Terraform States “workspaces” ● 3rd ENVs segregation WHY ? →Use a variable in Backend config ?
- 61. Terraform 103 - Workspaces Workspaces Terraform States “workspaces” ● 4th the State workspace
- 62. Terraform 103 - Workspaces
- 63. Terraform 104 Tooling & Automation
- 64. Terraform 104 Tooling & Automation Automation Monitoring Tools
- 65. Terraform 104 Automate your needs To meet your workflow
- 66. Why automation ? ● Terraform 104 Automate you needs data-privacy/ | ├── code/ ├── modules/ ├── shared/ └── README.md data-privacy/ └── code/ ├── modules -> ../modules/ ├── vars/ │ ├── aws-account/ │ │ ├── datadev.tfvars -> │ │ └── dataprod.tfvars -> │ │ │ └── env/ │ ├── prod.tfvars │ ├── qa0.tfvars │ └── qa2.tfvars │ ├── backend.conf ├── backend.tf -> ../shared/backend.tf ├── shared-variables.tf -> │ ├── privacy-access.tf ├── privacy-request.tf │ ├── route53.tf ├── security_groups.tf │ ├── tf-config.tf ├── data-sources.tf ├── outputs.tf └── variables.tf data-privacy/ └── modules/ └── privacy-access/ ├── alb.tf ├── ec2.tf ├── iam.tf ├── rds.tf ├── s3.tf ├── sg.tf | ├── outputs.tf └── input.tf
- 67. Why automation ? Terraform 104 Automate you needs $ cd YOUR_PROJECT_PATH $ terraform init -backend-config=./backend.conf $ terraform apply -var-file=./vars/env/{env}.tfvars -var-file=./vars/aws-account/{aws_account}.tfvars
- 68. Automated actions via “invoke” Terraform 104 Automate you needs $ invoke -l Available tasks: ... provision.apply Update the whole stack (More with '--help') provision.destroy Destroy the aws resources (More with '--help') provision.init Initialize Terraform (More with '--help') provision.list-stack-envs provision.list-stacks provision.status Display IDs of current resources (More with '--help') ...
- 70. Terraform 104 State Drift Detection ● TF is imperative by usage (No daemon) ● For better readability -> Split your code in “Stacks” ● Shared with data-sources among teams ● Manual actions in the AWS Console or other projects https://www.hashicorp.com/blog/detecting-and-managing-drift-with-terraform https://medium.com/build-acl/state-drift-detection-using-terraform-d0383628d2ea Monitoring
- 71. Terraform 104 State Drift Detection https://github.com/gibbster/terraform-plan-drift-checker Monitoring
- 73. Pre-commit - Terraform Terraform 104 Tools https://pre-commit.com/ https://github.com/antonbabenko/pre-commit-terraform
- 78. Conclusion
- 80. Terraform 102 Working together TF internals Remote state State locking
- 81. Terraform 103 Better, easier, stronger Modules Remote state access Workspaces
- 82. Terraform 104 Tooling & Automation Automation Monitoring Tools
- 83. Conclusion Terraform: ● 1 binary, for every OS ● Wide range of providers ● Simple concepts Answers our needs: ● Infra as Code ● Operations safety ● Share and reuse with ease
- 85. Links References Official doc: ● Terraform.io Modules registry: ● registry.terraform.io Some inspiring presentations ● https://speakerdeck.com/jmickey/introduction-to-terraform ● https://speakerdeck.com/so0k/terraform-at-honestbee Good tools: ● https://github.com/camptocamp/terraboard ● https://github.com/28mm/blast-radius ● https://github.com/segmentio/terraform-docs ● https://github.com/coinbase/terraform-landscape ● https://github.com/shuaibiyy/awesome-terraform