![ Configuration
●
Variables can have descriptions. (For generating docs)
– lists
– maps
variable "users" {
description = “Users with ssh access disabled”
type = "list"
default = ["admin", "ubuntu"]
}
variable "images" {
type = "map"
default = {
us-east-1 = "image-1234"
us-west-2 = "image-4567"
}
}](https://image.slidesharecdn.com/terraformintromeetup-170519231229/85/Introductory-Overview-to-Managing-AWS-with-Terraform-20-320.jpg)
Introductory Overview to Managing AWS with Terraform
- 1. An Introductory OverviewAn Introductory Overview https://github.com/beanaroo/aws_nz_meetup-terraform_introhttps://github.com/beanaroo/aws_nz_meetup-terraform_intro
- 2. Agenda ● IaC refresher ● Who’s Hashicorp? ● What’s Terraform? ● Why Terraform? ● How to Terraform ● Who’s Invenco? ● How do we Terraform? ● Q & A
- 6. About Me ● CMS/LTSP/ERP/POS/COMPNOR ● Penguins are fun ● New to DevOps!
- 7. About Me ● CMS/LTSP/ERP/POS/COMPNOR ● Penguins are fun ● New to DevOps! ● (Disclaimer) New to AWS
- 8. Infrastructure as Code ● Imperitive – How to create my environment... ● Declaritive – What my environment should be...
- 9. Infrastructure as Code ● Imperitive – Close ties with Config Management – Familiar tool sets – Troubleshooting not always straight forward when provisioning is orchestrated ● Declaritive – CM may be wanted/needed anyway – DSLs for high-level definitions of desired states – Subtle variations are inconvenient to manage
- 10. ● Founded in 2010 ● The “Atlassian” of DevOps ● Produce Open-Source Software Suite ● Enterprise versions with added functionality
- 11. ● Provisions compute, storage, and networking resources across multiple services. ● “It is an open source tool that codifies APIs into declarative configuration files...” ● DSL – HCL (HashiCorp Configuration Language) – HIL (HashiCorp Interpolation Language) ● All written in Go!!!
- 12. ● Alicloud ● Archive ● Arukas ● AWS ● Bitbucket ● CenturyLinkCloud ● Chef ● Circonus ● Cloudflare ● CloudStack ● Cobbler ● Consul ● Datadog ● DigitalOcean ● DNS ● DNSMadeEasy ● DNSimple ● Docker ● Dyn ● External ● Fastly ● GitHub ● Google Cloud ● Grafana ● Heroku ● Icinga2 ● Ignition ● InfluxDB ● Kubernetes ● Librato ● Local ● Logentries ● Mailgun ● New Relic ● Nomad ● NS1 ● Microsoft Azure ● Microsoft Azure (Legacy ASM) ● MySQL ● 1&1 ● Oracle Public Cloud ● OpenStack ● OpsGenie ● Packet ● PagerDuty ● PostgreSQL ● PowerDNS ● ProfitBricks ● RabbitMQ ● Rancher ● Random ● Rundeck ● Scaleway ● SoftLayer ● StatusCake ● Spotinst ● Template ● Terraform ● Terraform Enterprise ● TLS ● Triton ● UltraDNS ● Vault ● VMware vCloud Director ● VMware vSphere
- 13. Why Terraform... ● Multi-platform ● Single 30MB binary – No dependencies ● No configuration needed ● Agentless! ● Active and transparent development
- 14. How to Terraform... ● CLI ● Configuration (HCL) ● Interpolation (HIL) ● Providers – Resources – Data Sources ● Provisioners ● State – Backends ● Modules ● Plugins
- 15. terraform.tfstate ● JSON file mapping all managed resources and their metadata. ● Refreshes when needed or on demand. ● Not to be tampered with!!! ● May contain secrets ● Can not be used to roll back. (FOSS version) ● Stored locally by default.
- 16. Backends ● artifactory ● azure ● consul ● etcd ● gcs ● http ● manta ● s3 ● swift ● terraform enterprise ● States should be managed centrally. ● Sensitive information needs a home. ● VCS/SCM works well but humans don’t. ● State-locking prevents simultaneous changes.
- 17. terraform.tfstate ● JSON file mapping all managed resources and their metadata. ● Refreshes when needed or on demand. ● Not to be tampered with!!! ● May contain secrets ● Can not be used to roll back. (FOSS version) ● Stored locally by default.
- 18. Configuration ● All .tf files are loaded from the working directory ● Variables need to be defined ● Values are read from .tfvars or interactively prompted for. resource "aws_instance" "web" { ami = "ami-4836a428" source_dest_check = false private_ip = “10.20.0.10” }
- 19. Configuration ● Variables can have defaults. Lack of which implies “required”. ● Variable types: – strings variable "key" { type = "string" default = "value" } variable "long_key" { type = "string" default = <<EOF This is a long key. Running over several lines. EOF }
- 20. Configuration ● Variables can have descriptions. (For generating docs) – lists – maps variable "users" { description = “Users with ssh access disabled” type = "list" default = ["admin", "ubuntu"] } variable "images" { type = "map" default = { us-east-1 = "image-1234" us-west-2 = "image-4567" } }
- 21. Configuration ● Lists and Maps can be nested. BUT... ● Interpolation breaks. i.e. They aren’t useful at the moment.
- 22. Outputs ● Attributes that can be exposed to other plans ● Useful when working with modules output "address" { value = "${aws_instance.db.public_dns}" }
- 23. Interpolation ● Wrapped in ${} ● Used to access variables, perform math and functions variable "ami" { description = "the AMI to use" } resource "aws_instance" "web" { ami = "${var.ami}" source_dest_check = false private_ip = “${var.ip}” } # Amazon Linux 2017.03 AMI ami = “ami-4836a428” ip = “10.20.0.10”
- 24. Interpolation ● Functions include: – string/list/timestamp formatting – base64 encoding/decoding – uuid generation – md5/sha1/sha256 hashing – CIDR formatting – map lookups ● Provides basic logic through ternary operation CONDITION ? TRUEVAL : FALSEVAL . resource "aws_instance" "web" { subnet = "${var.env == "production" ? var.prod_subnet : var.dev_subnet}" }
- 25. Providers ● Provide resources to create and manage – AWS ● aws_vpc ● aws_instance ● aws_s3_bucket ● aws_route53_record ● Provide data sources to reference info. – AWS ● aws_ami (search for id) ● aws_hosted_zone_id (useful for R53) ● aws_iam_policy_document (generate policies using HCL) – terraform ● reference resources in other environments
- 26. Resources ● Infrastructure building blocks. – Have specific arguments – Provide attributes to reference elsewhere – Share common meta-parameters: ● count – No. of resource instances to create ● depends_on – Explicit dependencies ● provider – (i.e. different AWS account) ● lifecycle – create_before_destroy (replacement provisioned first) – prevent_destroy (protects resource) – ignore_changes (list of attributes)
- 27. Provisioners ● Perform local and remote command execution – chef – file – local-exec – remote-exec ● Useful for bootstrapping
- 28. Modules ● Reusable plans. ● Outputs need to be propegated. imperial_cloud_services/ ├── environments │ ├── dev │ ├── prod │ ├── qa │ └── uat └── modules ├── our_cloudfront_setup ├── our_custom_ec2_server └── our_vpc_layout
- 29. terraform --help Common commands: apply Builds or changes infrastructure console Interactive console for Terraform interpolations destroy Destroy Terraform-managed infrastructure env Environment management fmt Rewrites config files to canonical format get Download and install modules for the configuration graph Create a visual graph of Terraform resources import Import existing infrastructure into Terraform init Initialize a new or existing Terraform configuration output Read an output from a state file plan Generate and show an execution plan push Upload this Terraform module to Atlas to run refresh Update local state file against real resources show Inspect Terraform state or plan taint Manually mark a resource for recreation untaint Manually unmark a resource as tainted validate Validates the Terraform files version Prints the Terraform version All other commands: debug Debug output management (experimental) force-unlock Manually unlock the terraform state state Advanced state management
- 30. terraform --help Common commands: apply Builds or changes infrastructure console Interactive console for Terraform interpolations destroy Destroy Terraform-managed infrastructure env Environment management fmt Rewrites config files to canonical format get Download and install modules for the configuration graph Create a visual graph of Terraform resources import Import existing infrastructure into Terraform init Initialize a new or existing Terraform configuration output Read an output from a state file plan Generate and show an execution plan push Upload this Terraform module to Atlas to run refresh Update local state file against real resources show Inspect Terraform state or plan taint Manually mark a resource for recreation untaint Manually unmark a resource as tainted validate Validates the Terraform files version Prints the Terraform version All other commands: debug Debug output management (experimental) force-unlock Manually unlock the terraform state state Advanced state management
- 31. terraform --help Common commands: apply Builds or changes infrastructure console Interactive console for Terraform interpolations destroy Destroy Terraform-managed infrastructure env Environment management fmt Rewrites config files to canonical format get Download and install modules for the configuration graph Create a visual graph of Terraform resources import Import existing infrastructure into Terraform init Initialize a new or existing Terraform configuration output Read an output from a state file plan Generate and show an execution plan push Upload this Terraform module to Atlas to run refresh Update local state file against real resources show Inspect Terraform state or plan taint Manually mark a resource for recreation untaint Manually unmark a resource as tainted validate Validates the Terraform files version Prints the Terraform version All other commands: debug Debug output management (experimental) force-unlock Manually unlock the terraform state state Advanced state management
- 32. terraform --help Common commands: apply Builds or changes infrastructure console Interactive console for Terraform interpolations destroy Destroy Terraform-managed infrastructure env Environment management fmt Rewrites config files to canonical format get Download and install modules for the configuration graph Create a visual graph of Terraform resources import Import existing infrastructure into Terraform init Initialize a new or existing Terraform configuration output Read an output from a state file plan Generate and show an execution plan push Upload this Terraform module to Atlas to run refresh Update local state file against real resources show Inspect Terraform state or plan taint Manually mark a resource for recreation untaint Manually unmark a resource as tainted validate Validates the Terraform files version Prints the Terraform version All other commands: debug Debug output management (experimental) force-unlock Manually unlock the terraform state state Advanced state management
- 33. terraform --help Common commands: apply Builds or changes infrastructure console Interactive console for Terraform interpolations destroy Destroy Terraform-managed infrastructure env Environment management fmt Rewrites config files to canonical format get Download and install modules for the configuration graph Create a visual graph of Terraform resources import Import existing infrastructure into Terraform init Initialize a new or existing Terraform configuration output Read an output from a state file plan Generate and show an execution plan push Upload this Terraform module to Atlas to run refresh Update local state file against real resources show Inspect Terraform state or plan taint Manually mark a resource for recreation untaint Manually unmark a resource as tainted validate Validates the Terraform files version Prints the Terraform version All other commands: debug Debug output management (experimental) force-unlock Manually unlock the terraform state state Advanced state management
- 34. terraform --help Common commands: apply Builds or changes infrastructure console Interactive console for Terraform interpolations destroy Destroy Terraform-managed infrastructure env Environment management fmt Rewrites config files to canonical format get Download and install modules for the configuration graph Create a visual graph of Terraform resources import Import existing infrastructure into Terraform init Initialize a new or existing Terraform configuration output Read an output from a state file plan Generate and show an execution plan push Upload this Terraform module to Atlas to run refresh Update local state file against real resources show Inspect Terraform state or plan taint Manually mark a resource for recreation untaint Manually unmark a resource as tainted validate Validates the Terraform files version Prints the Terraform version All other commands: debug Debug output management (experimental) force-unlock Manually unlock the terraform state state Advanced state management
- 35. terraform --help Common commands: apply Builds or changes infrastructure console Interactive console for Terraform interpolations destroy Destroy Terraform-managed infrastructure env Environment management fmt Rewrites config files to canonical format get Download and install modules for the configuration graph Create a visual graph of Terraform resources import Import existing infrastructure into Terraform init Initialize a new or existing Terraform configuration output Read an output from a state file plan Generate and show an execution plan push Upload this Terraform module to Atlas to run refresh Update local state file against real resources show Inspect Terraform state or plan taint Manually mark a resource for recreation untaint Manually unmark a resource as tainted validate Validates the Terraform files version Prints the Terraform version All other commands: debug Debug output management (experimental) force-unlock Manually unlock the terraform state state Advanced state management
- 36. terraform --help Common commands: apply Builds or changes infrastructure console Interactive console for Terraform interpolations destroy Destroy Terraform-managed infrastructure env Environment management fmt Rewrites config files to canonical format get Download and install modules for the configuration graph Create a visual graph of Terraform resources import Import existing infrastructure into Terraform init Initialize a new or existing Terraform configuration output Read an output from a state file plan Generate and show an execution plan push Upload this Terraform module to Atlas to run refresh Update local state file against real resources show Inspect Terraform state or plan taint Manually mark a resource for recreation untaint Manually unmark a resource as tainted validate Validates the Terraform files version Prints the Terraform version All other commands: debug Debug output management (experimental) force-unlock Manually unlock the terraform state state Advanced state management
- 37. terraform --help Common commands: apply Builds or changes infrastructure console Interactive console for Terraform interpolations destroy Destroy Terraform-managed infrastructure env Environment management fmt Rewrites config files to canonical format get Download and install modules for the configuration graph Create a visual graph of Terraform resources import Import existing infrastructure into Terraform init Initialize a new or existing Terraform configuration output Read an output from a state file plan Generate and show an execution plan push Upload this Terraform module to Atlas to run refresh Update local state file against real resources show Inspect Terraform state or plan taint Manually mark a resource for recreation untaint Manually unmark a resource as tainted validate Validates the Terraform files version Prints the Terraform version All other commands: debug Debug output management (experimental) force-unlock Manually unlock the terraform state state Advanced state management
- 38. terraform --help Common commands: apply Builds or changes infrastructure console Interactive console for Terraform interpolations destroy Destroy Terraform-managed infrastructure env Environment management fmt Rewrites config files to canonical format get Download and install modules for the configuration graph Create a visual graph of Terraform resources import Import existing infrastructure into Terraform init Initialize a new or existing Terraform configuration output Read an output from a state file plan Generate and show an execution plan push Upload this Terraform module to Atlas to run refresh Update local state file against real resources show Inspect Terraform state or plan taint Manually mark a resource for recreation untaint Manually unmark a resource as tainted validate Validates the Terraform files version Prints the Terraform version All other commands: debug Debug output management (experimental) force-unlock Manually unlock the terraform state state Advanced state management
- 39. Terraform in action...
- 42. Let’s create a VPC
- 43. Needs some internet
- 44. Needs some private subnets
- 45. Let’s make it a module
- 46. Add some servers...
- 47. Spin up a whole new env!
- 48. Invenco Link L1-100 2-Port Network Link G6-200 All-in-one Payment Terminal Invenco Link L3-100 16-Port Network Link G7-100 Modular Payment Terminal Invenco Cloud Services (ICS) Remote Software Updates Remote Financial Key Injection (RKI) Asset Management and Diagnostics Usage and Uptime Metrics Prompt Management (Development) Media Management (Development) Time of Day Content (Development) C1-100 Site Controller Forecourt control application Invenco EPS Electronic Payment Server (EPS) Invenco Professional Services Custom User Experience Development Solutions Architecting Deployment and Support LAN
- 49. Credit Processor Remote Management / Analytics Media Management Loyalty Enabler Mobile Wallet Enabler TRANSACTION REPORTING POS API Advertising Provider CLOUD SERVICES PRODUCT LINE TERMINAL CONTROL EPS PRODUCT LINE TERMINAL PRODUCT LINE HOST INTERFACE
- 50. DevOps Structure git-repo ├── config │ ├── nz-south-5 │ └── us-north-7 │ ├── dev │ ├── qa │ └── uat │ └── config.yaml ├── infrastructure │ ├── environments │ │ └── us-north-7 │ │ ├── dev │ │ ├── qa │ │ └── uat │ │ └── main.tf │ └── modules │ ├── infrastructure_modules │ │ ├── api-nodejs │ │ ├── cloudfront_distro │ │ ├── queue-worker │ │ └── rds_postgres │ └── resource_modules │ ├── aws_beanstalk_server │ ├── aws_beanstalk_worker │ ├── aws_rds │ ├── aws_route53 │ ├── aws_sqs │ └── aws_vpc └── scripts ├── app_deploy.py └── terraform_wrapper.py
- 51. Terraform Structure Node API Elastic Beanstalk SQS S3 - EB Config - ELB Settings - Role - Queue - Deadletter - Policy - Bucket - Policy DEV || QA || UAT ...Environments Infrastructure Modules Resource Modules
- 52. Cloud Services ● All application and infrastrucure configuration templated using Jinja2 ● app_deploy.py – extracts build artifacts – renders configuration – produces release artifacts – produces deploy wrappers for AWS components
- 53. Cloud Services ● terraform_wrapper.py – renders environment terraform.tfvars.j2 – places environment plan in temp working dir – performs terraform init, plan/apply – produces a summarized change overview
- 54. Cloud Services ● PR initiates a WebHook ● Buid-Server runs terraform_wrapper.py on applicable environments ● Change summary reported back to PR for approval ● Mass-destroys are marked as failure to prevent merging.
- 55. Cloud Services ● In the pipeline: – Tagged Modules – Referencing application configuration from states... ● It is JSON after all! module "aws_lambda_bastion_server" { source = "git::http://our-git-service/foo/module.bar.git?ref=TAG_NAME" }
- 56. Lessons Learnt ● AWS Lambda – Provision == Deploy – Reference common zip in both processes ● Beanstalk – Three step deploy (can be approached better) ● Initial terraform with health checks blank ● Deploy Apps ● Terraform again with health checks updated – Unapplicable settings are ignored by API ● i.e. platform specific settings, worker vs. web server settings ● This results in changes detected EVERY plan
- 57. Lessons Learnt ● Beanstalk – Plenty of settings.... – Worked around by abusing maps. ● Maps have to be flat!!! ● Make them all strings. ● Cast to the required type in the resource module level ● Requires casting back and forth when referencing outputs. ● Nested lists/maps imminent.
- 58. Lessons Learnt ● Don’t deploy to Prod on Fridays...