Automating AWS Infrastructure Provisioning Using Concourse and Terraform
- 1. Automating AWS Infrastructure Provisioning Building a CI/CD pipeline with Concourse and terraform Cesar Rodriguez
- 4. 1. Code
- 5. 2. Format > terraform fmt
- 6. 3. Init > terraform init
- 7. 4. Plan > terraform plan
- 8. 5. Apply > terraform apply
- 10. 1.Bottlenecks 2.Testing 3.Source of Truth 4.Credentials 5.Change Control Workflow Challenges
- 12. Pipeline
- 13. Pipeline - Jobs Build Test Provision
- 14. Pipeline - Resources Build Test Provision S3 S3
- 15. Pipeline - Tasks Build Test Provision fmt init plan scan apply S3 S3
- 18. Resources • concourse-pipelines - Collection of concourse pipelines including terraform-pr and terraform-commit pipeline https://github.com/cesar-rodriguez/concourse-pipelines • terrascan - Static code analysis of terraform templates. https://github.com/cesar-rodriguez/terrascan • hello-hug - Example terraform project that uses concourse-pipelines https://github.com/cesar-rodriguez/hello-hug
Editor's Notes
- #3: How many of you are familiar with terraform? - Declarative language. Describe what the target state and terraform figures API calls - Version control - Preview any changes - Consistent infrastructure across all environments
- #4: Terraform workflow from your desktop
- #6: Writes templates into a canonical format. Templates look clean and consistent.
- #7: - Download terraform providers binaries - Downloads any modules - Terraform remote states
- #8: Check terraform templates against your terraform state file and calculates any changes to resources in your templates.
- #9: Execute any changes from the terraform plan into AWS.
- #10: Collaboration challenges. Native features: Remote states State locking
- #11: Challenges with native workflow: Reviewing Pull Requests No automated testing in this workflow, manual inspection. No guarantee that GitHub reflects what’s in production Credentials to AWS environments in our desktop No central place to verify testing was completed. No central audit trail
- #12: What is Concourse? Declarative YAML templates to design pipelines UI to view the pipeline’s workflow Easy to extend its functionality to solve the challenges faced with terraform provisioning at scale. Docker container define tasks within your pipeline. Ensuring repeatability and consistency Integration with GitHub OAuth for authentication/authorization. You can limit access to your pipeline and its secrets only to members of your team in GitHub. Native integration with Hashicorp Vault for pipeline secrets. Secrets are only retrieved at time of use by Concourse and are never persisted.
- #13: There are 4 different concepts within Concourse that define a pipeline. Pipeline - declarative YAML template where you define the inputs and outputs of your CI/CD tasks.
- #14: Jobs - which are a collection of tasks that form our build plans. For terraform provisioning, we’ll have 3 different jobs. 1 for building our infrastructure, 1 for testing the terraform templates, and 1 for provisioning the resources into AWS.
- #15: Resources - inputs and outputs to our jobs
- #16: Tasks - single purpose Docker containers.