Self-service PR-based Terraform

@magickatt
Andrew Kirkpatrick
DevOps Toronto, February 2021
Self-service PR-based
automated Terraform
Alexandre Dulaunoy
https://search.creativecommons.org
/photos/50a56587-4db5-4de7-a98e-06730497a283

@magickatt
Infrastructure-as-Code solves some problems, but not all problems...

@magickatt
There are many legitimate reasons why someone might need an
infrastructure change
● Need a VM for application
● Increase database capacity
● New build job
● Create user account
● Update DNS record
● … and so forth

@magickatt
Is it important that someone with infrastructure expertise carry out
the change? 🤷
What is important then? Surely that the change is
● Performed safely and securely
● Tracked against an individual/team/project/division
● Codified in a reproducible stream of changes
● Peer reviewed and approval by chain-of-command

@magickatt
● Infrastructure-as-Code codifies your infrastructure, using
Terraform (HCL) in many cases
● Modules abstract domain-specific* logic into re-usable building
blocks
● Using pre-built building blocks, could you let other people
create/update/delete those resources?
● If so, how would you validate and approve those resources?
* https://en.wikipedia.org/wiki/Domain_(software_engineering)

@magickatt
● View proposed resource
changes ahead of time
(terraform plan)
○ Creation
○ Modification
○ Deletion
● Automatically validate and
re-validate if requested changes
are modified (new commits in a
Pull Request)

@magickatt
● If not making changes
directly, who can and should
approve changes?
● Integrated into issue
tracking, auditable for
compliance
● How and when do proposed
changes occur?

@magickatt
1. Introduction
2. Terraform-at-a-glance
3. Self-service infrastructure
4. Atlantis versus alternatives
5. Pull Request workflows
6. Centralised execution
7. Examples
8. Advantages and Drawbacks
9. Summary

@magickatt
Terraform at-a-glance

@magickatt
● Resources (usually cloud
infrastructure) represented as code
(HCL*) that can be created,
modified or deleted
● Abstract the underlying APIs with a
common-ish syntax
● Different providers (AWS, GCP,
Azure, etc.) offer resources to use
* https://www.terraform.io/docs/configuration/syntax.html, https://github.com/hashicorp/hcl

@magickatt
● Code, what I want to be true
● State, what I last knew to be
true
● APIs, where I can ask for the
truth
Code State
API
Plan
A
p
p
l
y
A
p
p
l
y

@magickatt
Typical Terraform flow is as
follows...
● init
○ Links any Modules being used
○ Download Providers (API clients)
○ Configure Remote State*
● plan
● apply
* Unless using local state. Only use local state for testing purposes

@magickatt
● init
● plan
○ Refresh Remote State
○ Diff between code and state
○ Work out what to create that is present in code and not in state
○ Work out to modify that is different between code and state
○ Work out what to delete that is present in state and not in code
○ Figure out correct order to make changes in
● apply

@magickatt
● plan
● apply
○ plan
○ Attempt to create, update and/or
delete resources in order
○ Update state based on changes

@magickatt
● If multiple users are working with the same Terraform code, you will
need a way to “share” state
● Ensures changes made by different users and by automation are kept
in sync

@magickatt
Self-service infrastructure

@magickatt
● Infrastructure engineers usually
write infrastructure code
● Application engineers usually
write application code
● Often results in the Thrown
Over The Wall anti-pattern*
* https://wiki.c2.com/?ThrownOverTheWall
** https://www.commitstrip.com/en/2016/11/07/which-full-stack-developer-are-you/

@magickatt
This creates a couple problems…
● Infrastructure engineers want x change to
application(s) to prevent y problem with infrastructure
● Application engineers want x change to infrastructure
to allow y feature with application

@magickatt
Infrastructure engineers want x change to application(s) to prevent
y problem with infrastructure...

@magickatt
Application engineers want x change to infrastructure to allow y
feature with application
● JIRA ticket, Slack message, email, tap-on-the-shoulder…
● Could we make a safe self-service process for them to do
it themselves?
● How to validate and approve the changes?

@magickatt
● Writing Terraform from scratch is
daunting!
● How to encapsulate complexity?
● If Terraform Modules* are the
“classes” of HCL**, think about what
users should not be able to modify,
rather than what can they modify?
* https://www.terraform.io/docs/configuration/syntax.html, https://github.com/hashicorp/hcl
** https://www.terraform.io/docs/configuration/modules.html

@magickatt
Terraform Resources can be overly complex with too many arguments,
use Modules as guardrails to simplify
Configuration for Resources in Modules can be...
● Globally configured, for all usages of a Module (not configurable)
● Dynamically configured, based on an argument (indirectly
configurable)
● Individually configured per module usage (directly configurable)

@magickatt
Resource for VM
Module invocation for VM
https://registry.terraform.io/providers/digitalocean/digitalocean/latest/docs/resources/droplet

@magickatt
Module invocation for VM

@magickatt
Atlantis versus alternatives

@magickatt
Atlantis is an application for automating Terraform via pull requests.
It is deployed as a standalone application into your infrastructure.
No third-party has access to your credentials.
Atlantis listens for GitHub, GitLab or Bitbucket webhooks about
Terraform pull requests. It then runs terraform plan and comments
with the output back on the pull request.
https://www.runatlantis.io/guide/#overview-–-what-is-atlantis

@magickatt
● First PR-based Terraform automation
● Runs in a container (Go application)
● Responds to webhooks (GitHub/GitLab/Bitbucket)
● Configurable
○ Multiple repositories
○ Multiple projects per repository
○ Custom workflows

@magickatt
“Terraform Cloud is a platform that performs Terraform runs to provision
infrastructure, either on demand or in response to various events.”
“It manages Terraform runs in a consistent and reliable environment, and
includes easy access to shared state and secret data, access controls for
approving changes to infrastructure, a private registry for sharing
Terraform modules, detailed policy controls for governing the contents of
Terraform configurations, and more.”
https://www.terraform.io/docs/cloud/overview.html
https://www.terraform.io/docs/cloud/index.html#about-terraform-cloud-and-terraform-enterprise

@magickatt
● Cloud-hosted which saves maintenance, can use in conjunction
with on-premise execution (Business plan)
● Can be fully self-hosted* (Enterprise plan)
● Private Terraform Module registry
● Terraform Cloud enhanced backend for Remote State
● Non-trivial to setup Workspaces (UI/API/CLI rather than
configuration)
https://www.hashicorp.com/products/terraform/pricing

@magickatt
Atlantis is only PR-based, whereas Terraform Cloud can be CLI and API driven

@magickatt
Project cannot contain symlinks

@magickatt
Only supports Enhanced Backends, will not read Remote State from anything else

@magickatt
“env0 presents Terraform plans on your SCM pull requests so
you can approve your infrastructure change requests with
maximum confidence, all from within your existing workflows.
Shorten the cycle of pull requests and enable your team to
create pull requests without the need to distribute Terraform
credentials to everyone.”
https://www.env0.com

@magickatt
● Create Organization Templates from VCS* (GitHub, etc.)
● Create a Project Environment from a Project Template (which is
a selection of the available Organization Templates)
● Project Environments use Terraform Workspaces to
differentiate
● Non-trivial to setup Workspaces (UI/API rather than
configuration)
● Relatively new (public beta April 2020) but looks promising!
** https://en.wikipedia.org/wiki/Version_control

@magickatt
Uses long-running or ad-hoc Terraform Workspaces to manage multiple
Environments from a single Project Template

@magickatt
● Does not support remote state (manages state within env0)
● Project Templates created from directory within a VCS
repository, exist persistently as Project Environments
● Uses Terraform Workspaces to manage env0 Environments

@magickatt
env0 and Terraform Cloud support
● Environment variables
● Terraform variables
Atlantis can support
● Environment variables
● Files (including tfvars)
● … anything really

@magickatt
Pull Request workﬂows

@magickatt
● Possible with CircleCI and GitHub Actions
● Have problems that Atlantis, Terraform Cloud and env0 have
since solved
a. Plan and apply synchronisation
b. Approval of plans
c. Which project to plan
“it can be difficult or impossible to ensure that the plan and apply
subcommands are run on the same machine, in the same directory, with all of
the same files present.”
https://learn.hashicorp.com/tutorials/terraform/automate-terraform#plan-and-apply-on-different-machines

@magickatt
“To implement this robustly, it is important to ensure that either only one plan can
be outstanding at a time or that the two steps are connected such that approving a
plan passes along enough information to the apply step to ensure that the correct
plan is applied, as opposed to some later plan that also exists.”
There are 2 problems here…
● terraform plan outfile from a specific commit is supplied to terraform
apply
● Plan is either blocked or overwrites older plan for newer commit
https://learn.hashicorp.com/tutorials/terraform/automate-terraform

@magickatt
“Another challenge with automating the Terraform workflow is the desire
for an interactive approval step between plan and apply.”
When running in CI, how would you implement an approval step?
● You could get it to plan on feature branches and apply on master…
● ...but what if the apply fails? Then you have broken code on master
○ Terraform Cloud applies after merge
○ env0 also applies after merge
● Could use a manual gate on CI pipeline
https://learn.hashicorp.com/tutorials/terraform/automate-terraform#auto-approval-of-plans

@magickatt
● Atlantis responds to 2 comments
○ atlantis plan
○ atlantis apply
● Atlantis will also show feedback via
comments and the PR status check
● Terraform Cloud only provides
feedback on the PR status check
● env0 comments feedback, but will not
respond to comments, has no PR
status check

@magickatt
https://docs.github.com/en/free-pro-team@latest/github/collaborating-with-issues-and-pull-requests/about-status-checks

@magickatt
● Atlantis will lock each Project* so only 1 PR will plan/apply at once
● Terraform Cloud will lock each Workspace (1-n with Projects)
● env0 creates more Workspaces as it uses Project Templates
* Atlantis makes you specify a Workspace per Project in atlantis.yaml configuration

@magickatt
● Atlantis will “lock” a Pull Request (PR) once it has been planned
● New PRs try to obtain locks, will be granted lock in the future
● Locked plan must either be applied (PR approved) or discarded (PR
closed) before lock is released
https://www.runatlantis.io/docs/locking.html

@magickatt
● Approved – requires pull requests to be approved by at least one user other
than the author
● Mergeable – requires pull requests to be able to be merged
Atlantis will only apply if approved/mergeable conditions are met. Terraform Cloud
and env0 will apply after PR is merged

@magickatt
Centralised execution

@magickatt
● Atlantis is deployed into your infrastructure
● Terraform Cloud can run
○ Entirely in the cloud (Free, Team & Governance plans)
○ Control plane in the cloud, Agent Pools deployed into your infrastructure
(Business plan)
○ Fully deployed into your infrastructure (Enterprise plan)
● Env0 runs in the cloud
https://www.runatlantis.io/docs/deployment.html
https://www.terraform.io/docs/cloud/agents/index.html
https://docs.env0.com/docs/about-env0

@magickatt
● Big workflow change is getting Terraform
to use credentials from a robot account
than a real user account (you on your
workstation)
● Most Terraform providers can be
configured purely via environment
variables
● Different env var combinations can
determine how
authentication/authorisation is handled

@magickatt
● Need to supply credentials for
every Terraform Provider you
wish to use
● What permissions should each
service account, access key or
API token have?
● What Projects/Workspaces
should have access to which
variables/credentials?

@magickatt
If your Terraform workflow is non-standard in any way, how would you
accommodate this?
● Atlantis supports Custom Workflows and a customised
installation/container image
● Terraform Cloud will allow standalone binary execution, and discourages
provisioner usage
● env0 supports Custom Flows including multiple language support
https://www.runatlantis.io/docs/custom-workflows.html
https://www.terraform.io/docs/cloud/run/install-software.html
https://docs.env0.com/docs/custom-flows

@magickatt
Advantages and Drawbacks

@magickatt
● Allows contributions without needing to run Terraform locally
● Peer review before execution
● Ties into other workflow automation (JIRA for example)
● Potentially decreases credential theft as an attack vector
● Can alleviate “DevOps” bottlenecks, but depends on...
○ Your peer review allocation process
○ Documentation for existing Terraform usage
○ Readability of existing Terraform code

@magickatt
● Slow feedback cycle
● Less ideal for module development/prototyping
● Moves security controls from Cloud IAM* to VCS IAM
● Could become a skeleton key for your infrastructure
● Maintenance of yet another tool
● Can be problematic if run on same infrastructure that it
controls
* https://en.wikipedia.org/wiki/Identity_management

@magickatt
● Open Source, free to use
● Frequent contributions,
well-maintained
● Simplicity/easier-to-use
● Can inject file-based
configuration (useful for multiple
k8s clusters)
● Custom workflows allow it to do
anything weird you do on your
workstation

@magickatt
● Open source, free-to-use
● Customisable workflow and execution
● On-premise*
● Post-approval pre-merge apply
● Cloud-hosted
● Temporary environments, per user
quotas
● Fairly new
● Cloud-hosted or on-premise
● Many more features and integrations
● Less feedback via on Pull Request
● Approvals can happen outside of Pull
Requests (via Terraform Cloud)
* I don’t mean that it’s not in a cloud somewhere, I just mean your cloud 😛

@magickatt
● Infrastructure Engineer at PartnerStack (https://jobs.lever.co/partnerstack)
● Any questions?

Self-service PR-based Terraform

More Related Content

What's hot (20)

Similar to Self-service PR-based Terraform (20)

Recently uploaded (20)

Self-service PR-based Terraform