Head in the Clouds: Testing Infra as Code - Config Management 2020

Copyright © 2020 HashiCorp
Head in the Clouds: Testing
Infrastructure-as-Code
Conﬁg Management Camp 2020
Peter Souter
Sr. Technical Account Manager - HashiCorp

Introductions
Who is this bloke?
Technical Account Manager at HashiCorp
Peter Souter
Based in...
London, UK
This is my 6th CfgMgmentCamp!
Been coming since 2014 (missed 2018 - 👶)
Worn a lot of hats in my time...
Developer, Consultant, Pre-Sales, TAM
Interested in...
Making people’s operational life easier and
more secure
DEVOPS ALL THE THINGS

What are we here to talk about?

Treating the tooling that manages
your infrastructure with the same
way you would treat any other
code.

1. Easy to regenerate from scratch
2. Code review and collaboration
3. Easier to conceptualise as a code model
4. Auditing and policies
5. Iteratively improve over time
6. Modularise, reuse and hide complexity
7. Testing
Qui bono?
Who beneﬁts?

“Why should we test our
infrastructure as code anyways?”

“The concern of velocity loss of added testing is very much
worthwhile in my experience. It’s why we have a known, stable
configuration and deployment codebase and can move on to more
relevant business problems. In their absence, I’ve without
exception had to firefight tooling, silent regressions, and human
error burning out teams and reducing confidence in automation
from across teams. Infrastructure as code without testing to me
is self-sabotage”
- Devon Kim, @djk29a, HashiConf 2019 Slack

Lets have
a brief history
lesson...

And before we start,
let me give my
caveats...

I’m not on the engineering team!
Nothing I say is gospel, nor can I
commit to any future roadmap

Conﬁguration management tools install and
manage software on a machine that already
exists. Terraform is not a conﬁguration
management tool, and it allows existing
tooling to focus on their strengths:
bootstrapping and initializing resources.
- https://www.terraform.io/intro/vs/chef-puppet.html

Conﬁg Management Camp 2020?
Infra

Head in the Clouds: Testing Infra as Code - Config Management 2020

“How do we test the
cloud?”

Two types of people interested in
testing IaC and it’s interactions
with DA CLOUD: Producers and
Consumers

A little background on
testing...

https://martinfowler.com/articles/practical-test-pyramid.html

Unit Testing
Acceptance
Testing
Integration
Testing

The terraform seemed to
generate the environment
correctly last time I tried…
meh, ship it!

Let’s eat the cone from
the bottom...

Let's start with the
most simple “tests”...

TERMINAL
Chef: Ruby syntax check
$ ruby -c my_cookbook_file.rb
Syntax OK!
TERMINAL

TERMINAL
Terraform: validate
$ terraform validate
Error: Unsupported block type
on main.tf line 30, in resource "aws_instance" "foobar":
30: tags {
Blocks of type "tags" are not expected here. Did you mean to define argument "tags"?
If so, use the equals sign to assign it a value.
TERMINAL

TERMINAL
Ansible: --check
$ ansible-playbook apache.yml --check
ERROR! Syntax Error while loading YAML.
did not find expected '-' indicator
TERMINAL

TERMINAL
Puppet: parser validate
$ puppet parser validate
Error: Could not parse for environment production:
Syntax error at 'owner' at /tmp/foo.pp:5:5
TERMINAL

Terraform: tflint
TERMINAL
$ tflint
1 issue(s) found:
Error: instance_type is not a valid value (aws_instance_invalid_type)
on main.tf line 28:
28: instance_type = "t9.micro"
https://github.com/wata727/tﬂint

Chef: foodcritic
TERMINAL
$ foodcritic -C ./cookbooks/mysql
cookbooks/<cookbook Name>/attributes/server.rb
FC002: Avoid string interpolation where not required
85| default['<Cookbook Name>']['conf_dir'] = "#{mysql['basedir']}"
https://docs.chef.io/foodcritic.html

Puppet: puppet-lint
TERMINAL
$ puppet-lint /etc/puppet/modules
foo/manifests/bar.pp - ERROR: trailing whitespace found on line 1
apache/manifests/server.pp - WARNING: variable not enclosed in {} on line 56
http://puppet-lint.com/

Ansible: ansible-lint
TERMINAL
$ ansible-lint geerlingguy.apache
[ANSIBLE0013] Use shell only when shell functionality is required
/Users/chouseknecht/.ansible/roles/geerlingguy.apache/tasks/main.yml:19
Task/Handler: Get installed version of Apache.
[ANSIBLE0011] All tasks should be named
/Users/chouseknecht/.ansible/roles/geerlingguy.apache/tasks/main.yml:29
Task/Handler: include_vars apache-22.yml
https://docs.ansible.com/ansible-lint/

Ok, so now lets get deeper:
Unit Tests

Acceptance tests make sure that
you're building the right thing
Unit tests make sure that you're
building the thing right
https://stackoverflow.com/questions/4139095/unit-tests-vs-acceptance-tests

A unit tests is for the logic you have
written, not to test the language

Don’t just write
tautological unit tests,
test the edge cases

Producers and consumers
have diﬀerent intents for
unit-tests

Puppet - https://rspec-puppet.com/
TERMINAL
context 'repo enabled' do
context 'on CentOS' do
let(:facts) do
{
:operatingsystem => 'CentOS',
}
end
let(:params) {{ 'manage_repo' => true }}
it { is_expected.to contain_class('fish::Repo::Centos')}
it do
is_expected.to contain_yumrepo('shells_fish_release_2').with(
:descr => 'Fish shell - 2.x release series (CentOS_7)',
:baseurl => 'https://download.opensuse.org/repositories/shells:/fish:/release:/2/CentOS_7/',
:enabled => '1',
:gpgcheck => '1',
:gpgkey => "https://download.opensuse.org/repositories/shells:/fish:/release:/2/CentOS_7/repodata/repomd.xml.key",
)
end
end

Chef - https://docs.chef.io/chefspec.html
TERMINAL
context 'amazonlinux' do
cached(:chef_run) { ChefSpec::SoloRunner.new(platform: 'amazon', version: '2018.03').converge(described_recipe) }
it 'creates yum_repository[amzn-main]' do
expect(chef_run).to create_yum_repository('amzn-main')
end
it 'creates yum_repository[amzn-main-debuginfo]' do
expect(chef_run).to create_yum_repository('amzn-main-debuginfo')
end
it 'creates yum_repository[amzn-nosrc]' do
expect(chef_run).to create_yum_repository('amzn-nosrc')
end
end

Unit testing for Terraform:
Nothing oﬃcial yet...

The Future…
Core Unit Testing?
https://github.com/hashicorp/terraform/issues/21628

Possibly… but we have a big backlog!

In the meantime, some
possible contenders...

clarity
https://github.com/xchapter7x/clarity/tree
▪ Self-contained binary
▪ Gherkin style features
▪ Behaviour-Driven Development
▪ Parses HCL for running its checks
▪ Provides its own Terraform speciﬁc
matchers

clarity example
TERMINAL
$ clarity single_instance.feature
Feature: Single Instance in AWS
Scenario: Creation of a single AWS micro instance # single_instance.feature:3
Given Terraform # terraform.go:76 -> *Match
And a " aws_instance" of type " resource" # terraform.go:242 -> *Match
Then attribute " instance_type" equals "t1.micro" # terraform.go:351 -> *Match
And it occurs exactly 1 times # terraform.go:489 -> *Match
1 scenarios ( 1 passed)
4 steps (4 passed)
1.613589ms

terraform-compliance
https://github.com/eerkunt/terraform-compliance
▪ Python CLI application
▪ Gherkin style features
▪ Behaviour-Driven Development
▪ Parses plan outputs for it’s checks
▪ Provides its own Terraform speciﬁc
matchers

terraform-compliance example
TERMINAL
$ terraform-compliance -p plan.out -f terraform_compliance/
terraform-compliance v1.0.51 initiated
Scenario Outline: Ensure that specific tags are defined
Given I have resource that supports tags defined
When it contains tags
Then it must contain <tags>
And its value must match the "<value>" regex
Examples:
| tags | value |
| Name | .+ |
1 features (1 passed)
1 scenarios (1 passed)
4 steps (4 passed)
Run 1570975178 finished within a moment

terraform_validate
https://github.com/elmundio87/terraform_validate
▪ Written as Python
▪ Standard unit testing
▪ Parses HCL for it’s checks
▪ Provides it’s own Terraform
speciﬁc matchers

terraform_validate example
TERMINAL
$ python3 test/*.py
+ terraform_validate
======================================================================
FAIL: test_tags (__main__.TestResources)
Checks resources for required tags.
----------------------------------------------------------------------
Traceback (most recent call last):
File "test/terraform_validate_tests.py", line 27, in test_tags
should_have_properties(required_tags)
File
"/usr/local/lib/python3.7/site-packages/terraform_validate/terraform_validate.py",
line 207, in should_have_properties
raise AssertionError("n".join(sorted(errors)))
AssertionError: [ aws_instance.foobar.tags] should have property: 'Date'
----------------------------------------------------------------------
Ran 1 test in 0.031s
FAILED (failures=1)

Pros
Fast to run
Easy to write
No environment or
credentials required
Cons
Doesn’t test actual outcome
IaC Unit
Testing

Ok, now the hard part:
Integration and Acceptance

People use
them interchangeably...

https://stackoverflow.com/questions/4904096/whats-the-difference-between-unit-functional-acceptance-and-integration-test
Acceptance tests are an external client that encompasses
enough information on how to drive the system under test (SUT)
in order to bring it to the point of asserting something.
Integration tests on the other hand, are used for testing the
internals of the system and are geared towards validating
contracts between modules of code.

http://blog.jonasbandi.net/2010/09/acceptance-vs-integration-tests.html

Either way:
How can we test against
“DA CLOUD”, something we don’t
directly control?

Approach 1:
Mock the API’s with ﬁxtures

https://docs.pytest.org/en/latest/fixture.html
The purpose of test fixtures
is to provide a fixed baseline
upon which tests can reliably
and repeatedly execute.

These are often
pretty close to unit tests...

ACME Provider (Lets Encrypt)
https://www.terraform.io/docs/providers/acme/index.html
▪ We don’t want to ﬁre
requests at a real ACME
CA provider
▪ Instead use a ﬁxture
▪ Golang httptest

ACME Provider - Fixture mocking example

Pros
No dependency on cloud
Quick to run
Cons
Can be a lot of work to
maintain ﬁxtures
Won’t test drift or changes in
actual cloud
Fixture
Mocking

With the rise of Serverless/Lamba
frameworks, devs needed
something local to test against and
people started making Cloud
Simulators

Azurite
https://docs.microsoft.com/en-us/azure/storage/co
mmon/storage-use-azurite
DynamoDB Local
https://docs.aws.amazon.com/amazondynamodb/l
atest/developerguide/DynamoDBLocal.html
GCP Emulator
https://cloud.google.com/sdk/gcloud/reference/be
ta/emulators/

Localstack
https://localstack.cloud
▪ “Cloud in a box”
▪ Can run in Docker
▪ Provides real working
versions of AWS APIs

Pros
No dependency on cloud
More debugging and “real
life” experience than a
mocked service
Cons
Won’t exist for every service
Relies on simulated service
to keep in line with real
cloud
Still not _really_ testing an
actual Cloud service
Fixture
Mocking

Approach 3:
Don’t mock anything, do it for real!

Pull-request acceptance tests for Azure Provider

Full Suite example for Azure Provider:

From a producer point of view,
you’re testing the tool

As a consumer, you probably want
an integration test of your real full
stack of pieces

Ultimately,
you care
about state

terratest
https://github.com/gruntwork-io/terratest
▪ Full acceptance framework for
Terraform
▪ Developed by Gruntworks
▪ Used to test large module
deployments
▪ Written in Golang
▪ Helper methods for validating
state

At Gruntwork we test dozens of modules with terratest. We have
hooks to CircleCI to run tests on each commit. For us, the ROI is
huge - we've caught many regressions & problems thanks to
the tests.
It can definitely result in long test times. In some of our larger
repos, test can take 45-60 minutes. What I normally do is
add/update a test, run just that test locally to make sure it's
working, then let the full suite run in CircleCI.
- Ben Whaley, Gruntworks, HashiConf 2020 Slack

terratest example
TERMINAL
$ go test -v test/terraform_basic_example_test.go
=== RUN TestTerraformAWSInstanceType
=== PAUSE TestTerraformAWSInstanceType
=== CONT TestTerraformAWSInstanceType
TestTerraformAWSInstanceType 2020-10-13T16:09:10+01:00 retry.go:72: terraform [init -upgrade=false]
TestTerraformAWSInstanceType 2020-10-13T16:09:10+01:00 command.go:87: Running command terraform with args
[init -upgrade=false]
TestTerraformAWSInstanceType 2020-10-13T16:09:21+01:00 command.go:158: aws_instance.foobar: Creating...
TestTerraformAWSInstanceType 2020-10-13T16:09:31+01:00 command.go:158: aws_instance.foobar: Still
creating... [10s elapsed]
TestTerraformAWSInstanceType 2020-10-13T16:09:41+01:00 command.go:158: aws_instance.foobar: Still
creating... [20s elapsed]
TestTerraformAWSInstanceType 2020-10-13T16:09:43+01:00 command.go:158: aws_instance.foobar: Creation
complete after 21s [id=i-08d65b4371c14fc4e]
TestTerraformAWSInstanceType 2020-10-13T16:09:43+01:00 command.go:158:
TestTerraformAWSInstanceType 2020-10-13T16:09:43+01:00 command.go:158: Apply complete! Resources: 1
added, 0 changed, 0 destroyed.
TestTerraformAWSInstanceType 2020-10-13T16:09:43+01:00 command.go:158: Outputs:
TestTerraformAWSInstanceType 2020-10-13T16:09:43+01:00 command.go:158: instance_type = t1.micro
ok command-line-arguments 84.607s

Cons
Slower
Can be Costly
Can interfere with real
infrastructure
Pros
End-to-end testing
Interacts with real APIs
Closest to the real thing
Acceptance
and
Integration
Testing

Beyond conventional
testing...

Not going to go too deeply
into this...

Is it working beyond our
test suite?

https://www.weave.works/blog/gitops-part-3-observability

For more information...
https://www.youtube.com/watch?v=fOdtgHu_KeA&list=PLgeeFA2rwFRMIblCQi_8BkXyZALQSjSSO&index=3&t=0s
From Cfgmgmtcamp 2019

Every organisation has policies
Naming conventions, tagging, price
guides, cost centres and limits, legal
requirements and regulations etc.
Generally under the umbrella of
Governance

If we can reﬂect
Policies-as-code
we get the same
beneﬁts we get from
Infrastructure-as-code

HashiCorp Sentinel
https://www.hashicorp.com/sentinel/
“An embeddable policy as code
framework to enable ﬁne-grained,
logic-based policy decisions that can
be extended to source external
information to make decisions.”

TERMINAL
Sentinel example for Terraform
CODE EDITOR
import "tfplan"
main = rule {
all tfplan.resources.aws_security_group as _, instances {
all instances as _, sg {
all sg.applied.egress as egress {
egress.cidr_blocks not contains "0.0.0.0/0"
}
}

The beneﬁts of testing IaC
Regardless of what tool you’re using, Infrastructure-as-code without testing
is “self-sabotage”

Unit Testing for IaC
Allowing quick feedback loops and TDD

Acceptance and Integration testing for IaC
Allowing you assurance that your IaC ﬁnal state reﬂects what you expect

Beyond conventional testing
Using observability to check things are running ok after deployment...

The beneﬁts of Policy-as-code
How reﬂecting organisational governance as PoC can speed up deployment

Thank You
psouter@hashicorp.com
@petersouter
www.hashicorp.com
13
8

Head in the Clouds: Testing Infra as Code - Config Management 2020

More Related Content

What's hot (20)

Similar to Head in the Clouds: Testing Infra as Code - Config Management 2020 (20)

More from Peter Souter (10)

Recently uploaded (20)

Head in the Clouds: Testing Infra as Code - Config Management 2020