PuppetConf 2016: Nice and Secure: Good OpSec Hygiene With Puppet! – Peter Souter, Puppet

Nice and Secure: Good OpSec
Hygiene with Puppet!
Peter Souter
Professional Services Engineer | Puppet
@petersouter

@petersouter 2
Who
am I?
@petersouter
Professional Services
Engineer
5 years using Puppet
2 years @ Puppet Inc
Help customers deploy
Puppet Enterprise
Teach Puppet classes
petems
IRC/Slack/GitHub

@petersouter
Warning: I speak quickly
And I have a different accent...
3

@petersouter
My feelings on Q&A
http://bit.ly/why_no_talk_qa
● Tweet me @petersouter
● Come up after this talk
● Meet me in the hallway
4
When will this QA
be over so I can
leave?
We’ve got lots to cover - No Q&A!

@petersouter
So, why are we here?
(This room specifically, listening to this talk...)
5

@petersouter 6
Every time someone uses this picture,
Pete Cheslock gets his wings!
https://twitter.com/petecheslock/status/595617204273618944

@petersouter
Show of hands in the room
Let’s take the temperature of security here
7

@petersouter
Why is Puppet good for security?
Infrastructure as code RBAC Auditing Enforcement
8

@petersouter
Don’t let Puppet be the attack vector!
aka. How do we make sure we’re not pushing the
problem elsewhere?
9

@petersouter
What is OPSEC?
Before we talk about something we should define it
10

@petersouter
“Operations Security, or OPSEC, is the process by which
we protect unclassified information that can be used
against us. OPSEC challenges us to look at ourselves
through the eyes of an adversary (individuals, groups,
countries, organizations). Essentially, anyone who can
harm people, resources, or mission is an adversary.”
11
Department of Defense Education Activity
http://www.dodea.edu/offices/safety/opsec.cfm

@petersouter 12
● Keeping your code clear of
sensitive information
● Approaches to secrets
management with the Puppet
toolchain
● Making sure security is part of
your workflow, rather than an
afterthought
What are we going to cover?
https://flic.kr/p/7LcF2W

@petersouter
Let’s start with secrets...
13
We’ve all got them...

@petersouter 14
What are secrets in IT?
Radioactive
Consequences are
dire from a leak
Examples
Passwords, API
Keys, SSH Keys,
SSL Certs...
Small
A few kb at most
Required
The infrastructure
won't work without
them!
https://flic.kr/p/dHrwpb

@petersouter
Easiest to hardest
● Avoid exposing secrets in logs
● Remove data from code and into
the data layer (hiera)
● Encryption
15
How do we avoid exposing secrets in Puppet?
https://flic.kr/p/aCJZrf

@petersouter
Don’t expose secrets in logs
Keep your secrets hidden
16

@petersouter
show_diff
The first place for leaks
17

@petersouter 18
root@homebox:~# puppet agent --show_diff
Notice: Compiled catalog for homebox.home in environment production in 0.10 seconds
Notice: /Stage[main]/Main/File[/etc/sensitive]/content:
--- /etc/sensitive 2016-08-14 23:01:37.036863915 +0100
+++ /tmp/puppet-file20160814-24654-ak1ywd 2016-08-14 23:01:56.852882307 +0100
@@ -1 +1 @@
-Not Secret
No newline at end of file
+SECRET-CONTENT
No newline at end of file
Notice: /Stage[main]/Main/File[/etc/sensitive]/content: content changed
'{md5}2ab96390c7dbe3439de74d0c9b0b1767' to '{md5}44c7be48226ebad5dca8216674cad62b'
Notice: Applied catalog in 0.20 seconds
How it looks...

@petersouter
Anywhere reports go:
● syslog
● interactive terminal output
● PE Console
● ENC
● report processors
19
Where does the information from show_diff go?

@petersouter 20
file { ‘/etc/secrets.txt’:
ensure => 'file',
owner => 'root',
mode => '0600',
content => 'hunter2',
show_diff => false,
}
Setting show_diff to false at the resource level

@petersouter 21
An example from a Supported Module: mysql
file { "${::root_home}/.my.cnf":
content => template('mysql/my.cnf.pass.erb'),
owner => 'root',
mode => '0600',
}
# show_diff was added with puppet 3.0
if versioncmp($::puppetversion, '3.0') >= 0 {
File["${::root_home}/.my.cnf"] { show_diff => false }
}
https://github.com/puppetlabs/puppetlabs-mysql/blob/d58a100fa67bc99b4388d4ea3921b11647d483d7/manifests/server/root_password.pp#L39

@petersouter
Setting show_diff to false at resource scope
show_diff = false
22
root@homebox:~# puppet apply secret.pp
Notice: Compiled catalog for homebox.home in environment production in 0.10 seconds
Notice: /Stage[main]/Main/File[/etc/sensitive]/content: content changed '{md5}d3b07384d113edec49eaa6238ad5ff00' to
'{md5}44c7be48226ebad5dca8216674cad62b'

@petersouter
There’s a balance...
Hiding diffs reduces visibility of change...
23

@petersouter
inifile module to hide changes
Allows you to only hide the sensitive fields
24

@petersouter 25
ini_file now has show_diff from the 1.5.0 release

@petersouter
Setting show_diff on individual sensitive fields
show_diff = false
26
ini_setting { 'ACME App Timezone':
section => 'TimeDate',
setting => 'TimeZone',
value => $acme_app_time_zone,
}
ini_setting { 'ACME App Password:
section => 'Settings',
setting => 'Password',
value => $acme_app_password,
show_diff => false,
}

@petersouter
Sensitive type
New for the Puppet 4.6+ release
27

@petersouter 28
file { '/etc/sensitive':
ensure => 'present',
owner => 'root',
group => 'root',
content => Sensitive('hunter2'),
}
root@homebox:~# puppet apply secret.pp
Notice: /Stage[main]/Main/File[/etc/sensitive]/ensure: changed [redacted] to [redacted]
Ability to redact strings with the new Sensitive Type

@petersouter 29
$secret = Sensitive(‘Unwrapped’)
$unwrapped = $secret.unwrap |$sensitive| { $sensitive }
notice("Unwrapped: ${unwrapped}")
$secret.unwrap |$sensitive| { notice("Lambda: ${sensitive}") }
Unwrapping the secrets
https://www.devco.net/archives/2016/09/05/puppet-4-sensitive-data-types.php

@petersouter 30
You can use a dedicated redacted resource
Still on < 4.6?
30

@petersouter 31https://github.com/openstack/puppet-barbican/blob/2e2b10ae58fdc9ad27d88d3195260ef02af853ad/lib/puppet/type/barbican_config.rb
newproperty(:value, :array_matching => :all) do
desc 'The value of the setting to be defined.'
munge do |value|
value = value.to_s.strip
value.capitalize! if value =~ /^(true|false)$/i
value
end
newvalues(/^[S ]*$/)
def is_to_s( currentvalue )
if resource.secret?
return '[old secret redacted]'
else
return currentvalue
end
end
def should_to_s( newvalue )
if resource.secret?
return '[new secret redacted]'
else
return newvalue
end
end

@petersouter 3232
Encrypt secrets on a node by node basis
Try binford2k-node_encrypt
32

@petersouter 33https://github.com/binford2k/binford2k-node_encrypt
● Master encrypts secrets for each node using their own certificate
● Secret can only be decrypted with the node's private key
● Uses built-in Puppet CA, so the base case is zero-config
node_encrypt::file {'/etc/company_app/credentials':
ensure => file,
owner => 'root',
content => 'hunter2', # transparently encrypted
}
How does it work?

@petersouter
node_encrypt
34
$ puppet agent -t
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for master.puppetlabs.vm
Info: Applying configuration version '1450109738'
Notice: /Stage[main]/Main/Node[default]/Node_encrypt::File[/tmp/foo]/Node_encrypted_file[/tmp/foo]/ensure: created
$ echo blah > /tmp/foo
$ puppet agent -t
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for master.puppetlabs.vm
Info: Applying configuration version '1450109821'
Notice: /Stage[main]/Main/Node[default]/Node_encrypt::File[/tmp/foo]/Node_encrypted_file[/tmp/foo]/content: content changed '<<encrypted>>' to
'<<encrypted>>'

@petersouter
So data is no longer exposed in logs
But the data is still visible in the code!
35

@petersouter
Remove data from code
Especially organisation specific data
36

@petersouter
Bad!
Don’t do this...
37

@petersouter 38
class example_company_app {
if $::fqdn == 'prod.example.com' {
class {'company_app':
password => 'hunter2',
ssl_enable => true,
}
} else {
password => 'example123',
ssl_enable => false,
}
}
}

@petersouter
Good!
Do this...
39

@petersouter 40
class profile::example_company_app {
$app_password = hiera('profile::example_company_app::password')
$app_ssl_enable_password = hiera('profile::example_company_app::password')
password => $app_password,
ssl_enable => $ssl_enable,
}
}

@petersouter
Roles and profiles help a lot
Abstracting implementation specifics away
41

@petersouter
Allows you to set organisational defaults in your
roles and profiles
42
● Keep organisational specific data in hiera
● Move organisational specific setup into role and profile wrappers
Advantage:
Not only more secure, cleaner code that’s more reusable!

@petersouter
Default parameters are important!
Abstracting implementation specifics away
43

@petersouter
Storytime: Openstack
The perils of bad defaults...
44

@petersouter 45
https://archive.fosdem.org/2015/schedule/event/public_puppet/
The open source OpenStack
project infrastructure
Fully public Puppet

@petersouter
The more abstracted your
control-repo, the less chance of
leaking or compromising of secrets...
46

@petersouter
Another gotcha: hierarchy lookups
Aka. Why trusted facts are good!
47

@petersouter
Take a look at this hiera config...
48
# hiera.yaml
---
:hierarchy:
- "node/%{fqdn}"
- "common"
:backends:
- yaml
:datadir: '/etc/puppet/environments/%{environment}/hieradata'

@petersouter
Facts are spoofable!
49
[root@testbox]# facter fqdn
pe-201620-master.puppetdebug.vlan
[root@testbox]# FACTER_fqdn=evil.example.com facter fqdn
evil.example.com
Facts are spoofable

@petersouter
Trusted facts got your back!
50
Locked in from the certificate request

@petersouter
Trusted facts are stamped on Node
creation
51
Trusted facts are stamped on Node creation

@petersouter
We have a bunch of OIDs for this also...
52https://docs.puppet.com/puppet/latest/reference/ssl_attributes_extensions.html

@petersouter
New, better hiera hierarchy...
53
# hiera.yaml
---
:hierarchy:
- "node/%{trusted.certname}"
- "common"
:backends:
- yaml
:datadir: '/etc/puppet/environments/%{environment}/hieradata'

@petersouter
Theoretically, you should be
able to release most of the
code you write publically
without any sort of security
issues
54

@petersouter 55
This is actually a tenet of
12 Factor Apps...
Apps sometimes store config as constants in the code. This is a violation of
twelve-factor, which requires strict separation of config from code. Config varies
substantially across deploys, code does not.
A litmus test for whether an app has all config correctly factored out of the
code is whether the codebase could be made open source at any moment,
without compromising any credentials.
Note that this definition of “config” does not include internal application config, such
as config/routes.rb in Rails, or how code modules are connected in Spring. This
type of config does not vary between deploys, and so is best done in the code.
http://12factor.net/config

@petersouter
Example: GDS
Government Digital Service, UK
56

@petersouter 57
Meeting the Digital Service Standard
To meet point 8 (understand security and privacy issues) you must:
● Make all new source code open and reusable
● Publish code under an appropriate licence
● Explain your reasoning for any code you haven’t made open
You’ll have to explain how you did this at your service assessments.
https://www.gov.uk/service-manual/technology/making-source-code-open-and-reusable

@petersouter 58
Meeting the Digital Service Standard
When GOV.UK was first set up we were unable to publish our Puppet
repository because our code and secrets were tied together. This goes
against patterns like the 12-factor app which “requires strict separation
of config from code”
This wasn’t true for our Puppet repository, but we gradually moved our
credentials into a separate repository (rotating them as we did so).
“A litmus test for whether an app has all config correctly factored out of the
code is whether the codebase could be made open source at any moment,
without compromising any credentials.”

@petersouter 59
$ strings modules/**/*.pp | tr ' '
'n' | sort -n | uniq | view -
Check code for unique strings that look secret-y
Note: Requires zsh for the strings function!

@petersouter
It’s not just Puppet code!
Git commits can sustain sensitive data!
60

@petersouter 61
$ git commit -a -m "Changed the
password to password1"

@petersouter
$ while read line; do echo $line;
git --no-pager log -p -S $line; done
< puppet_search
62
Manually searching through git commits for
sensitive information...

@petersouter
Opening GOV.UK’s Puppet Repository
https://gdstechnology.blog.gov.uk/2016/01/19/opening
-gov-uks-puppet-repository/
Git Repo https://github.com/alphagov/govuk-puppet
Want to know more?
63

@petersouter
Your data is now separated. Hooray!
But it’s still plaintext in Hiera. Boo! :-(
64

@petersouter
Encryption
Told you we’d come back to it!
65

@petersouter
Bad!
Don’t do this!
66

@petersouter
Good!
Do this!
68

@petersouter 69
Preso title goes here. To update, go to File > Page Setup > Header/Footer, paste title, Apply All

@petersouter 70https://github.com/TomPoulton/hiera-eyaml
Hiera eyaml

@petersouter
hiera-eyaml is probably the best method
for internal data encryption with Puppet
It’s widely used, and has a number of plugins
71

@petersouter
eyaml plugins
72
● https://github.com/sihil/hiera-eyaml-gpg
● https://github.com/tehmaze/hiera-eyaml-secretbox
● https://github.com/acidprime/hiera-eyaml-pkcs11
● https://github.com/adenot/hiera-eyaml-kms
● https://github.com/gtmtechltd/hiera-eyaml-twofac

@petersouter
The idea is that Puppet will natively
support encrypted data in the future
Follow this ticket for the roadmap view: PUP-1974
73

@petersouter
● Transcrypt
Git-Crypt
Blackbox
● Turtles All The Way Down:
Storing Secrets in the Cloud
and in the Data Center
behind Closed Doors
VCS based encryption
74
http://danielsomerfield.github.io/turtles
https://www.youtube.com/watch?v=OUSvv2maMYI

@petersouter
Dedicated secret devices
Going deeper...
75

@petersouter
Why use a secret server?
76
● Dynamic secrets
● ACL (Access control policies)
● Leasing and renewal
● Revocation
● Encryption
● Auditing
● Supportability

@petersouter 77
Conjur, Vault, Keywhiz, Amazon KMS,
Confidant

@petersouter 78
Hiera will plug into any secret service
app with a little bit of Ruby glue
Hiera is just key/value lookup
A hiera backend to basically any secret server setup is possible

@petersouter 79
$planet = conjur_variable('planet')
file { '/etc/hello.txt':
content => "Hello ${planet}!n"
}
conjurize_file { '/etc/hello.txt':
variable_map => {
planet => ‘!var puppetdemo/planet’
}
}
https://www.conjur.net/puppet-secret-server
https://forge.puppet.com/conjur/conjur
Conjur and Puppet

@petersouter 80https://github.com/jsok/hiera-vault
hiera-vault

@petersouter
If you want to know more about
Puppet + Vault, Seth Vargo from
Hashicorp is presenting tomorrow
81

@petersouter
Behind Closed Doors - Managing Passwords
in a Dangerous World by Noah Kantrowitz
● Really great in-depth presentation
● https://coderanger.net/talks/secrets/
● https://www.youtube.com/watch?v=TVEfY
O-5-RA
● Great breakdown of secret management,
advantages and disadvantages of approaches
and tooling
Want to know more about secrets?
82

@petersouter
Cleaning up the current codebase
83
How to find secrets currently exposed

@petersouter
--------------------------------------------------------------------
gittyleaks' Bot Detective at work ...
--------------------------------------------------------------------
file: site/profiles/templates/rhn/RHN-ORG-TRUSTED-SSL-CERT.erb
what: Key
value: (2048
match:
Public-Key: (2048 bit)
num_of_revisions: 59
Gittyleaks
85https://github.com/kootenpv/gittyleaks

@petersouter
Scumblr
86https://github.com/Netflix/Scumblr

@petersouter
Unfortunately, there’s no silver bullet to
detect leaked credentials or
proper usage of encryption...
87

@petersouter
A lot of it is about process,
gating and reviews
88

@petersouter 89
Sometimes the job is too big for a repository, and it’s
better to migrate to a cleaner repo

@petersouter
Ensuring it stays clean
90
After cleanup, ensuring it stays clean

@petersouter 91
At a minimum, you want to make sure
that what you’re doing with Puppet isn’t
making things worse!

@petersouter 92
It’s largely a people and process problem

@petersouter
Making sure security is part of your
workflow, rather than an afterthought
“Shift security left”
93

@petersouter 94
“To keep up with the pace of Continuous Delivery, security must “shift
left,” earlier into design and coding and into the automated test cycles,
instead of waiting until the system is designed and built and then trying to
fit some security checks just before release. In DevOps, security must fit
into the way that engineers think and work: more iterative and
incremental, and automated in ways that are efficient, repeatable, and
easy to use.”
- DevOpsSec: Delivering Secure Software Through Continuous Delivery, Jim Bird
Shifting left!

@petersouter 95
How do we check things aren’t getting worse?
● Game days and internal evil attempt teams
● Continuous security integration (Gittyleaks/code-review)
● Dedicated security stories for sprints
○ Evil users or (mis)use cases
● Embedded security team members
● Dedicated audits on sensitive apps/stacks from external firms

@petersouter 96
Game Day example: Agent spoofing
Let's say someone gets access to an agent.
What’s the worst they can do?

@petersouter
As soon as security becomes a blocker,
you’ve lost!
Security has to be automated where possible, otherwise
we’re back to the throw-over-the-wall problems of
pre-DevOps!
97

@petersouter
Summary
What have we learnt?
98

@petersouter
Remove sensitive data from your logs
Use the new sensitive type or write custom providers
99

@petersouter
Use the roles and profiles pattern for
original defaults
Separate secrets, reduce the surface area for problematic setups and always
read the docs before using Puppet modules
100

@petersouter
Use Trusted Facts in your hiera hierarchy
Make sure hiera lookups can’t spoofed
101

@petersouter
Encrypt the separated data
Hiera-eyaml or a dedicated secret server
102

@petersouter
Ensure your code stays clean
People, processes and automated testing
103

@petersouter
Move security left
Make it a part of your process, rather than an afterthought
104

PuppetConf 2016: Nice and Secure: Good OpSec Hygiene With Puppet! – Peter Souter, Puppet

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to PuppetConf 2016: Nice and Secure: Good OpSec Hygiene With Puppet! – Peter Souter, Puppet (20)

More from Puppet (20)

Recently uploaded (20)

PuppetConf 2016: Nice and Secure: Good OpSec Hygiene With Puppet! – Peter Souter, Puppet