DevSecCon Singapore 2018 - Remove developers’ shameful secrets or simply remove shameful developers by Fabian Lim

Join the conversation #DevSecCon
BY Fabian Lim
REMOVE DEVELOPERS’
SHAMEFUL SECRETS
OR SIMPLY REMOVE SHAMEFUL DEVELOPERS…

First thing First!
Materials can be found at:
github.com/
DevSecOpsSG/
devseccon2018
Do the Prerequisite!

github.com/DevSecOpsSG/devseccon2018
Disclaimer
This presentation may or may not contain information about services under GovTech. The
information contained in this presentation is classified as Public.
This presentation and its contents does not represent the views of GovTech, or any other
entities. They are the sole views of the author. I take full responsibility for my work
and any errors fall on my shoulders.
Be happy and awesome; and
help others to be happy and
awesome.

missions:
- energetic DevSecOps Engineer and Evangelist
- physical and cyber security educator
education:
- Singapore Management University, BS Info System
- Carnegie Mellon University, MS Info Security Policy Mgmt
employers:
- Intuit Inc.
- GovTech [Formerly known as IDA] check out tech.gov.sg for more
- Nectar PaaS, security features, etc.
presentations:
- ADDO 2016 [Blue-Green Deployment] http://bit.ly/2fLfHgr
- RSA APJ 2017 [PaaS] http://bit.ly/2ylUyB9
hobbies:
- krav maga; self defense & martial arts
- food
whoami - about.me/fabian.lim

NECTAR
GovTech’s
Platform as a Service
Read more about it here:
https://blog.gds-gov.tech/nectar-10e0eb1581cf

Takeaways
1) Learned a thing about secret management
2) Learned a thing about design a secure workflow / pipeline
3) Learned a cool, new tool to integrate into your workflow
4) Or… Made a new friend :)

Tone
1) Interactive
2) Technical
3) Open

Agenda
1) Get to know each other, and the problems
2) Open discussion for designing a solution
3) Improve on current pipeline
4) Debrief and possible future integrations

My Mistakes!
OOPS! Personal mistake -
commit secrets into repository...

DevSecCon Singapore 2018 - Remove developers’ shameful secrets or simply remove shameful developers by Fabian Lim

Get to know each other, and the
problems

Get to know each other, and the problems
1) With 2 or 3 in a group, introduce and get to know each other
2) What are some common password problems?
a) What are the credentials for X database, Y API, etc?
b) Where am I supposed to store these credentials?
c) Who should have these credentials? Can I pass it to ABC?
d) How do I rotate credentials for Y API because ABC left?
e) How do I keep track of X secrets of Y apps?
Discussion Time!

Lab “Tech Stack”
Database (MongoDB)
Cloud Infrastructure (AWS)
App (Node)

Lab “Tech Platforms”
Cloud Infrastructure (AWS)
Docker
Vault
App1
Jenkins
App2 App3
mlab.com
Database1 Database2 Database3

Lab Workflow
Jenkins
Docker
App1
Source Code
github.com/<username>/
devseccon2018.git
1. Pull source code
2. Build and run in Docker

Lab
https://docs.google.com/presentation/d/12qNpVXpSxNuOE4wG9CBSGINau
c7cBjOmIiQo3w7w9AA/edit#slide=id.g31f475055f_0_238
LET’S GO!

Open discussion for designing a
solution

How to retrieve secrets?
1. Environment variables (static)
a. How do I manage the environment variables in dev, staging and
prod?
2. Run-time API retrieval (pull)
a. What API keys to use? Where do I store it?
3. Run-time deployment variable injection (push)
a. How does the deployer trust THIS instance of build?
b. How does the deployer know what secrets THIS instance need?

Chain of Trust
Manage Trust in workflow / pipeline

secret[0]
The idea of secret[0] is the first piece of credential, or the first entity
of trust, needed to initiate a trusted chain of actions like producing
the second piece of secret, etc.

Shift of Responsibility to Hold secret[0]
App
Deployer / Orchestrator
SSMHuman HSM

The App is a Box, in a Pipeline
Build
App
Source
Code
(GitHub)
Builder
(Jenkins)
App
Artifact
= Neutral Entity
= Trusted Entity
= Trusted Child Entity

= Neutral Entity
= Trusted Entity
Deployer
(Jenkins)
Secret
Mgmt
Server
(Vault)
App
Artifact
Approve!
Give Token
to Jenkins
Deploy
issues
secret
App
(Node)

= Neutral Entity
= Trusted Entity
Dev
Environment
Database
(MongdoDB)
App
(Node)

Open discussion for designing a solution
Put on the Security Architect’s hat:
1) What’s a good technical solution that removes (or the risk of
storing) secrets in code repositories?
2) How do you establish trust in a workflow?
3) How does a good development pipeline or workflow look like?
Discussion Time!

Scenario
• You are a new security developer in the team
• Audit flagged a high risk in the plaintext secrets that was checked
in t the code repository
Objectives
Task #1: Remove secrets in code repository, but still run
Task #2: Prevent secrets from exposing in build environment (logs)

Debrief, and possible future
integrations

Pipelines are Fundamental in DevSecOps
Doing so, benefits are:
• Security is built-in by design
• Containment; Blast Radius
• It can scale healthily!
• It has the ability to be re-build
By:
• Thinking like water
pipelines engineers
• Building as modular as
possible - API
• Building resilience
Because it:
• Supports SDLC; and Agile

Find your recipe
1)

Debrief, and possible future integrations
1) What are your takeaways?
a) Find # of hard-coded secrets
b) “Variablize” found secrets
c) Remove habits of checking in secrets
d) Remove guilty developers [optional]
2) How would you start to embark this journey and start to
communicating this with your developers?
3) What are possible integrations improvements to this workflow?

Join the conversation #DevSecCon
Thank you for your attention, patience, and
enthusiasm during the workshop!
Happy Lunar New Year!
Cheers!

Fair Warning
Although minimal, my scripts are written mainly in MacOS.
For security concerns and compatibility, you might consider executing them in a
Linux VM.
The permissions set on the services trust that developers (you) are trustworthy
and responsible, with the ability to change certain settings.
Please exercise caution while making changes and limit those changes to
your own environment in order to have a conducive learning environment.

Lab Audience and Objective
● Developers - who want to implement and fix the problem
● Managers - who wants the problem to be fixed but don’t know how
● Compliance / Auditors - who wants to see how a problem can be fixed
...
Remove and rotate secrets
Use a secret management server
Integration with Jenkins

Setup / Login GitHub
● Login to your GitHub account
● Click on “Issues”
● Click “New issue” to create an issue in the devseccon2018 project so I know
your username and I can invite you now!

● Fill in some description about yourself and Click “Submit new issue”
● You will receive an email to join
● membership
● to https://github.com/DevSecOpsSG

● Go to https://github.com/DevSecOpsSG/devseccon2018.git
● Click “Fork”
● You should be redirected to
https://github.com/<username>/devseccon2018.git
● This is your fork (copy) of the repository

● Open a terminal and run to clone code to your local machine:
git clone https://github.com/<username>/devseccon2018.git
● You now have a local version of the code

Setup mlab (MongoDB)
● Sign up an account at https://mlab.com
● Click “Create new” for MongoDB Deployments

● Choose any Cloud Provider (doesn’t matter)
● Choose Plan Type SANDBOX (FREE)
● Click “Continue”
● Select a region (doesn’t matter)

● Give your database a cool name (doesn’t matter)

● Check your final order
● Click “SUBMIT ORDER”

● Click on the new database

● Click on “Users”
● Click on “Add database user”

● Enter a username and password (don't use anything personal or sensitive)
○ To avoid syntax error in the later steps, do not use the “@” symbol at all; you can use it but
you must encode the character in the script later
● Leave “Make read-only” box unchecked
● Click “Create”

From the database page, construct your mlab mongodb instance URL (this
contains secret) replacing <dbuser> and <dbpassword> that you previously
entered.
In my example, mine is:
mongodb://triplejhacker:<dbpassword>@ds241668.mlab.com:41668/coolname

Paste mlab URL as MONGODB_CREDENTIALS

Push changes to your fork
git add start_app_server.sh
git commit -m ‘added mongo credentials’
git push origin master
Yes, check your secret into the code repository :P
We will remove and rotate it later, don’t worry.

Access to Jenkins
Go to http://13.228.110.97:8080
Membership in
https://github.com/DevSecOpsSG
allows access to this Jenkins server.
Click “Authorize 3jmaster” (That’s me)

Denied Access to Jenkins
If you see this
Please perform steps in slide 5 and 6 to get access to the Github organization
membership

Build and Run with Jenkins
Click “New Item”

● Enter your username or any name as a project name
● Choose “Freestyle Project”
● Click “OK”

● Go to your fork of the repository and copy the URL from “Clone or download”

● Paste it in Jenkins under “Configure”, under Source Code Management ->
Git -> Repositories -> URL

● Go to: Build -> Add Build Step -> Execute Shell

● Copy and paste the contents of jenkins.build.sh from the code repository
● Replace IMG_NAME (with your username), CTNR_NAME (with your
username), PORT (with a random number between 9000 and 9999)
● Remember this PORT value, you will need to append it to the URL later
● Click “Save”

● Click “Build Now”
● Under “Build History”, there should be a
build number like “#1”

● Click on the arrow beside the build number
(shown here)
● Click on “Console Output” to show the logs
from the build

● If all goes well, it should look something like
this ending with “Finished: SUCCESS”
● If you encounter error with container name,
the CTNR_NAME that was already been
used. So, change the value of CTNR_NAME,
save project and re-run “Build Now”.
● If you encounter error with port number, a
PORT that was already been used, so change
the value of PORT, save project and re-run
“Build Now”.

Access your deployed app
● Append the port number you specified
in jenkins to
http://13.228.110.97:<specified_port>
and go to this URL in your browser
● In my example, I go to
http://13.228.110.97:9999
● A simple app should display
● Interact with the app by adding quotes
● These quotes are stored in your
mongodb in mlab. You can go back to
mlab and check the changes in
database

● Docker UI is at http://13.228.110.97:8100
● Login with username and password “readonly” to view the state and logs
of containers
Debug your deployed app

Back to Slide
https://docs.google.com/presentation/d/1jW0pPXheS2aZqsXvfPATQLbY5sD
RyGVpuNpswS5Zv4I/edit#slide=id.g31d5e508b0_0_590

Solution 1 - Remove secret from App,
secret stays in Jenkins

Rotate mlab (MongoDB) credentials
● Back on the mlab page,
● Click on “Users”
● Click on the trash bin icon to delete the user

● Go ahead and Click on “DELETE”

● Click on “Add database user”

● Enter a NEW username and password (don't use anything personal or
sensitive)
○ To avoid syntax error in the later steps, do not use the “@” symbol at all; you can use it but
you must encode the character in the script later
● Leave “Make read-only” box unchecked
● Click “Create”

From the database page, construct your mlab mongodb instance URL (this
contains secret) replacing <dbuser> and <dbpassword> that you previously
entered.
In my example, mine is:
mongodb://new_tester:<dbpassword>@ds241668.mlab.com:41668/coolname

Remove secret from code repository
From your local machine, delete or comment out the secret mongodb url:

Remove secret from code repository
From your local machine in the directory where the git repository is, run:
git add start_app_server.sh
git commit -m 'removed secret'
git push origin master

Store secrets within Jenkins
● Go to your previously created Jenkins project, under “Configure”
● Check “Use secret text(s) file(s)”
● Click “Secret Text”

● Fill Variable as “mongodb”
● Add to “Jenkins”

● Choose “Secret text” as kind
● Fill Secret as your mongodb URL (from slide 7)
● Fill Description with your username (for easy identification)
● Click “Add”

● In the “Execute Shell”, add a line under `docker run`:
“--env MONGODB_CREDENTIALS=$mongodb ”
MUST add the backslash “” at the end!

● Click “Build Now”
● Under “Build History”, there should be a
build number like “#1” or “#2” or…

● If all goes well, it should be pulling from your latest code commit, check the
commit message.
○ It should be the same as the value in “git commit -m ‘remove secret’” ran earlier

● If all goes well, it should look something like this ending with “Finished:
SUCCESS”
● Note that the secret is also masked out. Good job Jenkins!

Congrats!
You have just removed (and rotated) a
shameful secret!
But is this good enough?

Solution 2 - Remove Secrets from App
and Jenkins, using Vault and its App
Role

What is App Role?
https://www.vaultproject.io/docs/auth/approle.html

Access to Vault
● Membership in https://github.com/DevSecOpsSG allows access to Vault
● Generate a GitHub personal access token to login -> Follow Steps 1-9 on:
https://help.github.com/articles/creating-a-personal-access-token-for-the-co
mmand-line/
● Scopes define the access for personal tokens: Check “read:org” only

Access to Vault (GUI version)
● This is Vault’s UI (web container)
Note:
Vault UI
runs on port 8300
While Vault Server
runs on port 8200

Access to Vault (CLI version)
● Download the vault binary
https://www.vaultproject.io/downloads.html
● Replace the GitHub Personal Token with your own and run:
$ export VAULT_ADDR=http://13.228.110.97:8200
$ vault auth -method=github token=<$YOUR_GITHUB_PERSONAL_TOKEN>
Successfully authenticated! You are now logged in.
The token below is already saved in the session. You do not
need to "vault auth" again with the token.
token: ************************
token_duration: 2764799
token_policies: [default]

Access to Vault (CLI version)
● Replace the username and secret with your own and run:
$ vault write secret/example/triplejhacker mongodb=<$MONGODB_URL_SECRET>
Success! Data written to: secret/example/triplejhacker
$ vault read secret/example/triplejhacker
Key Value
--- -----
refresh_interval 768h0m0s
mongodb <$MONGODB_URL_SECRET>
You can now skip slides for Access to Vault (GUI version)
This command overwrites any existing
values

● Go to http://13.228.110.97:8300
● Paste the GitHub Personal Token and login
● If it is not showing this, then Click on settings and choose "GitHub" as Login
Method. Do not change the Vault Server URL
● Click “OK”
● Paste the GitHub Personal Token and login

● Under Secret Backend, click
“secret/”
● Click “example/” folder

● Click “test” item to view its key value
● Click “NEW SECRET” at the far right to create a new item

● Fill <Insert object key> as your username
● Click on the box icon (arrow pointing) and choose “Append”,
“field:value” boxes will appear

● Fill “field” as ‘mongodb’
● Fill “value” as the secret mongodb
URL in your app from slide 7
○ mongdb://...

● Your secret is created in Vault, click on it to view
● Note its path: i.e.secret/example/<username>

Retrieve secret from Vault with Jenkins
● Navigate back to your Jenkins
project, under “Configure”
● Uncheck previous “Use secret
text(s) file(s)” box
● Check “Vault Plugin” box

Retrieve secret from Vault with Jenkins
● Fill Vault URL as http://172.18.0.2:8200
● Click “Add” -> “Jenkins”

● In the same terminal, run:
$ vault token-lookup
Key Value
--- -----
accessor 4fb12012-fb92-8d84-a4ed-bdb820532739
creation_time 1519349293
creation_ttl 2764800
display_name github-triplejhacker
entity_id ab3297fe-5fc2-5dca-f38e-c2716151774f
expire_time 2018-03-27T01:28:13.581448089Z
explicit_max_ttl 0
id ce38db15-****-524f-482c-************
issue_time 2018-02-23T01:28:13.581440761Z
meta map[org:DevSecOpsSG username:triplejhacker]
num_uses 0
orphan true
path auth/github/login
policies [default]
renewable true
ttl 2764132
Retrieve secret from Vault with Jenkins (CLI version)
This is your vault token

● In the same terminal, run:
$ vault read auth/approle/role/example/role-id
Key Value
--- -----
role_id e4964208-6fed-882b-7739-ace170ec5aba
$ vault write -f auth/approle/role/example/secret-id
Key Value
--- -----
secret_id acb24ed4-1232-298a-abd4-4ad0ac77c461
secret_id_accessor b84b516e-eb53-c5ff-8b5c-************
You can now skip slides for Retrieve secret from Vault with Jenkins (GUI version)
Retrieve secret from Vault with Jenkins (CLI version)
Secret-id is uniquely
generated each time
this command is ran

Retrieve secret from Vault with Jenkins (GUI version)
● Go to http://13.228.110.97:8300
● Click on the top right corner and “Show token”
● Copy the vault token
● Note: This is generated by Vault and is different from the GitHub
Token

● Sorry there’s no GUI for this!
● Paste your vault token from the previous slide
● Using curl, or https://www.getpostman.com/apps or any request tool, to GET
request:
curl --header "X-Vault-Token: <REPLACE WITH VAULT TOKEN>"
http://13.228.110.97:8200/v1/auth/approle/role/example/role-id
● Copy the role-id
○ {"request_id":"8d789757-ab4a-de80-9783-0927ac926f35","lease_id":"","renewable":false,"leas
e_duration":0,"data":{"role_id":"e4964208-6fed-882b-7739-ace170ec5aba"},"wrap_info":null,"w
arnings":null,"auth":null}

● Sorry there’s no GUI for this!
● Paste your vault token from the previous slide
● Using curl, or https://www.getpostman.com/apps or any request tool, to POST
request:
curl --header "X-Vault-Token: <REPLACE WITH VAULT TOKEN>" --request POST
http://13.228.110.97:8200/v1/auth/approle/role/example/secret-id
● Copy the secret-id
○ {"request_id":"a2bdb1d5-28d9-d7c8-da4e-94800ba496e3","lease_id":"","renewable":false,"leas
e_duration":0,"data":{"secret_id":"a6f53a92-96bd-9fc4-9d8a-**********","secret_id_accessor":"6
0e54752-c6d1-320f-574a-a1ee3f7a219b"},"wrap_info":null,"warnings":null,"auth":null}

● Choose “Vault App Role Credential” as kind
● Fill Role ID from previous, previous slide
● Fill Secret ID from previous slide
● Fill Description with your username (for easy identification)
● Click “Add”

● Choose your newly created item as the Vault Credential i.e.
“triplejhacker approle”
● Click “Add a vault secret”

● Fill Environment Variable as “mongodb” (all small caps)
● Fill the rest as illustrated matching from the Vault UI
Path
(secret/example/triplejhacker)
Key Name
(mongodb)

● If all goes well, it should be pulling from your latest code commit, check the
commit message.
○ It should be the same as the value in “git commit -m ‘remove secret’” ran earlier
● We didn’t push any code changes so this is correct.

Access your deployed app
● Append the port number you specified
in jenkins to
http://13.228.110.97:<specified_port>
and go to this URL in your browser
● In my example, I go to
http://13.228.110.97:9999
● A simple app should still display
● Interact with the app by adding quotes
● These quotes are stored in your
mongodb in mlab. You can go back to
mlab and check the changes in
database

Congrats!
You have just removed a shameful
secret AND use App Role to control
access to the secrets!
But is this good enough?

https://www.vaultproject.io/

Solution 3 - Remove Secrets from App
and Jenkins, using App Role and +++

… until next time...
https://www.vaultproject.io/docs/concepts/response-wrapping.html
Or try
https://medium.com/what-about-security/all-day-devops-2017-removing-dev
elopers-shameful-secrets-f5aca3960316

DevSecCon Singapore 2018 - Remove developers’ shameful secrets or simply remove shameful developers by Fabian Lim

More Related Content

What's hot (20)

Similar to DevSecCon Singapore 2018 - Remove developers’ shameful secrets or simply remove shameful developers by Fabian Lim (20)

More from DevSecCon (20)

Recently uploaded (20)

DevSecCon Singapore 2018 - Remove developers’ shameful secrets or simply remove shameful developers by Fabian Lim