SlideShare a Scribd company logo
@omerlh
Can Kubernetes Keep a
Secret?
Omer Levi Hevroni
AppSec Cali 2019
@omerlh@omerlh
@omerlh
I’m a builder
@omerlh
@omerlh
DevSecOps @
@omerlh
I OWASP
• Zap contributor
• Proud member
• Glue project leader
@omerlh
@omerlh
Super-Devs: Full Responsibility
● Writing Code
● Deploying to Production
● Monitoring
https://www.imdb.com/title/tt4016454/mediaviewer/rm2380811776
@omerlh
Super-Devs Need Help
● Good tools to support them
● Make it harder to do mistakes
● Secure by design
@omerlh
@omerlh
Manifests
Files
Code
A GitOps Solution
Kubernetes Icons Source: Kubernetes Community, Apache 2 license
@omerlh
How do we manage secrets?
@omerlh
Manifests
Files
Code
Secret
A GitOps Solution
Kubernetes Icons Source: Kubernetes Community, Apache 2 license
@omerlh
Requirements
 GitOps
 Kubernetes native
 Secure
 “One-way encryption”
@omerlh
Pod is out of scope
● Who can “SSH” into it?
● What is running on the pod?
● Does the code leaked the secrets?
@omerlh
Let’s Go!
@omerlh
First iteration – Kubernetes Secrets
@omerlh
https://kubernetes.io/docs/concepts/configuration/secret/
@omerlh
Requirements
 GitOps
 Kubernetes native
 Secure
@omerlh@omerlh
@omerlh
Let’s take a deeper look…
 Producing (dev)
 Consuming (application)
Kubernetes Icons Source: Kubernetes Community, Apache 2 license
@omerlh
Producing - File Manifest?
@omerlh
Well, that complicates things…
http://i.imgur.com/5ebYy62.jpg
@omerlh
@omerlh
Producing – Naive Approach
@omerlh
Producing - Encrypted Secrets?
● Secrets that can be committed
● Transparent for the application
● Multiple solutions
○ Helm Secrets
○ Sealed Secrets
@omerlh
A Sealed Secret
@omerlh@omerlh
@omerlh
Issues
● Key Management
○ Sealed Secret – single key-pair in the cluster
○ Helm Secret – based on Mozilla mops (AWS/GCP KMS support)
● Coupling to a specific cluster/deployment method
● Any change to the secret requires decryption
@omerlh
Let’s take a deeper look…
ⓧ Producing (dev)
 Consuming (application)
@omerlh
Consuming – Environment Variables
@omerlh
The Environment Variable Dispute
https://i0.wp.com/www.rogerogreen.com/wp-content/uploads/2015/06/Disputation.jpg
@omerlh
• Some log libraries collects
env vars
• Accessible via /proc
• Visible when inspecting
docker image
• RCE – run env to leaked all
env vars. Simpler than
finding all sensitive files and
exporting them (even with
LFI)
• Harder to commit accidently
• Simpler than files
• If you can access /proc or
inspect docker images, you can
inspect mounted volumes
• Better permissions model on
windows (thanks @swisshttp!)
• Leaked files (thanks
@sporkmonger!)
The Environment Variable Dispute
Cons Pros
https://tvtropes.org/pmwiki/pmwiki.php/Main/GoodAngelBadAngel
@omerlhhttps://twitter.com/omerlh/status/1079088158929797121
@omerlh
Consuming – Volume Mount
@omerlh
Consuming/Producing: Configuration Files
Configuration File
Base64 Encoder
Secret Manifest
@omerlh
Let’s take a deeper look…
ⓧ Producing (dev)
ⓧ Consuming (application)
Kubernetes Icons Source: Kubernetes Community, Apache 2 license
@omerlh
Requirements
 GitOps – under some serious limitations
 Kubernetes native
 Secure – depend on usage
@omerlh
Second iteration – Hashicorp Vault
@omerlh
What?
● Secure secrets storage
● Native Kubernetes integration
● Seamless consuming
○ Side-car to generate config files
https://www.vaultproject.io/
@omerlh
Workflow
/my-app/super-sensitive
Kubernetes Icons Source: Kubernetes Community, Apache 2 license
@omerlh
@omerlh
@omerlh
Workflow
Access Control
/my-app/super-sensitive
Kubernetes Icons Source: Kubernetes Community, Apache 2 license
@omerlh
Manifests
Files
Code
Secret
Vault - Threat Modeling
Kubernetes Icons Source: Kubernetes Community, Apache 2 license
Kubernetes Icons Source: Kubernetes Community, Apache 2 license
@omerlh
Imperfect solution
●Separate storage of secrets and deployment files
○ No single source of truth
●External Permission Model
●Deployment
○ Cloud vendor alternatives (Azure KeyVault, AWS secret manager)
○ Vault users authn/authz
@omerlh
Requirements
ⓧ GitOps
 Kubernetes native
 Secure – depend on usage
@omerlh
@omerlh
Travis Encrypted Secrets
https://docs.travis-ci.com/user/encryption-keys/
@omerlh
Eureka!
http://theunprofessionalblog.blogspot.com/2016/04/whatsapp-this-is-killing-me.html
@omerlh
Third iteration – Kamus
Travis secret encryption – for Kubernetes
@omerlh
Kamus?
@omerlh
https://github.com/Soluto/kamus/tree/master/example
@omerlh
A perfect solution?
 GitOps
 Kubernetes native
 Secure
@omerlh
Let’s talk about security
@omerlh
Permission Model
Encrypt Decrypt
User Yes (Can be
limited)
No
Pod Yes Only it’s own
secrets
@omerlh
Kamus – Threat Model
Encryptor Decryptor
Kubernetes Icons Source: Kubernetes Community, Apache 2 license
@omerlh
Mitigations: User
● Secure by default permission model
● Secured CLI
○ Enforce HTTPS
○ Support for certificate pinning
@omerlh
Mitigations: Git
● Strong encryption (using Azure KeyVault/GCP KMS)
○ HSM protection
○ IP Filtering
● One-way encryption
@omerlh
Mitigations: Pod
● Secure by default permission
model
● In-Memory volume for
decrypted files
Kubernetes Icons Source: Kubernetes Community, Apache 2 license
@omerlh
Mitigations: Kamus API
● Separate pods
● Authentication support for encryptor
● Security tests
○ SAST (Checkmarx)
○ DAST (Zaproxy)
○ Packages scan (Snyk)
@omerlh
Accepted Risks
●Clear text traffic inside the cluster
●Any pod in the same namespace can mount any service account
○ Pod impersonation
●Service account token never expires
@omerlh
Public Threat Model
https://github.com/Soluto/kamus/blob/master/docs/features
@omerlh
Security.md
https://github.com/Soluto/kamus/blob/master/security.md
@omerlh
Kamus - A perfect solution
 GitOps
 Kubernetes native
 Secure
@omerlh
How can I use it?
● Simply using helm:
helm install kamus soluto/kamus
● Checkout the install guide for a secure
installation
● Blog post - https://bit.ly/2T2Nhgs
@omerlh
Kamus Roadmap
● AWS support
● Custom Resource Descriptor
● Rolling encryption keys
● Quality – improve test coverage
● FaaS
@omerlh
Wrapping Up
@omerlh
Solutions
GitOps Kubernetes
Native
Secure
Kubernetes
Secrets
It depends Yes It depends
Vault No Yes Yes
Kamus Yes Yes Yes
@omerlhhttp://www.applestory.biz/hermione-hand-raise-gif.html
Questions?
@omerlh
Feedback appreciated
@omerlh
Can Kubernetes Keep a Secret?
@omerlh
@omerlh
Kamus Enable Super-Devs to Fly Higher
https://www.imdb.com/title/tt4016454/mediaviewer/rm2380811776
@omerlh
Thank You!
Omer Levi Hevroni
AppSec Cali 2019

More Related Content

PPTX
Docker Basic to Advance
PPTX
Docker introduction & benefits
PDF
Introduction to Docker - VIT Campus
PDF
Mongo db dhruba
PDF
Cgroups, namespaces, and beyond: what are containers made from? (DockerCon Eu...
PPTX
Introduction to Redis
PDF
Docker Registry V2
Docker Basic to Advance
Docker introduction & benefits
Introduction to Docker - VIT Campus
Mongo db dhruba
Cgroups, namespaces, and beyond: what are containers made from? (DockerCon Eu...
Introduction to Redis
Docker Registry V2

What's hot (20)

PPTX
Docker Swarm for Beginner
PDF
Introduction to Docker storage, volume and image
PDF
Docker volume
PPTX
PDF
Docker Introduction
PDF
A brief introduction to version control systems
PDF
Kubernetes - A Comprehensive Overview
PDF
Dockerハンズオン
PDF
Docker in real life
PDF
Helm - Package Manager for Kubernetes
PPTX
DevOps with Kubernetes
PDF
Dockers and kubernetes
PPTX
Docker 101 - Nov 2016
PDF
Docker 101 - Getting started
PPTX
Containers and Docker
PDF
Docker Containers Deep Dive
PDF
Docker & kubernetes
PPTX
Cloud computing
ODP
containerD
PPT
Linux history & features
Docker Swarm for Beginner
Introduction to Docker storage, volume and image
Docker volume
Docker Introduction
A brief introduction to version control systems
Kubernetes - A Comprehensive Overview
Dockerハンズオン
Docker in real life
Helm - Package Manager for Kubernetes
DevOps with Kubernetes
Dockers and kubernetes
Docker 101 - Nov 2016
Docker 101 - Getting started
Containers and Docker
Docker Containers Deep Dive
Docker & kubernetes
Cloud computing
containerD
Linux history & features
Ad

Similar to Can Kubernetes Keep a Secret? (20)

PPTX
FTRD - Can Kubernetes Keep a Secret?
PPTX
Can Kubernetes Keep a Secret? - Women in AppSec Webinar
PDF
K8 Meetup_ K8s secrets management best practices (Git Guardian).pdf
PDF
Continuous Security for GitOps
PDF
Securing an NGINX deployment for K8s
PDF
Introduction to Kubernetes Security (Aqua & Weaveworks)
PDF
Kubernetes Summit 2019 - Harden Your Kubernetes Cluster
PDF
Securing Kubernetes Workloads
PPTX
Unlocking DevOps Secuirty :Vault & Keylock
PDF
The Hacker's Guide to Kubernetes
PDF
Kubernetes Secrets Management Meap V06 1 All 8 Chapters Alex Soto Bueno Andre...
PPTX
Kubernetes and container security
PDF
Knolx_ Sealed Secrets
PDF
Docker security 101 (CfgMgmtCamp 2019)
PPTX
Devoops: DoJ Annual Cybersecurity Training Symposium Edition 2015
PDF
Commit 2024 - Secret Management made easy
PPTX
Security best practices for kubernetes deployment
PPTX
Security best practices for kubernetes deployment
PPTX
Kamus intro
PDF
Vincent Ruijter - ~Securing~ Attacking Kubernetes
FTRD - Can Kubernetes Keep a Secret?
Can Kubernetes Keep a Secret? - Women in AppSec Webinar
K8 Meetup_ K8s secrets management best practices (Git Guardian).pdf
Continuous Security for GitOps
Securing an NGINX deployment for K8s
Introduction to Kubernetes Security (Aqua & Weaveworks)
Kubernetes Summit 2019 - Harden Your Kubernetes Cluster
Securing Kubernetes Workloads
Unlocking DevOps Secuirty :Vault & Keylock
The Hacker's Guide to Kubernetes
Kubernetes Secrets Management Meap V06 1 All 8 Chapters Alex Soto Bueno Andre...
Kubernetes and container security
Knolx_ Sealed Secrets
Docker security 101 (CfgMgmtCamp 2019)
Devoops: DoJ Annual Cybersecurity Training Symposium Edition 2015
Commit 2024 - Secret Management made easy
Security best practices for kubernetes deployment
Security best practices for kubernetes deployment
Kamus intro
Vincent Ruijter - ~Securing~ Attacking Kubernetes
Ad

More from Soluto (20)

PPTX
Solving trust issues at scale - AppSec California
PPTX
Solving trust issues at scale
PPTX
Things I wish someone had told me about Istio, Omer Levi Hevroni
PPTX
The Dark Side of Monitoring
PPTX
Hacking like a FED
PPTX
Monitoria@Icinga camp berlin
PPTX
Secure Your Pipeline
PDF
React new features and intro to Hooks
PPTX
Secure the Pipeline - OWASP Poland Day 2018
PDF
Monitoria@reversim
PPTX
Languages don't matter anymore!
PPTX
Security Testing for Containerized Applications
PPTX
Owasp glue
PPTX
Unify logz with fluentd
PPTX
Storing data in Redis like a pro
PPTX
Monitor all the thingz slideshare
PPTX
Authentication without Authentication - AppSec California
PPTX
Authentication without Authentication - Peerlyst meetup
PPTX
Security Testing with Zap
PPTX
Authentication Without Authentication
Solving trust issues at scale - AppSec California
Solving trust issues at scale
Things I wish someone had told me about Istio, Omer Levi Hevroni
The Dark Side of Monitoring
Hacking like a FED
Monitoria@Icinga camp berlin
Secure Your Pipeline
React new features and intro to Hooks
Secure the Pipeline - OWASP Poland Day 2018
Monitoria@reversim
Languages don't matter anymore!
Security Testing for Containerized Applications
Owasp glue
Unify logz with fluentd
Storing data in Redis like a pro
Monitor all the thingz slideshare
Authentication without Authentication - AppSec California
Authentication without Authentication - Peerlyst meetup
Security Testing with Zap
Authentication Without Authentication

Recently uploaded (20)

PDF
Reach Out and Touch Someone: Haptics and Empathic Computing
PPTX
Detection-First SIEM: Rule Types, Dashboards, and Threat-Informed Strategy
PDF
Chapter 3 Spatial Domain Image Processing.pdf
PDF
Unlocking AI with Model Context Protocol (MCP)
PDF
CIFDAQ's Market Insight: SEC Turns Pro Crypto
PPTX
Cloud computing and distributed systems.
PPTX
KOM of Painting work and Equipment Insulation REV00 update 25-dec.pptx
PPTX
VMware vSphere Foundation How to Sell Presentation-Ver1.4-2-14-2024.pptx
PDF
Agricultural_Statistics_at_a_Glance_2022_0.pdf
PDF
Encapsulation theory and applications.pdf
PDF
KodekX | Application Modernization Development
PPTX
Understanding_Digital_Forensics_Presentation.pptx
PPTX
Digital-Transformation-Roadmap-for-Companies.pptx
PDF
Advanced methodologies resolving dimensionality complications for autism neur...
PDF
Encapsulation_ Review paper, used for researhc scholars
PDF
Shreyas Phanse Resume: Experienced Backend Engineer | Java • Spring Boot • Ka...
PPT
Teaching material agriculture food technology
PDF
The Rise and Fall of 3GPP – Time for a Sabbatical?
PDF
Bridging biosciences and deep learning for revolutionary discoveries: a compr...
PDF
Spectral efficient network and resource selection model in 5G networks
Reach Out and Touch Someone: Haptics and Empathic Computing
Detection-First SIEM: Rule Types, Dashboards, and Threat-Informed Strategy
Chapter 3 Spatial Domain Image Processing.pdf
Unlocking AI with Model Context Protocol (MCP)
CIFDAQ's Market Insight: SEC Turns Pro Crypto
Cloud computing and distributed systems.
KOM of Painting work and Equipment Insulation REV00 update 25-dec.pptx
VMware vSphere Foundation How to Sell Presentation-Ver1.4-2-14-2024.pptx
Agricultural_Statistics_at_a_Glance_2022_0.pdf
Encapsulation theory and applications.pdf
KodekX | Application Modernization Development
Understanding_Digital_Forensics_Presentation.pptx
Digital-Transformation-Roadmap-for-Companies.pptx
Advanced methodologies resolving dimensionality complications for autism neur...
Encapsulation_ Review paper, used for researhc scholars
Shreyas Phanse Resume: Experienced Backend Engineer | Java • Spring Boot • Ka...
Teaching material agriculture food technology
The Rise and Fall of 3GPP – Time for a Sabbatical?
Bridging biosciences and deep learning for revolutionary discoveries: a compr...
Spectral efficient network and resource selection model in 5G networks

Can Kubernetes Keep a Secret?

Editor's Notes

  • #2: Hey, good morning everyone My name is Omer I want to start this talk by showing gratitude First, to all the people who worked hard on organizing this conf and all the people who are working today so we all could enjoy it - thank you Second, I want to thank the organizers who choose me to speak here, so thank you. It is a big honor <pause> Can kubernetes keep a secret? <pause> Why? Raise you’re hand if you ever worked on a project and you had to deal with credentials: API Key, client secret, certificates etc
  • #3: What you need to do is pass these credentials securely to the platform running your code, so the app in production can use it. Different platforms have different solutions to the problem (sometime good, sometime bad). This talk will focus on Kubernetes – how we can tell a secret to Kubernetes, and make sure only Kubernetes will know it? But first – let me introduce myself quickly, so you could understand what are my credentials and where I’m coming from.
  • #4: I’m a builder, this is what I love doing and doing it from a really early age Doing it professionally for the last 8 years I’m from Israel, married etc Who else is a builder? This talk is for you!
  • #5: Today I’m working at Soluto, our missing is to help people with their technology My job is DevSecOps, or as I see it - helping the entire team to build a more secure software I’m achieving it via many approaches, including education, reviewing and threat modeling – but what I love the most is threat modeling
  • #6: Big part of my work is OWASP, I’m enthusiast and familiar to many project. I contributed code to projects, mainly Zap and Glue and I’m a paid memember and project leader of Glue. Glue is a tool that helps to integrate security tools into the CI/CD pipeline – I will not have time to dive into the tool, but come talk with me later about it – I have stickers 
  • #7: What you need to do is pass these credentials securely to the platform running your code, so the app in production can use it. Different platforms have different solutions to the problem (sometime good, sometime bad). This talk will focus on Kubernetes – how we can tell a secret to Kubernetes, and make sure only Kubernetes will know it?
  • #8: Explain why it is a challenge – you cant expect one person to manage all secrets Why not solved it manually
  • #9: Explain why it is a challenge – you cant expect one person to manage all secrets Why not solved it manually
  • #10: And that’s why we love GitOps: Git is a tool that all devs are familiar with.
  • #13: And we started to look for solutions. It want not an easy path, and today I want to share with you the process we want through. So, let’s start we talking on what we want.
  • #19: Choose one sentence
  • #20: Security is what we all here love
  • #23: Security features like encryption at rest
  • #24: Encoding is not encrypting Adding native approach
  • #27: Add meme
  • #32: Add slide with links
  • #34: Laugh at my bad english
  • #35: Example of 3 items/JSON representatiom
  • #38: Security is what we all here love
  • #41: Add the user here
  • #43: Add the user here
  • #45: Valut policies, policy assignment etc
  • #46: Security is what we all here love
  • #52: Make it more visual
  • #53: Security is what we all here love
  • #54: Battle tested
  • #55: Add attributation
  • #56: Add headlines – encryptor & decryptor
  • #64: We really love Kamus, we’re been using it in production for the past 6 months
  • #67: End of journey meme/image
  • #68: Today I discussed 3 different solutions for secret management on Kubenretes. All are good solutions, depend on your requirments.
  • #71: I started the talk by asking “Can Kubernetes keep a secret?” Now you that yes – Kubernetes can. You just need to use the right tool for you’re use case.
  • #72: For us, it was Kamus What Kamus can do for you?