Helm Charts Security 101

Helm Charts Security 101
with Deep Datta

Agenda
About Me
Helm 3 Basics
About Helm Charts
● Chart.yaml
● values.yaml
● Templates
● Charts
Helm Chart Security Overview
● Signing Charts (Provenance)
● Secrets Management Tools
● RBAC
● Certificates
Using a Trusted Source
● Dependency Management
● CVEs
● Mitigation
Today, we’ll discuss Helm charts, commands, best
practices, and tools from the community that help
automate security in Helm and Kubernetes.
Why use trusted registries, how to keep your
dependencies safe, and and how chart authors can
provide mitigation details for their consumers.
Blueprint for Helm Security

About Me
Deep Datta
deepd@jfrog.com
Community Product Manager
Twitter @DeepDattaX

Helm is a Package Manager for Kubernetes
Helm is a package manager for Kubernetes which helps users create
templated packages called Helm Charts to include all Kubernetes
resources that are required to deploy a particular application.
Helm then assists with installing the Helm Chart on Kubernetes:
● Install
● linting
● status
● test
● verify
● deploy
● upgrade
● rollback
November of 2019

Helm 2 vs Helm 3
Tiller was removed in Helm 3:
Removal of Tiller

Helm 2 vs Helm 3
Helm 3 interacts directly with the Kubernetes API
Role Based Access Controls

Here are more improvements to Helm 3:
Dependencies: used to live in a requirements.yaml file, but are now part of the Chart.yaml file.
Releases in namespaces
Three-way strategic merge patch
OCI Registries for charts
Chart validation: JSONSchema support is added
Improved CRD support: Kubernetes Custom Resource Definition (CRD) installations
Library charts: a class of charts called “library charts” are introduced in Helm 3

New commands for monitoring
Helm status [RELEASE]
Displays the status of the named
release
Helm ls
Lists all the releases
Helm history [RELEASE]
The history of releases is printed
$ helm history demo-rel
REVISION UPDATED STATUS CHART APP VERSION DESCRIPTION
1 Mon Oct 3 10:15:13 2016 superseded alpine-0.1.0 1.0 Initial install
2 Mon Oct 3 10:15:13 2016 superseded alpine-0.1.0 1.0 Upgraded successfully
3 Mon Oct 3 10:15:13 2016 superseded alpine-0.1.0 1.0 Rolled back to 2
4 Mon Oct 3 10:15:13 2016 deployed alpine-0.1.0 1.0 Upgraded successfully

Helm 3 Charts
Helm Charts Summary
When you publish a Helm chart, you
can take care of all the security
issues beforehand.
Helm holds the final package with all of your
previously approved configuration options and
pieces in place and creates an immutable way
to manage security with each chart version and
each build being tracked.
Chart.yaml
Charts
Templates
values.yaml

Demo Chart
charts
templates
Chart.yaml
values.yaml

Chart.yaml
This is where metadata about the chart lives. You also declare dependencies here.
apiVersion: v2
name: demochart
description: A Helm chart for Kubernetes
type: application
version: 0.1.0
appVersion: 1.16.0
dependencies:
- name: nginx
version: "1.2.3"
repository: "https://example.com/charts"
- name: memcached
version: "3.2.1"
repository: "https://another.example.com/charts"

values.yaml
This is where you define your configurations options for each deployment
replicaCount: 1
image:
repository: nginx
pullPolicy: IfNotPresent
# Overrides the image tag whose default is the chart appVersion.
tag: ""
imagePullSecrets: []
nameOverride: ""
fullnameOverride: ""
serviceAccount:
# Specifies whether a service account should be created
create: true
# Annotations to add to the service account
annotations: {}
# The name of the service account to use.
# If not set and create is true, a name is generated using the fullname template
name: ""
podAnnotations: {}
podSecurityContext: {}
# fsGroup: 2000

Templates Folder
This is where Helm finds the YAML definitions
service.yaml
deployment.yaml
hpa.yaml
Ingress.yaml
Serviceaccount.yaml
helpers.tpl
NOTES.txt

service.yaml
Here you define your set of services for the pods in Kubernetes
apiVersion: v1
kind: Service
metadata:
name: {{ include "demochart.fullname" . }}
labels:
{{- include "demochart.labels" . | nindent 4 }}
spec:
type: {{ .Values.service.type }}
ports:
- port: {{ .Values.service.port }}
targetPort: http
protocol: TCP
name: http
selector:
{{- include "demochart.selectorLabels" . | nindent 4 }}

deployment.yaml
Generates the metadata of your deployment
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ include "demochart.fullname" . }}
labels:
{{- include "demochart.labels" . | nindent 4 }}
spec:
{{- if not .Values.autoscaling.enabled }}
replicas: {{ .Values.replicaCount }}
{{- end }}
selector:
matchLabels:
{{- include "demochart.selectorLabels" . | nindent 6 }}
template:
metadata:

Charts folder
demochart/
Chart.yaml
values.yaml
charts/
templates/
...
The charts/ directory contains subcharts

The Helm community is developing expertise with a
number of built-in processes that can help with security
SHA and Checksums
Signatures and signing charts (GPG)
Provenance Files and Helm Verify
Secrets Management Tools
RBAC and Service Accounts
POD Security Policies
Network Security
Image security and subcharts
Cert-Manager for certificate signing
Helm lint
CVE Information
Mitigation Notes

first...minikube
https://tech.paulcz.net/blog/getting-started-with-helm/
Minikube is a tool that quickly sets up Kubernetes
locally. Minikube runs a single-node Kubernetes cluster
inside a Virtual Machine for users looking to try out
Kubernetes or develop with it day-to-day.
brew install hyperkit
https://minikube.sigs.k8s.io/docs/drivers/hyperkit/

● Create a chart
● Define services
● Sign the chart
● Verify the chart’s provenance
● Secrets Management (encode and enrypt)
● Helm Lint | Helm Status | Helm History
Steps
Helm create demochart

Set Resource Quotes!
apiVersion: v1
kind: ResourceQuota
metadata:
name: mem-cpu-demo
spec:
hard:
requests.cpu: "1"
requests.memory: 1Gi
limits.cpu: "2"
limits.memory: 2Gi

Navigating Security with Helm
Signing Charts
Verifying Charts (Provenance)
Secrets Management
Helm Lint
RBAC and Service
Accounts
POD Security
Dependencies
Certificates
Chart Hashes

SHA-256 and SHA-512 Hash as a
Checksum

Navigating Security
Signing Charts
Secrets Management
Helm Lint and Helm
Status
RBAC and Service
Accounts
POD Security
Dependencies
Certificates
Chart Hashes

Signing with GnuPGP Signature
brew install gpg
helm plugin install https://github.com/technosophos/helm-gpg

Create the public-private key
gpg --gen-key
Keyname: demokey
--------------------------------------
pub rsa2048 2020-07-21 [SC] [expires: 2022-07-21]
DBA3 F0A7 F87B D112 800D A50A 388E B5D0 D62C 07FF
uid [ultimate] demokey <deepd@jfrog.com>
sub rsa2048 2020-07-21 [E] [expires: 2022-07-21]

Sign the chart with:
Helm package --sign --key ‘demokey’ --keyring ~/.gnupg/secring.gpg demochart
GNUPG 2.1
Use the following command to transfer your keys into the old file format:
gpg --export-secret-keys >~/.gnupg/secring.gpg

You’ve signed and created a provenance ﬁle to track lineage:
demochart-0.1.0.tgz
demochart-0.1.0.tgz.prov

Navigating Security
Signing Charts
Secrets Management
Helm Lint
RBAC and Service
Accounts
POD Security
Dependencies
Certificates
Chart Hashes

Helm verify
helm verify demochart-0.1.0.tgz
Signed by: demokey (demokey) <demokey@gmail.com>
Using Key With Fingerprint: 1CFE2BD91BD3847C@9743661D82D917761CFEA75
Chart Hash Verified:
sha256:471c655ef1d4de91a782ecfcb2a83aeee341e8fc786ebfd9ee34d682f3895e0

Secrets Management
Don’t store sensitive information (passwords,
authentication credentials, API keys...) in ConﬁgMaps
Secrets
Sensitive data
ConfigMaps
Key:value pairs that not intended to be
hidden

env:
- name: MYSQL_ROOT_PASSWORD
valueFrom:
secretKeyRef:
name: mariadb-root-password
key: password
These are secret!

Encrypting Secrets
helm plugin install
https://github.com/futuresimple/helm-secrets

Helm Secrets Plugin
Usernames, Passwords, Database Credentials, API Tokens, TLS Certiﬁcates
We end up putting this in plain text in
many diﬀerent places
...don’t store this in source control

$ gpg --generate-key
$ gpg —-fingerprint
--------------------------------------
pub rsa2048 2020-07-21 [SC] [expires: 2022-07-21]
DBA3 F0A7 F87B D112 800D A50A 388E B5D0 D62C 07FF
uid [ultimate] demokey <demokey@gmail.com>
sub rsa2048 2020-07-21 [E] [expires: 2022-07-21]

Sops step
secrets.yaml diff=sopsdiffer
secrets.*.yaml diff=sopsdiffer
creation_rules:
-pgp:"DBA3 F0A7 F87B D112 800D A50A 388E B5D0 D62C 07FF"
helm_varsCreate a ﬁle .sops.yaml inside helm_vars folder.

helm plugin install:
https://github.com/zendesk/helm-secrets

Create a ﬁle secrets.yaml inside helm_vars folder. Supply with our key pair value in plain text.
mysecret:pAssw0rd
Lets encrypt our secrets.yaml using Helm-secret plugin.
$ helm secrets enc ~/helm_vars/secrets.yaml
Encrypting secrets.yaml
Encrypted secrets.yaml

mysecret:ENC[AES256_GCM,data:sxfEX+kK0U4=,iv:55BozyMoAIB8dD7i3JcOtlzQO6gkwVfSx90
J+27y/SY=,tag:mCHsxJVdFxZ6h9mfkcAG8A==,type:str]
sops:
kms:[]
gcp_kms:[]
azure_kv:[]
lastmodified:'2019-08-20T13:40:56Z'
mac:ENC[AES256_GCM,data:28KV+jAT+L7lZSTiIJTL7XC5XvPH4Vzc3R/P/KOdUwBoLBt8Ozo9Z6qQ
Mn4QI6XBYhS127GhD9xOLwMnzjm1yEXxM1dRUpy68jzczDghmUXJx494ZK4klIGEDoQLMaGI6s4rAQoa
flix8Tewo3H0ZmQH4P3H/oxcPhRURJY2qns=,iv:f6ZY1L5/Dg9zcIwO5CO8RZ3weQXsHa4+ufkf/iM3
GUo=,tag:ivGMwTGyG27JnbQLzsTlXA==,type:str]
pgp:
-created_at:'2019-08-20T13:40:54Z'
enc:|
-----BEGINPGPMESSAGE-----
hQEMA94e2vEPuuuwAQf/fj/fbhRG7w9OeqAmyMKu6UQzEA7HD54287WGbNSmihAc
SgGsUpcgPLRLPO+n2MOrHLlPdx15gUIFfE7q6y1POcYJmTCHiolNDigQFSQj5mQZ
rZ3xT0kzjcrOw8q0HUoHcKgQsn7jQr3Y3MX3z/63wq9jW2lDCqoYk61s7z5SUKLD
J13Q+TUvCEXVfkjO/n2lmuSEgV7rmc5Gq5GkQ3o07hbcJTmCXkapo/s11I4oRMjw
ssxz1psquvx8awKG/SfVhmVllRCIQTINDhCGUHhAqPHhDQ/v79Akh4Snk6Q2Pxfw
QcWJsJvYCMyU68F28j3SNOKkqtHcDOGNSWMATj8+b9JcAZqFXGBjxHdSFUEUs7GJ
tAOH1uBnzW6Jmr3cMluX1mb3YNSG5VYTJ1fmpK/fEgOCZylz91MYt0mMmh+GNVTS
IBN5Z/7V3HAGL24Sq8/+SNB9OZGecyD29Xhcxlw=
=vSV3
-----ENDPGPMESSAGE-----
fp:1189AF85735D53F1285FBBD59EE16A2EC4F2E8FB

You can also use Hashicorp Vault for advanced Secrets
Management
https://www.vaultproject.io/

Helm lint
helm lint demochart
==> Linting demochart
[INFO] Chart.yaml: icon is recommended
1 chart(s) linted, 0 chart(s) failed

Pod Security Policies
PodSecurityPolicy (PSP) admission controller
When you enable Pod Security Policies, you can control things like:
● The running of privileged containers
● Use of host namespaces
● Use of host networking and ports
● Use of volume types
● Use of the host filesystem
● A whitelist of Flexvolume drivers
● The allocation of an FSGroup that owns the pod’s volumes
● Requirements for use of a read only root file system
● The user and group IDs of the container
● Escalations of root privileges
kubectl create -f your-new-policy.yaml

Disable privileged containers
apiVersion: policy/v1demobeta1
kind: PodSecurityPolicy
metadata:
name: prevent-privileged-containers
spec:
privileged: false
https://resources.whitesourcesoftware.com/blog-whitesource/kubernetes-pod-security-policy

Read-only file system
metadata:
name: read-only-fs
spec:
readOnlyRootFilesystem: true

Prevent privilege escalation
metadata:
name: no-privilege-escalation
spec:
allowPrivilegeEscalation: false

Prevent containers from running as root
apiVersion: policy/v1beta1
metadata:
name: no-privilege-escalation
spec:
MustRunAsNonRoot: true

Group your policies together
apiVersion: policy/v1beta1
metadata:
name: example
spec:
privileged: false
spec:
readOnlyRootFilesystem: true
spec:
allowPrivilegeEscalation: false
spec:
MustRunAsNonRoot: true

Network Policies
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: access-nginx
spec:
podSelector:
matchLabels:
app: nginx
ingress:
- from:
- podSelector:
matchLabels:
access: "true"
To limit the access to the nginx service so that only Pods with the label access: true can query it,
create a NetworkPolicy object as follows:

Let’s talk about Pods
The desired state of each cluster and access privileges within each
node is highly conﬁgurable. For example, namespaces and service
accounts can be used to divide the cluster’s resources to multiple users
and grant unique permissions within each group.
Even pods have security features that can be activated with the
admission controller and by assigning unique privileges to users and
groups using Role-Based Access Control (RBAC).

RBAC
lock things down
Audit2rbac: https://github.com/liggitt/audit2rbac

RBAC
Role
ClusterRole
RoleBinding
ClusterRoleBinding
(ServiceAccount, User or Group)

Service Accounts
Who is the user working within the pod?
rbac:
create: true
serviceAccounts:
client:
create: true
name:
server:
create: true
name:
Service accounts are tied to a set of
credentials are mounted into pods
allowing in-cluster processes to talk
to the Kubernetes API.
API requests are tied to either a
normal user or a service account.

We need to install cert-manager to do the work with Kubernetes to
request a certiﬁcate and respond to the challenge to validate it.
We can use Helm to install cert-manager.

Let’s use cert-manager by Jetstack for TLS

Install Cert-Manager using Helm Charts

TLS with Cert-Manager
Then you’ll need to get a TLS certificate by installing cert-manager:
# Install the CustomResourceDefinition resources separately:
$ kubectl apply --validate=false -f
https://github.com/jetstack/cert-manager/releases/download/v0.15.0/cert-manager.crds.yaml
# Create the namespace for cert-manager:
$ kubectl create namespace cert-manager
# Install the cert-manager Helm chart from ChartCenter:
$ helm install cert-manager center/jetstack/cert-manager
You can do a final rollout status check with:
$ kubectl -n cert-manager rollout status deploy cert-manager

Last Item: Checking Dependencies
Image security and sub-charts

More...
Helm Charts Summary
demochart/
Chart.yaml
values.yaml
charts/
templates/
...
The charts/ directory contains subcharts

Creating a subchart
$ cd demochart/charts
$ helm create mysubchart
Creating mysubchart
$ rm -rf mysubchart/templates/*.*

Checking Dependencies (Image Security and Subcharts)

CVE and Mitigation
with JFrog ChartCenter

For Chart Maintainers,
Introducing:
security-mitigation.yaml

Here is the spec:
## Schema version of this YAML file
schemaVersion: v1
## Overall mitigation summary
summary: text
## External URL if you'd like to link to an external page
securityAdvisoryUrl: URL
## If you want to point us to a file instead of filling out the CVE's here
useMitigationExternalFile: boolean
mitigationExternalFileUrl: URL
## Mitigation notes for individual CVEs
mitigations:
cves:
## Indicates package Uri for which the security mitigation is provided. helm://… || docker://…
affectedPackageUri:
## Which chart versions this cve note belongs to
affectedVersions: mastermind SemVer constraint
## Description / note
description: text
https://github.com/jfrog/chartcenter/blob/master/docs/security-mitigation.yaml

Here is an example of what these notes look like on ChartCenter

How charts create reproducible security
Organizations do not have to
replicate each security step.
If teams are distributed throughout the
world and have multiple environments,
such as test, QA, staging and production.
Immutable Conﬁgurations can be shared
Feat
Test QA
Stage
Prod
Chart version: 1.5.1

Debugging
helm lint is your go-to tool for verifying that your chart follows best practices
helm install --dry-run --debug or helm template --debug: We've seen this trick already.
It's a great way to have the server render your templates, then return the resulting
manifest file.
helm get manifest

Build and Deploy
$ helm install my-demo-chart demochart/ --values demochart/values.yaml
Release “my-demo-chart” has been upgraded. Happy Helming!

Wrapping Up:
Blueprint for Security

Navigating Security
Chart Hashes
Signing Charts
Secrets Management
Helm Lint
RBAC and Service
Accounts
POD Security
Dependencies
Certificates

Starting a Blueprint for Security
Using a trusted repository
Helm repo add...
Checking dependencies
CVE Information
Reviewing Mitigation Notes
When creating a chart
Signing a chart with gpg
Verification (Provenance and SHA256)
Encrypting secrets
Helm lint
Certificates
Permissions with RBAC
Pod Security Policy (PSP)
Linting and Debugging

Chartcenter.io
$ helm repo add center https://repo.chartcenter.io
$ helm repo update
Next, you can use this command to see a list of all the charts available in
ChartCenter:
$ helm search repo center/
To check a specific Helm repository, you can use something like this:
$ helm search repo center/jfrog
$ helm install jfrog center/jfrog/artifactory-jcr

Q&A
deepd@jfrog.com
Twitter @DeepDattaX

Helm Charts Security 101

More Related Content

What's hot (20)

Similar to Helm Charts Security 101 (20)

More from Deep Datta (12)

Recently uploaded (20)

Helm Charts Security 101