Kube cfg-mgmt

Kubernetes Configuration
Management
Lee Briggs
Principle Infrastructure Engineer
07/11/2018

© 2016 Apptio, All rights reserved (v2.5)2
$(whoami)
 Based in Seattle (formerly London)
 Work for Apptio
 We are hiring! (remote US, remote EMEA, remote APAC)
 Using Kubernetes since 2015
 We tried out v1.0. It was a bad idea.
 Multiple cluster over many environments/regions
 Github:
https://github.com/jaxxstorm
https://github.com/apptio
 Twitter:
https://twitter.com/briggsl
 Blog:
https://www.leebriggs.co.uk

Quick Overview
 Remove “snowflakes”
 Hand crafted servers in your environment
 Provision New Servers
 Version control of Server configuration
 Replication of server environments

Cfg Mgmt Players

Enter Kubernetes
Or insert your fav orchestrator here

Configuration Layer
 With the classic tools, the configuration layer is the server
 Apps, config and resources are managed at the individual server level
 K8s provides an abstraction layer above many servers
 API Driven
 Idempotent
 Convergent
 Lots of orgs run only a few clusters
 Configuring the clusters relatively straightforward.
 K8s solves a lot of the server config mgmt. problem
 DaemonSets are a good example here

“Components”
 Components is a term we use to describe a thing that runs on
Kubernetes that is needed for a cluster to be useful
 Examples of components:
 Ingress Controller: https://github.com/kubernetes/ingress-nginx
 Cluster Autoscaler: https://github.com/kubernetes/autoscaler
 Sealed Secrets: https://github.com/bitnami-labs/sealed-secrets
 RBAC Config: https://kubernetes.io/docs/reference/access-authn-authz/rbac/
 Prometheus: https://prometheus.io/
 These components are vital for an operating cluster
 We need to install them on all clusters!

Components: What’s the problem?
 A lot of the config for these components is the same
 But, a lot is slightly different:
 Path to the SSL certificate for the Ingress Controller
 List of endpoints to scrape for Prometheus
 RBAC config between dev and prod
 ASG names for cluster autoscaler
 Different namespaces on different clusters
 How can you manage this using the traditional config management tools?
 Spoiler Alert: You can’t!

Helm
 Helm seems to be the #1 tool for this
 Lots of helm charts written by the community
 Allows you to set “values” for config in manifests
 You can specify these values via the cmdline or via a yaml based “values” file
 Templated Golang
 Relatively shallow learning curve

Helm: The downsides
 Helm is a security nightmare
 Cluster admin access for tiller!
 Helm 3 addressing this
 Managing values files doesn’t help
 You end up with lots of <dc>-values.yml for each DC
 Templated Golang is a not fun
 Having to range through values with templates became a chore
 Once you have all the config, how do you apply it to multiple clusters?
 You need to specify the right values file for each cluster
 We tried to use ansible and puppet for this, but it didn’t work

Helm: The downsides

Ansible
 Ansible supports using clusters as endpoints
 You simply define the API endpoint when using kubernetes manifests
 Has a k8s_raw module which can be used
 This allows you to hit specific endpoints when running ansible
 Also has a helm module
 Unfortunately it’s very broken

Ansible: The downsides
 Ansible uses yaml
 K8s already has us in yaml hell 
 You’re changing one yaml templating tool (Golang) with another (jinja2)
 Templating yaml is not fun
 Using the k8s_raw module meant fully writing all deployments
 You can’t easily make use of existing helm charts
 You would need to write and then retemplate all the config you’d need 
 Anecdotally, setting up python for the required libraries wasn’t very fun

Terraform
 Terraform has a nice HCL language construct
 Makes writing JSON much easier
 Has a Kubernetes & Helm Provider
 The Kubernetes provider is ”official”
 The Helm provider is community based
 Is easily extensible

Terraform: The downsides
 Both the helm and kubernetes provider weren’t very active
 No merges for a long time
 Seemed extremely buggy
 Often would error hitting APIs
 Because the k8s API changes slightly with each release, issues occurred
 For example, as APIs moved from beta to stable, the provider needs updating
 It needs to be continuously updated for each k8s release

Some other options:
 Ksonnet
 Seemed promising
 Dramatically overcomplicated
 You have to use the ksonnet ecosystem and way of deploying
 Kapitan
 Jinja2 templates 
 Was very promising conceptually
 Has to be pure jsonnet, no support for existing helm templates etc ]
 Pulumi
 Relatively new player in this space
 Terraform based
 Requires subscription to use now 

Jsonnet
 It became clear very quickly that using jsonnet was desirable
 Language written by Google specifically for interacting with JSON
 Used in both ksonnet and kapitan
 Has Golang wrappers, easy to embed into Go apps
 External variables mean ability to template JSON
 Pass in a variable at compile time, it changes the JSON

Design Goals
 Simple
 As few moving parts as possible
 Only write parts we can’t find
 We are a very small team, we don’t want to manage a large codebase
 Opinionated
 We wanted to enforce a structure that works for us
 Helps when fixing bugs
 Flexible
 Ability to render components based on Helm or any other yaml source input

What is kr8?
 Attempt to leverage the power of jsonnet
 Render jsonnet and other things into usable manifests for each cluster
 Written in Go
 Open Source!
 Features:
 Automatic population of jsonnet external variables
 Automatic concatenation of jsonnet files
 Ability to patch helm charts
 Uses jsonnet to render yaml, then patches the json on top of that
 Ability to leverage other config tools like Kasane and Kustomize

Other Automation Components
 Task: https://github.com/go-task/task
 Like Make. Written in Go. Takes yaml/json config
 Instead of writing makefiles, we make templated jsonnet taskfiles
 Helm: https://helm.sh/
 We use helm charts, render them locally and then patch them with jsonnet
 Kubecfg: https://github.com/ksonnet/kubecfg
 Applies manifests in a sane way
 Can use ”garbage collection” to remove unused manifests

Future Tasks
 Split components into separate repos
 Allow them to be downloaded like Terraform modules
 Unit Tests
 Documentation
 Pull requests welcome!

Links
 Introductory Blogpost: https://leebriggs.co.uk/blog/2018/11/07/kr8-
kubernetes-config-mgmt.html
 kr8: https://github.com/apptio/kr8
 Example Configs: https://github.com/apptio/kr8-configs
 My Configs! https://github.com/jaxxstorm/cluster_config

Thankyou!
Colin Spargo
Sanyu Melwani
Shawn Xue

Kube cfg-mgmt

More Related Content

What's hot (20)

Similar to Kube cfg-mgmt (20)

Recently uploaded (20)

Kube cfg-mgmt

Editor's Notes