Kubernetes for Developers - 7 lessons learned from 7 data centers in 7 months (meetup)

© 2019 Adobe. Kubernetes for Developers Meetup – May 13, 2019
Kubernetes - 7 lessons learned from 7 data centers in 7 months
Kubernetes for Developers Meetup – May 13, 2019
Mike Tougeron –
Senior Site Reliability Engineer @
Adobe

$ whoami && id | grep Adobe
 Mike Tougeron
 Senior Site Reliability Engineer @ Adobe
 Twitter: @mtougeron
 Started using Kubernetes in 2015

Agenda
 Quick Introduction to Adobe Advertising Cloud’s Kubernetes Infrastructure
 Lesson 1: Communication, Teamwork & Training
 Lesson 2: Code to production pipelines
 Lesson 3: The ABCs of Production apps
 Lesson 4: Multi-cloud challenges
 Lesson 5: Knowing your application
 Lesson 6: Metrics based monitoring
 Lesson 7: Take a deep breath

High Traffic
350 billion requests
a day
Latency
<50ms @ 95th
percentile
Huge Datasets
Billions of objects to
store

Adobe Advertising Cloud’s Kubernetes Overview
 ~225 worker nodes; growing to ~300
in May/June
 6 OpenStack data centers across 4
regions
 Running on VMs
 No persistent storage
 No autoscaling; “fixed” footprint
 Smaller but growing
 3 AWS clusters in us-east-1
 Running on m5d.12xlarge ec2 instances
 EBS volumes for persistent storage
 Uses cluster-autoscaler
 Autoscaling events many times per hour
 Prometheus for monitoring
 Dozens of Machine Learning
workloads in AWS
 Reason for frequent autoscaling events
 Cluster updates done via new Image
and rolling update of existing nodes
 Updates are deployed approx every
4-6 weeks

Lesson 1: Communication, Teamwork & Training

Communication: Reaching large, distributed teams

Teamwork: Who’s responsible for what?

Abstraction vs Experts
 Need understanding of core
resources but also need easy
onboarding
 Pair programming training sessions
 Remove need for boiler plate
 Don’t duplicate efforts by avoiding
abstraction
 Don’t abstract to the point where
you’re not using Kubernetes
 kubectl should *not* be your
entrypoint

Lesson 2: Code to production pipelines
De
v
Pull
Request
maste
r
Unit
testin
g
merge
Deplo
y bot
Production
Integration
testing
Insert your steps here!

Tools to help build application resources
 Helm (templating and/or tiller)
 Kustomize
 Kapitan
 and more…
 We use a combination of Helm
templating for infrastructure/3rd-party
and Kustomize for application teams

$> helm template --name opa --namespace opa --values ./values/globals.yaml
--values ./values/mgmt/cluster.yaml --values ./values/mgmt/adcloud-
opa/values.yaml --output-dir ../../../cloud/opa/mgmt charts/adcloud-opa
versus
$> ./build.py --chart adcloud-opa --cluster mgmt

Lesson 3: The ABCs of Production
 HorizontalPodAutoscaler
 PodDisruptionBudget
 "DevOps"
 Cluster Upgrades

HorizontalPodAutoscaler
 Easily scale on CPU or Memory usage
 Also able to scale on custom metrics like
http_requests from Ingress resources
 Don’t set replicas in your Deployment

PodDisruptionBudget
 Not the same thing as a Deployment
strategy
 Helps prevent taking down so many Pods
that the application is overwhelmed
 Can set by minAvailable or
maxUnavailable by number or
percentage
 Good for helping keep quorum
 Doesn’t apply to manual deletions

DevOps
 Expertise/specialists
 But empowerment & speed
 Things get lost in shuffle
 Everyone can do everything; aka don’t forget your guardrails

deny[msg] {
input.request.kind.kind = "Ingress"
input.request.operation = "CREATE"
host = input.request.object.spec.rules[_].host
ingress = ingresses[other_ns][other_ingress]
other_ns != input.request.namespace
ingress.spec.rules[_].host = host
msg = sprintf("invalid ingress host %q (conflicts with
%v/%v)", [host, other_ns, other_ingress])
}
patch[patchCode] {
isCreateOrUpdate
input.request.kind.kind == "Ingress"
not hasAnnotation(input.request.object,
"kubernetes.io/ingress.class")
patchCode =
makeAnnotationPatch("add",
"kubernetes.io/ingress.class", "nginx-
internal", "")
}

Cluster Upgrades - Blue/Green or Canary?
 Who really has the hardware to run a 2nd full
Kubernetes cluster in their datacenter?
 Public cloud is easier, but you still have cost
considerations
 Are the application team(s) able to handle
deploying to a 2nd mirrored cluster?
 Does it make more sense to run N workers of a
different version/config for a period of time?
 Do you have the visibility into the cluster to know
how one performs vs the other?

Lesson 4: Multi-Cloud Challenges

Multiple code-bases but consistent infrastructure
 Packer – Shared modular code base, different builders
 Terraform – Separate but closely aligned code bases
 Puppet – Same code base
 Helm – Same modular code base
 Leverage templating to build the same deployments for
different (and future) clouds
 Re-use, re-use, re-use!
 Lab environments in all clouds
 OSSIA for HV/rack metadata for region/zone

Lesson 5: Knowing your applications
 Seems like an obvious statement but it’s easy to forget to
think about
 Kubernetes brings advantages, but not all the ones that
bare metal and virtual machines bring out of the box
 Think about how your app actually functions
 Service Discovery
 Persistent Storage
 Shared Storage (e.g. replication, sharding, etc)
 Scheduling / Restarting
 Networking Ingress / Egress
 Think about how your app is going to handle the way
Kubernetes does things
https://imgur.com/gallery/B4D7Lf1

Elasticsearch as Deployment (What We Did)
https://www.slideshare.net/JoergHenning/elasticsearch-on-kubernetes (slightly
modified)

Oops…yeah Touge, I think something is wrong…

Elasticsearch as StatefulSet (What We Should Have Done)
https://www.slideshare.net/JoergHenning/elasticsearch-on-kubernetes (slightly
modified)

Lesson 6: Metrics-Based Monitoring

Lesson 7: Take a deep breath
 Same team so we all learn & fix together
 Experience has been enlightening &
engineers have had fun
 Teams already onboarded are moving
faster than before
 Dev cycle to production is faster as we
integrate more automated testing

Thanks!
Slides: https://touge.me/k8s-7lessons-meetup
Mike Tougeron
Email: tougeron@adobe.com
Twitter: @mtougeron
Images from https://stock.adobe.com

Kubernetes for Developers - 7 lessons learned from 7 data centers in 7 months (meetup)

More Related Content

Similar to Kubernetes for Developers - 7 lessons learned from 7 data centers in 7 months (meetup) (20)

Recently uploaded (20)

Kubernetes for Developers - 7 lessons learned from 7 data centers in 7 months (meetup)

Editor's Notes