The Return of the Dull Stack Engineer

The Return of the DullThe Return of the Dull
Stack EngineerStack Engineer
Kris Buytaert
@krisbuytaert

Kris BuytaertKris Buytaert
● I used to be a Dev,I used to be a Dev,
● Then Became an OpThen Became an Op
● CTO and Open Source ConsultantCTO and Open Source Consultant
@@inuits.euinuits.eu
● Everything is a freaking DNSEverything is a freaking DNS
ProblemProblem
● Evangelizing devopsEvangelizing devops
● Organiser of #devopsdays,Organiser of #devopsdays,
#cfgmgmtcamp, #loadays, ….#cfgmgmtcamp, #loadays, ….

C(L)AMSC(L)AMS
● CultureCulture
● (Lean)(Lean)
● AutomationAutomation
● MeasurementMeasurement
● SharingSharing
Damon Edwards and JohnDamon Edwards and John
Willis

A global movement to improve theA global movement to improve the
quality of software delivery leveragingquality of software delivery leveraging
Open Source experience, started in GentOpen Source experience, started in Gent
in 2009in 2009

Because the old days:Because the old days:
● ““Put this Code Live, here's a tarball”Put this Code Live, here's a tarball”
NOW!NOW!
● What dependencies ?What dependencies ?
● No machines available ?No machines available ?
● What database ?What database ?
● Security ?Security ?
● High Availability ?High Availability ?

10 days into operations10 days into operations

Topic Evolution : CultureTopic Evolution : Culture
● Agile, Kanban, LeanAgile, Kanban, Lean
● Startups as examplesStartups as examples
● UnicornsUnicorns
● Poster ChildsPoster Childs
● Agile, Fake Agile, Less,Agile, Fake Agile, Less,
Safe,Safe,
● Enterprises asEnterprises as
examplesexamples
● Engage managementEngage management
● BurnoutBurnout

Topic Evolution : AutomationTopic Evolution : Automation
● Cfengine, Puppet, ChefCfengine, Puppet, Chef
● Mcollective, Noah,Mcollective, Noah,
RundeckRundeck
● Hudson, JenkinsHudson, Jenkins
● EC2, OpenStack,EC2, OpenStack,
Cloudstack,Cloudstack,
Eucalyptus,OpenNebulEucalyptus,OpenNebul
a, openQRMa, openQRM
● Bash in Yaml “Basic”Bash in Yaml “Basic”
● Ansible, Rundeck,Ansible, Rundeck,
ChoriaChoria
● Jenkins, GitlabCI,Jenkins, GitlabCI,
CircleCI, TravisCICircleCI, TravisCI
● EC2, GCE, Azure,EC2, GCE, Azure,
OpenStack,OpenStack,

Topic Evolution : AutomationTopic Evolution : Automation
● Xen, KVM, openVZ, lxcXen, KVM, openVZ, lxc
● VagrantVagrant
● Linux-HA, CoroSyncLinux-HA, CoroSync
● Ceph, Gluster, DRBD,Ceph, Gluster, DRBD,
● Docker , Docker,Docker , Docker,
Docker, Docker, Moby,Docker, Docker, Moby,
Rkt, lxcRkt, lxc
● CR-IOCR-IO
● Docker , Docker,Docker , Docker,
Dockrer,Mesos,Dockrer,Mesos,
KubernetesKubernetes
● NFSNFS

Topic Evolution : MonitoringTopic Evolution : Monitoring
● NagiosNagios
● CactiCacti
● MuninMunin
● RRDRRD
● Zenoss, ZabbixZenoss, Zabbix
● #monitoringsucks#monitoringsucks
● IcingaIcinga
● ELKELK
● Graphite + GrafanaGraphite + Grafana
● CollectdCollectd
● PrometheusPrometheus
● #monitoringlove#monitoringlove

Culture vs ToolsCulture vs Tools

These DaysThese Days
● ““Put this Code Live, here's a DockerPut this Code Live, here's a Docker
Container ”Container ”
● No machines available ?No machines available ?
● What database ? Where to store theWhat database ? Where to store the
data ?data ?
● Security ? What distro is this even ?Security ? What distro is this even ?
Bad Cows ?Bad Cows ?
● How do we monitor his ?How do we monitor his ?
● Backups ?Backups ?
● How did you build this ?How did you build this ?

Tools will not fix yourTools will not fix your
broken culturebroken culture
Tooling hype is not helpingTooling hype is not helping

Hype SeasonHype Season
● Conference Visitors numbers skyrocketConference Visitors numbers skyrocket
•
OpenStack 3000+ marketeersOpenStack 3000+ marketeers
•
DockerCon : 1000+ devs & unicornsDockerCon : 1000+ devs & unicorns
•
Kube Con : 1500+ unicorns & vendorsKube Con : 1500+ unicorns & vendors
● Everybody claims to be doing it , nobodyEverybody claims to be doing it , nobody
is actually doing it.is actually doing it.

Why adopt shiny new stuff ?Why adopt shiny new stuff ?
● Attention Span ?Attention Span ?
● Resume building ?Resume building ?
•
0.5 year = novice in 20 techs0.5 year = novice in 20 techs
vsvs
•
7 year = expert in 4 techs7 year = expert in 4 techs

Works on My MachineWorks on My Machine

New != BetterNew != Better
● It’s called Bleeding edge for aIt’s called Bleeding edge for a
reasonreason
● Missing FeaturesMissing Features
● OperatabilityOperatability
● StabilityStability

Hello world todayHello world today
● Micro service architectureMicro service architecture
● Displaying content in a fileDisplaying content in a file
● Kubernetes 2.6+ StorageKubernetes 2.6+ Storage
● OpenStackOpenStack

Complexity is theComplexity is the
enemy of reliabilityenemy of reliability

U are not facebookU are not facebook

U are not LinkedinU are not Linkedin
● Kafka is awesomeKafka is awesome
● But you have 100 messages an hourBut you have 100 messages an hour

Welcome to EuropeWelcome to Europe

Why don’t ops adopt ?Why don’t ops adopt ?
● They do ..They do ..
•
It just takes more timeIt just takes more time
•
More worriesMore worries
•
More careMore care

Improve your sleepImprove your sleep
Put your {devs,managers,qa}Put your {devs,managers,qa}
ON CALL !ON CALL !

Why don’t ops adopt ?Why don’t ops adopt ?
● We already fixed that problemWe already fixed that problem
● We have a different use caseWe have a different use case
● That won’t work becauseThat won’t work because
•
AA
•
BB
•
CC
● Cool lets try it ..Cool lets try it ..

Puppet AdoptionPuppet Adoption
● Focus on the wrong partFocus on the wrong part
•
Expert in PuppetExpert in Puppet
•
Doesn’t understand apache configDoesn’t understand apache config

Operations trying to adoptOperations trying to adopt

$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$
● We don’t have timeWe don’t have time
● Initial Budget was too low,Initial Budget was too low,
unrealisticunrealistic
● Existing Technical DebtExisting Technical Debt
● FirefightingFirefighting
● Time pressured projectTime pressured project

Operations
Developers
Security

Workload is invertedWorkload is inverted
● X devs work on 1-2 concurrentX devs work on 1-2 concurrent
projectsprojects
● X/20 ops/security work on all theX/20 ops/security work on all the
projects the devs touched over theprojects the devs touched over the
past 5 yearspast 5 years
● AND need to build the next genAND need to build the next gen
platformplatform

● GrumpyGrumpy
•
Woken up oneWoken up one
to many timesto many times
for the wrongfor the wrong
reasonreason
● Old , akaOld , aka
ExperiencedExperienced
Typical Ops DudeTypical Ops Dude

The Return of the Dull Stack Engineer

Devops = CoolDevops = Cool
● All the cool kids are doing thisAll the cool kids are doing this
● We want to be coolWe want to be cool
● People only want to work for coolPeople only want to work for cool
companiescompanies
● Banks etc are boringBanks etc are boring

FOMOFOMO
● We need to do something CoolWe need to do something Cool
•
Docker Docker DockerDocker Docker Docker
•
Bi ModalBi Modal

Broken StrategiesBroken Strategies
● Hiring Devops EngineersHiring Devops Engineers
● Building Devops TeamsBuilding Devops Teams
● Docker Docker Docker => VM’s asDocker Docker Docker => VM’s as
ContainersContainers
s/ops/oops/g;s/ops/oops/g;

YOLOYOLO
● StartupStartup
● VCVC
● Exit StrategyExit Strategy
● BusinessBusiness
● CustomerCustomer
● SurvivalSurvival

Late AdoptionLate Adoption
● You need to do IACYou need to do IAC
● We don’t need IACWe don’t need IAC
● You really need to do IACYou really need to do IAC
● IAC is too difficult for us ..IAC is too difficult for us ..
● Really you need to do IACReally you need to do IAC
● We’re doing Ansible now …We’re doing Ansible now …

0 day software0 day software
0900: “Whow Hashicorp released0900: “Whow Hashicorp released
Consul X.Y”Consul X.Y”
Let’s have this in prod by thisLet’s have this in prod by this
afternoonafternoon
Ego driven developmentEgo driven development

R&D vs ProdR&D vs Prod
● Schedule time to Research new topicsSchedule time to Research new topics
● Not all of them are irrelevantNot all of them are irrelevant
● But limit the number of patterns you need to manageBut limit the number of patterns you need to manage
● e.g Limited number of OS Variants, Deploymente.g Limited number of OS Variants, Deployment
Patterns, Storage BackendsPatterns, Storage Backends

Stop DreamingStop Dreaming
There is no way your 3 person ops team can manageThere is no way your 3 person ops team can manage
and scale VMWare, EC2, Azure, OpenStack, RHEV,and scale VMWare, EC2, Azure, OpenStack, RHEV,
Gluster, DRBD, LibVirt,NFS, Ceph, Mongo, MySQL,Gluster, DRBD, LibVirt,NFS, Ceph, Mongo, MySQL,
PgSQL, Cassandra, mcollective, Rabbit, collectd,PgSQL, Cassandra, mcollective, Rabbit, collectd,
dhcpd, sonar, aptly,ActiveMQ, Kafka, Nomad,dhcpd, sonar, aptly,ActiveMQ, Kafka, Nomad,
zookeeper, haproxy, iptables, Consul, Etcd, Puppet,zookeeper, haproxy, iptables, Consul, Etcd, Puppet,
Cacti, corosync, Docker Distribution, vault, bind,Cacti, corosync, Docker Distribution, vault, bind,
owncloud, openswan,Nexus, Redmine, diamond,owncloud, openswan,Nexus, Redmine, diamond,
Jenkins,Mule, GoCD, Chef, memcached, foreman, PHP,Jenkins,Mule, GoCD, Chef, memcached, foreman, PHP,
Java,Ruby, nextcloud, Kubernetes, pulp, Spring Boot,Java,Ruby, nextcloud, Kubernetes, pulp, Spring Boot,
Ansible, Django, Apache Storm, redis, Nginx, ApacheAnsible, Django, Apache Storm, redis, Nginx, Apache
httpd, Jboss, Jetty, Hadoop, Elastic, Graphite, Grafana,httpd, Jboss, Jetty, Hadoop, Elastic, Graphite, Grafana,
freeswan, Graylog, Apache Spark, Bamboo, Jirafreeswan, Graylog, Apache Spark, Bamboo, Jira

Stop DreamingStop Dreaming
There is no way your 3 person ops team can manageThere is no way your 3 person ops team can manage
and scaleand scale VMWare,VMWare, EC2,EC2, Azure,Azure, OpenStack,OpenStack, RHEV,RHEV,
Gluster, DRBD, LibVirt,Gluster, DRBD, LibVirt,NFS,NFS, Ceph, Mongo, MySQL,Ceph, Mongo, MySQL,
PgSQL,PgSQL, Cassandra,Cassandra, mcollective, Rabbit, collectd,mcollective, Rabbit, collectd,
dhcpd, sonar, aptly,ActiveMQ,dhcpd, sonar, aptly,ActiveMQ, KafkaKafka,, Nomad,Nomad,
zookeeper, haproxy, iptables, Consul, Etcd, Puppet,zookeeper, haproxy, iptables, Consul, Etcd, Puppet,
Cacti,Cacti, corosync,corosync, Docker Distribution,Docker Distribution, vault, bind,vault, bind,
owncloud, openswan,Nexus, Redmine,owncloud, openswan,Nexus, Redmine, diamond,diamond,
Jenkins,Jenkins,Mule, GoCD, Chef, memcached,Mule, GoCD, Chef, memcached, foreman, PHP,foreman, PHP,
Java,Ruby, nextcloud, Kubernetes, pulp,Java,Ruby, nextcloud, Kubernetes, pulp, Spring Boot,Spring Boot,
Ansible, Django, Apache Storm, redis, Nginx, ApacheAnsible, Django, Apache Storm, redis, Nginx, Apache
httpd,httpd, Jboss, Jetty,Jboss, Jetty, Hadoop,Hadoop, Elastic, Graphite, Grafana,Elastic, Graphite, Grafana,
freeswan, Graylog, Apache Spark, Bamboo, Jirafreeswan, Graylog, Apache Spark, Bamboo, Jira

Full Stack EngineerFull Stack Engineer
● Knows the MEAN StackKnows the MEAN Stack
● Is still confused aboutIs still confused about
the Javascriptthe Javascript
framework he wants toframework he wants to
use yesterdayuse yesterday
● Is familiar with all theIs familiar with all the
previously mentionedpreviously mentioned
toolstools
● Understands the LinuxUnderstands the Linux
KernelKernel
● Debugs Java CodeDebugs Java Code
● Understands NetworkingUnderstands Networking
and Securityand Security
● Understands deploymentUnderstands deployment
● Tunes DatabasesTunes Databases

Dull Stack EngineerDull Stack Engineer

Dull Stack EngineerDull Stack Engineer
● Knows the stack in depthKnows the stack in depth
● Invests in peopleInvests in people
● Knows his coworkers in depthKnows his coworkers in depth
● Is there for the business, not for theIs there for the business, not for the
fun of the new toolfun of the new tool

$value$value
● Devops is about generating valueDevops is about generating value
for your organisation , togetherfor your organisation , together
● What’s the value of shiny $tool vsWhat’s the value of shiny $tool vs
boring tool ?boring tool ?

Back to the rootsBack to the roots
● Collaborating for a better platformCollaborating for a better platform
● Open SourceOpen Source
● EmpathyEmpathy
● Common SenseCommon Sense

ContactContact
Kris Buytaert kris.buytaert@inuits.euKris Buytaert kris.buytaert@inuits.eu
Further ReadingFurther Reading
@krisbuytaert@krisbuytaert
http://www.krisbuytaert.be/blog/http://www.krisbuytaert.be/blog/
http://www.inuits.eu/http://www.inuits.eu/
Find Inuits inFind Inuits in
Brasschaat,GhentBrasschaat,Ghent
,,
Rotterdam,PraguRotterdam,Pragu
e,e,

The Return of the Dull Stack Engineer

More Related Content

What's hot (20)

Similar to The Return of the Dull Stack Engineer (20)

More from Kris Buytaert (18)

Recently uploaded (20)

The Return of the Dull Stack Engineer