Longhorn PHP - Stop Doing It Wrong

STOP THE WRONG
DEVOPS - WHY YOU DO THINGS

04/20/2018 - STOP THE WRONG
BOYD HEMPHILL - CTO - VICTORY CTO
▸ Community guy …
▸ Austin DevOps Meetup
▸ DevOps Days Austin
▸ Container Days Austin
▸ Docker Austin
▸ Former - Austin PHP Meetup Host
▸ Former - Austin MySql Meetup
▸ Developer, DBA, Ops, Director, Exec
▸ I was The Director of DevOps. Be sure to pwn me
after this talk!
▸ Extroverted Nerd

VICTORY
CTO
WE DON’T WIN
BUSINESS, WE HELP
BUSINESSES WIN.

STOP THE WRONG
RULES OF ENGAGEMENT

RULES OF ENGAGEMENT
▸ You will be uncomfortable - Good

RULES OF ENGAGEMENT
▸ You will want to yell at me - Good

RULES OF ENGAGEMENT
▸ You will think I am wrong - I think
you’re wrong too, but I have the
mic

RULES OF ENGAGEMENT
mic
▸ I don’t necessarily believe what I
am saying. But I will argue it with
you to our mutual beneﬁt.

RULES OF ENGAGEMENT
mic
▸ I don’t necessarily believe what I
am saying. But I will argue it with
you to our mutual beneﬁt.
▸ Don’t get your knickers in a twist

THERE ARE INFINITE WAYS TO SCREW
UP. THERE ARE ABOUT A DOZEN THAT
ARE RIGHT THEREFORE, THE
PROBABILITY OF GETTING IT RIGHT BY
CHANCE = 0.
Math

ENGINEER THE AGILES!
LETS TALK ABOUT YOUR AGILE
DEPARTMENT …

AGILE DEVELOPMENT IS A PHILOSOPHY
▸ Agile has a manifesto.

▸ Agile can be applied in many
different situations.

▸ There is no department of the
Agiles

▸ There is no department of the
Agiles
▸ Scrum is one model of the Agile
philosophy that is frequently
implemented

▸ There is no department of the Agiles
▸ Scrum is one model of the Agile
philosophy that is frequently
implemented
▸ Scrum sucks if your on an ops team.
Agile does not.

SCRUM IS A TRAIL OF TEARS ON
OPS TEAMS. AGILE IS A BREATH OF
LIFE. THESE ARE NOT A
CONTRADICTORY STATEMENTS.
School of Hard Knocks

DEVOPS IS A PHILOSOPHY
▸ Philosophies have pillars.

▸ C.A.L.M.S.

▸ C.A.L.M.S.
▸ I really would rather a fabric
metaphor. Each pillar is really
strands of interwoven fabric. The
weave determines the strength of
the model.

DEVOPS IS THE WAY A TECHNOLOGY
PRACTICE EMBEDS ITSELF IN AN
ORGANIZATION TO THE BENEFIT OF
THAT ORGANIZATION.
Some Know-it-all

HAMMERS & NAILS
LET’S TALK ABOUT YOUR TOOL CHOICES
…

THINKING DEVOPS
▸ I have a hammer, so everything
looks like a … (Jenkins)

THINKING DEVOPS
▸ Why are you implementing a CI
tool? - Philosophy

THINKING DEVOPS
tool? - Philosophy
▸ Does it align with your philosophy?
- Model

THINKING DEVOPS
tool? - Philosophy
▸ Does it align with your philosophy?
- Model
▸ How will you make it happen? -
Implementation

THINKING DEVOPS
▸ I have a hammer, so everything looks
like a … (Jenkins)
▸ Why are you implementing a CI tool?
- Philosophy
▸ Does it align with your philosophy? -
Model
▸ How will you implement it? -
Implementation
▸ What tool should it be? - Tool
(https://bit.ly/1fkKn1Q)

IF YOU DON’T KNOW WHAT YOU ARE
DOING, AUTOMATION ONLY MAKES
YOU DO IT FASTER.
Some Know-it-all

JENKINS IS A SIRENS CALL
▸ Swiss army knife allure. Common.

▸ Doesn’t allow the build to be
described in code

▸ Doesn’t allow the build to be
described in code
▸ Creates immense friction for build
management, ownership and
change.

KNOWLEDGE OF A TOOL HELPS TO
UNDERSTAND THE BROADER
PROBLEM. DON’T USE CRAP
BECAUSE YOU KNOW HOW TO POOP.
Some Know-it-all

RUNDECK NEEDS A REWRITE
▸ Job code stored as XML via the
software on disk.

software on disk.
▸ No developer workﬂow

software on disk.
▸ This is the worst of old school
systems administration - hack, cut,
paste, pray.

software on disk.
▸ This is the worst of old school
systems administration - hack, cut,
paste, pray.
▸ What would a tool born of DevOps
philosophy really look like?

DEVOPS ASSUMES THE AGILE
PHILOSOPHY AND USES THE
MODELS OF WORKFLOWS TYPICAL
TO DEVELOPERS.
That Same Know-it-all

LEARNING ORGS
YOU DON’T KNOW JACK … GET OVER IT,
YOU NEVER WILL

YOU ONLY NEED TWO SKILLS …
▸ The ability to own your outcomes
▸ Owning outcomes your not done
until the feature is in production,
defect free and driving revenue.

YOU ONLY NEED TWO SKILLS …
▸ The ability to own your outcomes
▸ Owning outcomes your not done
until the feature is in production,
defect free and driving revenue.
▸ The ability to rapidly learn and
implement
▸ Implement implies adhering to the
philosophy driving your
organization.

SKILL ACQUISITION IS YOUR
PRIMARY SKILL.
Paul Czarkowski

GO FORTH AND DEVOP
▸ Once you have chosen a way
forward learn why and how others
did it.

GO FORTH AND DEVOP
did it.
▸ Learn about common mistakes.

GO FORTH AND DEVOP
did it.
▸ Get certiﬁed while doing the ﬁrst
implementation.

GO FORTH AND DEVOP
did it.
▸ Get certified while doing the first
implementation.
▸ Start certification bounties in your
organization.

IF YOU ARE NOT A LEARNING
ORGANIZATION YOU ARE LOSING TO
ONE.
Andrew Clay Shafer (knows a lot!)

SILO COMPOSITION
THE DEPARTMENT OF THE DEVOP …

SILO EFFECTS - 1
▸ Developer introduces a
performance killing bug.

SILO EFFECTS - 1
▸ Site crashes.

SILO EFFECTS - 1
▸ Site crashes.
▸ “DevOps” (see how stupid that
term is in this context) wakes up
and solves the problem

SILO EFFECTS - 1
▸ Site crashes.
▸ “DevOps” (see how stupid that
term is in this context) wakes up
and solves the problem
▸ Developer sleeps through the
night.

DEVELOPERS ARE SELF ABSORBED
ASSHOLES WHO ARE TOO LAZY TO
ENSURE THEIR SHIT WORKS.
Ops who can’t Dev

SILO EFFECTS - 2
▸ A developer needs infrastructure
to do performance testing.

SILO EFFECTS - 2
▸ Developer needs infrastructure to
do performance testing.
▸ He launches infrastructure and
tests.

SILO EFFECTS - 2
tests.
▸ He leaves it up.

SILO EFFECTS - 2
tests.
▸ He leaves it up.
▸ Director of DevOps budget is
blown. She jumps off a ledge.

DEVELOPERS ARE SELF ABSORBED
ASSHOLES WHO ARE TOO LAZY TO
ENSURE THEIR RESPONSIBILITIES.
Mangers who’ve never developed in the cloud.

CULTURE OF EMPATHY
▸ Developers introduce error to the
system because they are human,
not because they are jerks.

CULTURE OF EMPATHY
▸ Developers have no incentive to
care and no real view of the
consequences.

CULTURE OF EMPATHY
consequences.
▸ Saying, “You developers should
care” is not a solution. It is
codependent and stupid.

CULTURE OF EMPATHY
consequences.
▸ Saying, “You developers should
care” is not a solution. It is
codependent and stupid.
▸ Real world pain is a solution.

NOTHING SAYS, “I CARE,” LIKE,
“OUCH”. BUILD A CULTURE OF
EMAPTHY.
That Know-it-all (again)

SHARED PAIN
SOLUTION 1, DEVELOPERS ON CALL …

REASONS TO NOT TO …
▸ Developer: “But I don’t have the
right access”

right access”
▸ Developer: “I don’t know what to
do.”

right access”
do.”
▸ DevOps (dumb title, yeah?):
“Those snowﬂakes will just harsh
my mellow.”

right access”
do.”
▸ DevOps (dumb title, yeah?):
“Those snowﬂakes will just harsh
my mellow.”
▸ Developer: “That’s not what I
signed up for!”

SOLVE THE BLOODY PROBLEM! WE
ARE ENGINEERS RIGHT? WE SORTA
GET PAID TO SOLVE PROBLEMS?
That Know-it-all (again)

DEVELOPER:
“BUT I DON’T HAVE THE RIGHT ACCESS”
▸ Not the point.

DEVELOPER:
▸ Not the point.
▸ When there is a production
incident, nobody’s paycheck is
getting funded.

DEVELOPER:
▸ Not the point.
▸ When there is a production
incident, nobody’s paycheck is
getting funded.
▸ Your simple ability to triage an
event just saved some poor
burned-out soul from doing it for
the 1000th time.

DEVELOPER:
▸ Not the point.
▸ When there is a production incident,
nobody’s paycheck is getting funded.
▸ Your simple ability to determine it is
not an incident worth waking up at
the team just saved some poor
burned-out soul from doing it for the
1000th time.
▸ You don’t want to be the person who
introduced the fault.

ACCESS IS ONLY ONE ASPECT OF A
SOLUTION TO A PRODUCTION
INCIDENT. OFTEN KNOWLEDGE OF
THE CODE IS JUST AS IMPORTANT.
An Ops who Devs

DEVELOPER:
“I DON’T KNOW WHAT TO DO.”
▸ Not the point.

DEVELOPER:
▸ Not the point.
▸ Remember the thing about
learning organizations?

DEVELOPER:
▸ Not the point.
▸ You gain a very valuable skill set:
incident response team

DEVELOPER:
▸ Not the point.
▸ You gain a very valuable skill set:
incident response team
▸ You gain a valuable skill set: use
and conﬁguration of monitoring
and logging.

WE IN OPS DON’T KNOW EITHER.
BUT IT STILL NEEDS TO GET FIXED.
IF WE KNEW WHAT WAS WRONG, IT
WOULDN’T HAVE HAPPENED.
A cry for help

DEVOPS (DUMB TITLE, YEAH?):
“THOSE SNOWFLAKES WILL JUST HARSH MY MELLOW”
▸ Strike one. This person may need
to go.

to go.
▸ The “snowﬂakes” are intelligent
professionals who can learn to
read graphs and log messages.

to go.
▸ The “snowﬂakes” can be taught to
lead the incident so you can work
the incident.

to go.
▸ The “snowﬂakes” can be taught to
run the incident so you can work
the incident.
▸ Boredom is the goal. Hero’s mean
your organization is losing.

DON’T MOVE MY CHEESE.
Some angry dude in the unemployment line.

DEVELOPER:
“THAT’S NOT WHAT I SIGNED UP FOR”
to go.

DEVELOPER:
to go.
▸ Check the clause in your job
description that reads, “And other
duties as assigned …”

DEVELOPER:
to go.
▸ You can learn this stuff now or risk
unemployment later because you
cannot get through an interview
without it.

DEVELOPER:
to go.
▸ You can learn this stuff now or risk
unemployment later because you
cannot get through an interview
without it.
▸ World’s changing, get on board.

WHATEVER DUTIES YOU THINK YOU
HAVE TODAY WILL BE DIFFERENT IN
18 MONTHS. (UMM ….
SERVERLESS … SHHH)
The Future

NERD-ING TOGETHER
SOLUTION 2, SKILLS AND
PERSONALITIES

SELF ACTUALIZATION
▸ Developer - lines of code written.

SELF ACTUALIZATION
▸ Ops - Love to help out on one-off
problems.

SELF ACTUALIZATION
problems.
▸ Someone has to create the features
and someone must ensure the
underlying systems are secure and
available.

SELF ACTUALIZATION
problems.
▸ Someone has to create the features
and someone must ensure the
underlying systems are secure and
available.
▸ Both should be responsible for that
production outcome.

PASSIONS SHOULD ALIGN WITH
INCENTIVES AND NATURAL
CONSEQUENCES. ESPECIALLY IN THE
PRODUCTION ENVIRONMENT.
The Future

SKILLS AND CONSEQUENCES
▸ Developer - create features,
introduce defects

introduce defects
▸ Ops - Problem Solvers, abhor risk

introduce defects
▸ There is a risk in humans
introducing change.

introduce defects
▸ There is a risk in humans
introducing change.
▸ There is a need to engineer the risk
to a minimum. This outcome is
owned by both skills. (e.g. unit
tests & monitoring)

RISK IS JUST ANOTHER
ENGINEERING PROBLEM.
ENGINEERING PROBLEMS ARE ALSO
KNOW AS JOB SECURITY.
The practical know-it-all

ACCESS
▸ Developer - only in their local
environment, never production

ACCESS
▸ Ops - Thinking about production,
never about the development
environment.

ACCESS
▸ Ops - Thinking about production,
never about the development
environment.
▸ Environment parity is a key success
metric.

PRODUCTION CANNOT PRODUCE
MORE MONEY, IF DEVELOPERS
CANNOT EFFECTIVELY DELIVER NEW
FEATURES TO MARKET.
DevOps

A TEAM IS A COLLECTION OF SKILLS
▸ In soccer there are goalies,
defenders, mid ﬁelders and
strikers. The are on the same ﬁeld
at the same time with responsibility
for the same outcome.

A TECH TEAM IS A COLLECTION OF SKILLS
▸ In tech there are multiple teams on
the same ﬁeld with multiple
orthogonal outcomes.

▸ In tech there are multiple teams on
the same ﬁeld with multiple
orthogonal outcomes.
▸ Which scenario makes more
sense?

▸ In soccer there are goalies, defenders,
mid fielders and strikers. The are on
the same field at the same time with
responsibility for the same outcome.
▸ In tech there are multiple teams on the
same field with multiple orthogonal
outcomes.
▸ Which scenario makes more sense?
▸ Note that is not the same as, “Which is
easier, more convenient or
comfortable?” - That is what PTO is for.

ANY ORG THAT DESIGNS A SYSTEM
WILL PRODUCE A DESIGN WHOSE
STRUCTURE IS A COPY OF THE ORGS
COMMUNICATION STRUCTURE.
Conways Law

ENGINEERS HAVE PERSONALITIES
▸ Makers - engineers with a
penchant for creating features for
the system.

the system.
▸ Fixers - engineers with a penchant
for ﬁnding ﬂaws in the system and
correcting them.

the system.
▸ Fixers - engineers with a penchant
for ﬁnding ﬂaws in the system and
correcting them.
▸ Operators - engineers concerned
with the availability, stability and
performance of the system.

▸ Makers - engineers with a penchant
for creating features for the system.
▸ Fixers - engineers with a penchant for
ﬁnding ﬂaws in the system and
correcting them.
▸ Operators - engineers concerned with
the availability, stability and
performance of the system.
▸ Penetrators - engineers concerned
with penetrating the system and
plugging its holes.

TO BE EFFECTIVE A TEAM MUST
HAVE THE PERSONALITIES, ACCESS
AND MOTIVATION TO DELIVER.
Know-it-all

DOING IT WRONG
▸ DevOps Engineers
▸ Tool driven implementations
▸ Scrum is Agile … SRE is
DevOps
▸ Failing at knowing everything,
hoarding knowledge.
▸ DevOps Department
▸ Teams separated by skill set.
▸ Philosophy in practice
▸ Values driven implementations
▸ Assessing desired outcomes and
building a practice based on
organizational culture.
▸ Building the need to learn into
everything you do
▸ Culture of share responsibility.
▸ Teams separated by system
boundaries.
DEVOPS

Longhorn PHP - Stop Doing It Wrong

Longhorn PHP - Stop Doing It Wrong

More Related Content

Similar to Longhorn PHP - Stop Doing It Wrong (20)

More from Boyd Hemphill (20)

Recently uploaded (20)

Longhorn PHP - Stop Doing It Wrong