Fixing security by fixing software development

Fixing Security by Fixing Software Development
Using Continuous Deployment
Nick Galbreath, VP Eng IPONWEB, http://www.iponweb.com/
@ngalbreath http://www.client9.com/
May 14-15, San Francisco, California

Follow Along
Latest version posted online:

http://slidesha.re/144OIv3

http://www.client9.com/

Nick Galbreath (@ngalbreath)
! Spoken at Black Hat, DEFCON, OWASP
! Author of libinjection: advanced SQLi
detection used in > 2 WAFs, 1 Honeypot
! Book on cryptography
! but really...
! Engineering Management and Software
Development for high growth startups.
! Personal site http://www.client9.com/

IPONWEB
! customized online advertising
infrastructure and exchanges
! engineering oﬃces in Moscow, with
business oﬃces in London, New York and
Tokyo.

Original Abstract
Fixing Security by Fixing Software Development Using Continuous
Deployment
Do you have an eﬀective release cycle? Is your process long and archaic? Long
release cycle are typically based on assumptions we haven't seen since the
1980s and require very mature organizations to implement successfully. They
can also disenfranchise developers from caring or even knowing about security
or operational issues. Attend this session to learn more about an alternative
approach to managing deployments through Continuous Deployment, otherwise
known as Continuous Delivery. Find out how small, but frequent changes to the
production environment can transform an organization’s development process
to truly integrate security. Learn how to get started with continuous deployment
and what tools and process are needed to make implementation within your
organization a (security) success.

Well that's a bold statement...
"Fixing Security by Fixing Development
Using Continuous Deployment"

and here's another
For web applications, our release-based
software development lifecycle is still
based on a pre-Internet model and is
harmful to organizations and 
particularly harmful for security.

What needs ﬁxing?
! SQLi dropped from #8 to #14 in the latest
White Hat "The State of Web Security"
report. Good news, right?
! This means SQLi is only 7% of websites.
That's 1 in 15. And this is #14
vulnerability!
! And time to ﬁx was on average 196 days.  
That's embarrassing.
Veracode claims 32% of incoming web applications have SQLi 
https://info.veracode.com/state-of-software-security-report-volume5.html
https://reg.whitehatsec.com/WPstats0513

Even worse...
! Number 1 driver to ﬁx security problems... 
compliance.
! Number 1 reason to not ﬁx security is... 
compliance.
! Not..
! keeping our employees and customers safe
! protecting corporate interests.
! improving quality
! being good at what we do.

Security Products #1 .. in security bugs
VeraCode: State of
Software Security, V4
December 2011

Let's Just Give Up
! “You could spend all your resources chasing such
things as this,” William Ribich, the former president of
Technology Solutions Group [ QintelIQ ], said in an
interview in January. Ribich, who retired in November 2009,
shortly after the discovery of a major data theft, said he
needed to balance the uncertain risk that the hackers could
use what they stole against a growing shopping list of
security products and consulting fees.
! "You ﬁnally have to reach a point where you say 
’let’s move on,’” he said.
http://www.bloomberg.com/news/2013-05-01/china-cyberspies-outwit-u-s-stealing-military-secrets.html

I would call that broken
! But preventing SQLi isn't a technically
hard problem.
! And most security patches are very small.
! How did we get here?

Software Product Model
! Code ﬂows between functional groups
! Product Managers spec code
! Engineers write code
! QA engineers tests code
! Release engineers package code
! Operations runs code
! ... and Security does something too.

High Distribution Cost
! The Software Product Model is designed
for software where the cost of distribution
is high. "High" might be ﬁnancial, risk,
time, resources, customer annoyance.
! Retail, physical product, CD/DVD
! Embedded of Exotic Hardware
! Safety, Medical or Defense Systems
! Operating Systems (desk or phone)
! Homework (1-time deploy)

Web Applications Year 2000
! Mostly followed Software Product Model
since that's all we knew.
! High barrier to entry
! Specialized Hardware, Software and
People needed to get started.
! Lots of engineering needed to keep things
running.
! (side note: CERT/CVE started in 1999)

True Story #1

! "Can't push out the spelling error ﬁx – it's
too risky"
! "That code as already been through QA,
it's locked down."
! "Product has to prioritize that change, else
we aren't touching it."

True Story #2
! We'll do an iteration, where we try to ﬁx as
many things as possible.
! This won't be a scheduled iteration, it will
be done because things are so bad
! So the spelling error will get ﬁxed...  
uhh, who knows when.

Web Applications 2013

! Almost no barrier to entry
! Commodity hardware
! Programming not that hard
! Scaling problems can  
be mostly outsourced 
(mostly)

Cost of Distribution 2013
! Frequently no compile step 
or it's very fast.
! Moving to production a few kilobytes or
megabytes of code over 1Gbps, 10Gbps
link.
! In other words... free

Failure is very diﬀerent however
! Most web applications are data-driven.
! Frequently have social features, APIs,
user-generated content.
! Failures might be due to algorithmic
problems... but...
! Most likely to due to user input, bad data
in database or operational load.
! this means data in past can cause
problems in the future.

Releases and Problems
! When a web-release goes out, and has
problems....
! Next week is spent tracking down who
changed what, where.
! Re-QA
! Re-Push
! meanwhile new code is piling up.

When SPM meets Web Apps
! A long time between code being written
and code being released.
! Might be weeks or months
! Feedback loop between code-in-dev and
code-in-production is broken
! When security or bug reports come in, the
author is likely on a diﬀerent project.

Hypothesis
! It is impossible to simulate the production
environment in development, either due to
operational diﬀerences or data diﬀerences.
! No amount of QA or Security Testing can
prove you don't have bugs, vulnerabilities,
or cause severe operational problems.
! You have bugs and vulnerabilities, 
right now, in your application.

Impedance Mismatch
! Easy to write code, +
! Long release cycles +
! Security as an end-of-line or out of band
process ==
! no one cares

So the Answer is...
! Going slower? I'm sure your boss will love
that suggestion
! More steps and process? In other words,
slower.
! Asking for more people? Sure but good
luck hiring them. Doesn't scale.
! Asking for more products? Since the
others have worked so well.

Continuous Deployment

! Also known as Continuous Delivery.
! A System of Software Production
Characterized by Numerous Small
Changes the Production Environment,
initiated by the author of the change.

You change it, you push it to prod.

"Writing Software"
! Software Developers think their job is
writing software.
! And so, they love to make things perfect
before anyone else sees it.
! Impolite: "data hiding"
! code is hiding on developer's computer
! or on some branch
! in other words invisible until it's ready.

Actually
! The software engineer's job is actually
writing running software, that works well.
! This idea is so alien, that companies have
to remind the engineers of this.

Rackspace Haiku
writing code is hard 
if you cannot deploy
it does not matter
@paulvx from DevOpsDays Austin 2013

Facebook
"Move Fast and  
Break Things....
Except "Push" (deployment system)
via http://mitadmissions.org/blogs/entry/
move_fast_and_break_things

Today's goal
! but for today the goal is getting the
developer to care about their code 
in production.
! If you don't have that, I don't think you can
really solve security problems.

How does this work?

! Really? Developers push their own code
out?
! How is this not a disaster.
! How is this not a security disaster?

The Deploy Button
! What is you had a button that said 

"DEPLOY"
! That pushed to production, whatever is
current in your source control system.
! And took about a minute
! The change and who pressed the button is
logged, but that's it.

Part 1: Fear
! No one is going to push it ;-)
! Meanwhile code is piling up

Real example: A new hire I had at Etsy was afraid  
of deploying an HTML change that they made.
"But I don't want to break the site!"

Part 2: First Push
! Someone brave will press the button
! And very likely the site will explode, and a
rollback will need to be done.
! They'll know since someone else will have
told them.

Part 3: With Graphs
! Let's get all those operational graphs out
in the open. And put them right next to
the button.

Part 4: Push #2
! Repush
! Site might still explode
! But the developer is aware and can
rollback.

Take 5: Isolation
! Hmmm, the developer notices that in the
change set, a million things are going out.
! Maybe just pushing out a smaller change
will help isolate the issue.

Take 6: Success!
! Yes, the developer just pushed out some
code and made the site better.
! The secret about continuous deployment
is small changes that can be easily
understood.

Take 7: Dark Pushes
! Now we got some bugs ﬁxed, let's push a
feature.
! First let's push out all the supporting ﬁles.
Since they aren't being called, they do
nothing and are safe to push out.
! Now everyone can see them

Take 8: Getting the feature live
! Instead of "all at once", we slowly ramp up
a feature.
! if (user_id % 20 == 0): 
do new feature;
! we change change the percentage easily
with another code push.
! or turn it down. Much nicer change log.
! While the site didn't explode, it's hard to
see if the feature is being used or not.

Take 9: Application Level Graphs
! Allow developers to instrument their code
so they can see what is happening in
production.
! Enter StatsD and other UDP-based tools
! Enter centralized logging and in-
application method to make it easy to log
problems.

Take 10: Communication
! So far good for one developer.
! To scale up, you'll need a system to allow
developers.
! IRC-like tools work well (e.g. "the push
channel") – skype, jabber, hangouts, etc

Along the way
! Expose production logs to developers
! Add in a staging-step where the code
goes to faction of the cluster, so
developers can test with real traﬃc
! Try to make development closer to prod.
! Make "smoke tests" to catch basic errors
! Add syntax checkers to eliminate obvious
issues.
! Use static analysis to ﬁnd bugs

Mistakes will happen
! Do postmortem analysis
! Everyone thought they were doing the
right thing at the time.
! "How can the environment be changed to
prevent this" and build tools to enforce it.
! (Rarely can you truly change people)

Sounds good,  
but what about...

That guy who pushes at 3am
! Courtesy and convention will converge
very quickly when the site goes down at
3am and the developer starts getting
calls ;-)
! Of hours pushes of course can happen,
when they notify operations.

What About Code Reviews?
! Yes, please do them.
! Nothing here prevents code reviews.
! In fact code reviews are easier since
! they are small
! they are in mainline not some branch

What about Security Reviews
! Please do them.
! Nothing here eliminates architectural
planning or review.
! This actually doesn't change the SDLC
very much.

What about Agile Methods
! (everyone seems to have a diﬀerent idea of
what Agile is but..)
! Agile methodologies typically work to
improve the business spec / development
cycle. (are you building what the customer
wants)
! But doesn't address code deployment.
! They are complimentary practices.

What about Customer Service?
! "Don't they freak out with all the
changes?"
! Remember: deployment != feature release
! Most deployments do very little from the
customer point of view
! Feature releases (frequently controlled by
ramp-ups or ﬂags) always needs to be
coordinated with product and customer
service.

What about Compliance? PCI?
! Let me tell you about compliance...
! mechanism not policy

Obvious Beneﬁt to Security
! Security patches can go out quickly
! You know this since they are now just part
of a normal development cycle.

More Importantly
! That Engineer who previously didn’t push
code is now sensitized that their code has
consequences and are responsive and
empowered to fix it.
! It’s amazing how interested engineers
become in security when you find
problems with their code when they are
able to fix quickly themselves.

Hack The Stack
! A side eﬀect of this you now have tools to
repurpose for security and 
monitoring of production
! Note that most changes are not security
problems.

Logging
! Due to allow developers to see application
logging, it's now very easy to instrument
the application to log security events.
! Or add logs to times when you are under
attack.

Graphing
! Make dashboards of
! SQLi and XSS attacks
! Every type of log-in failure
! Core Dumps
! Database Syntax Errors

Static Analysis

! You now have a place to insert them.
! Work with QA group to add more code
quality tests.

Post-Commit Checks
! Alert on when sensitive areas of the code
are changed (auth)
! Alert on crypto usage (why is developer
using MD5.. hmmm)
! Alert your programming languages
"dangerous functions"
This allows you to engage the developer at the start of the cycle.

Faster is Better
! You could do most of this in a normal
release-cycle software lifecycle.
! The difference is you are finding problems
at the start instead of 10m before the
launch and telling everyone to stop.
! The feedback loop works.

New Roles, Less Silos
! Developers: works with operations
! QA: works on building systems for testing,
to empower others to write better tests
! Release Engineering: tools to enable code
to ﬂow faster
! Security: in-house consultancy, secure-by-
default architecture, monitoring

Goal: 50% reduction in deploy times
times

! Whatever your state of deployment is, no
matter how many people are involved, no
matter how long it currently takes, make a
goal of cutting it in half.
! This is an easy sell to management just on
cost basis.
! Everything else ﬂows from this.

Mechanism not Policy
! Strive for the fastest deployment mechanism for
possible
! But you deﬁne the "continuous" in Continuous
Deployment
! Yes, Etsy was 60+ deploys per day, with each having
multiple authors.
! Current gig? we have rules of no more than 3 per week
since our customer have asked for that, and only
deployed at "low-tide"

In other contexts: Operations
! How fast can you deploy OS changes to
you production environment?

In other contexts: IT
! How fast can you deploy desktop
changes?

In other contexts: software product
! here "production" might be getting code
into the main branch and running
automated build / test.
! It's the ﬂow of code: little changes vs big.

In other contexts: silicon
! Continuous deployment already done for
silicon! wut?
! Only small changes, with tests are allowed
to be committed!
! Big changes are rejected. 
Learned the hard way that big changes are
completely unmanageable

Your Attackers Use

Start with 50% Reduction
Build the Deploy Button
Nick Galbreath ★ @ngalbreath ★ nickg@client9.com ★ http://www.client9.com/
http://slidesha.re/144OIv3

© 2013 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it
should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO
WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
Fixing Security by Fixing Software Development Using Continuous Deployment
Nick Galbreath ★ @ngalbreath ★ http://www.client9.com ★ nickg@client9.com

Fixing security by fixing software development

More Related Content

What's hot (19)

Viewers also liked (14)

Similar to Fixing security by fixing software development (20)

More from Nick Galbreath (13)

Fixing security by fixing software development