Building an AppSec Program From the Ground Up: An Honest Retrospective

Building an AppSec
Program
From the Ground Up:
An Honest Retrospective
John Melton
@_jtmelton

whoami
- Past: Dev/Security engineering in defense, finance, technology companies
- Current: Leading AppSec at technology company (Oracle NetSuite)
- Side: OWASP*, AppSensor, Manna
- Opinions are my own, not my employers
@_jtmelton

mumble, mumble, mumble …
Jim is great
Neil is great, Jer is great, Ron is great
Thanks to all the organizers and
sponsors
@_jtmelton

Agenda:
An honest retrospective of
the last ~2 years building
an appsec program from
scratch: the good, the bad,
and the ugly ...

With an eye towards
immediate applicability of
lessons learned ...

And wrapping up with my
observations and beliefs
about what I would do if I
had it to do all over again

Gather round y’all, it’s ...
@_jtmelton
story time

Context
Environment
15yr old (acquired)
startup
Very smart engineers
No “security” people
CI/CD µServices /
DevOps
Very culture-protective
~350 people
Management
20yr old (acquirer)
startup
Very smart engineers
Full security team
Monolith, traditional ops
Broad, varied culture
~5,500 people
Me
The new guy
meh
“The” security team
I’ve done both
Need to get stuff done
1 person
@_jtmelton

Tasks
ToDo List (Aiming for 2yr
timeline)
● Learn environment
● Embed into team / culture
● Translate
● Secure everything
● Prep for compliance
● Share / leverage successes upstream
Resources
● Tooling
● Training material
● “Policies”
@_jtmelton

● Be humble
@_jtmelton
Lessons
Learned

● The business is still running
● They wouldn’t be paying you
if all the work was already
done
● Take a breath
@_jtmelton
Lessons
Learned

Proposed deliverables
Introductions
● Get to know people
● Get to know tech/processes
Update Training
● I’ve written training before
● We have a pretty good base to start from
Dependency Analysis ● DependencyCheck is awesome
@_jtmelton

Results
Introductions (100%)
● People are great
● Tech is great, Processes are good /
maturing
Update Training
(30%)
● Updated team HR training
● Made a dent in tech training
Dependency Analysis ● DependencyCheck is awesome
@_jtmelton

● Find key stakeholders
● Learn existing processes
● Learn the environment /
tools
@_jtmelton
Lessons
Learned

Static Analysis
● We already have a tool
● I built one - setting one up should be easy!
Dynamic Analysis
● ZAP is awesome
● It’s just a script right?
Dependency Analysis
● DependencyCheck is awesome
● Need to get a handle on these 3rd party libs
Update Training
@_jtmelton

Results
Static Analysis (80%)
● A factory is hard work (especially on * apps)
● Oh yeah, we should vet and explain results
Dynamic Analysis (0%)
● ZAP is still awesome
● No time
Dependency Analysis
(100%)
● DependencyCheck is still awesome
● There’s a lot of low-hanging fruit here
Update Training (-10%)
● Way more stakeholders than I thought
● More work to do than initially planned
@_jtmelton

● Start to formulate questions
you want to answer (metrics)
(fail)
● Start an application
inventory (hard fail!)
● Tools _may_ not be the right
place to start (fail)
● CI/CD is security’s friend
● Get a handle on 3rd party
libraries - it’s low hanging
(often rotten) fruit
@_jtmelton
Lessons
Learned

Credential Storage
● Need to clean this up
● Like Vault as a base
Dynamic Analysis
● Work with QA / Selenium
● Need help from project leader
Champions
● Extend relationship with interested folks
● Offer extra training and security info
Update SDLC
● The existing SDLC is restrictive
● Need to separate policy from standard
@_jtmelton

Results
Credential Storage
(100%)
● Good planning, coordination
● Vault operationally works in our env
Dynamic Analysis (100%)
● ZAP/Selenium/QA automation are powerful
● Project leaders want to help you succeed
Champions (50%)
● People are interested in security
● No formal program, small sharing network
Update SDLC (20%)
● This is complex, and needs high-level
approval
● SDLC touches everything
@_jtmelton

● Pick operationally
compatible tools for your
env (work with eng team for
input)
● Leverage open source where
possible (also contribute
back code, docs, stories,
etc.)
● Reach out to people on
github, twitter, phone, etc. -
they will actually help (fail,
see BSIMM)
● Share security knowledge
freely (fail)
● Policies matter *
@_jtmelton
Lessons
Learned

Metrics
● Need to measure progress on desired goals
● Should be easy to collect and drive change
Threat Modeling
● We should fix arch/design issues too
● Aiming to start simply
Track Attack Surface
● Lots of micro-services
● How are they changing?
Update SDLC
● Work is now scoped
● Know stakeholders for approval
@_jtmelton

Results
Metrics (20%)
● There are lots of bad metrics
● Lots of stakeholders with significant input
Threat Modeling (100%)
● Starting small was key
● A picture and a list of threats
Track Attack Surface (75%)
● Surprisingly simple to support
● Really helpful over time and good signal
Update SDLC (100%)
● This was a lot of work
● A solid, clear policy is really helpful
@_jtmelton

● Measuring what you are
doing is critical.
Communicating that
information well is a
challenge.
● Security talks about the
“what’s wrong” all the time,
but rarely about proactive
controls (fail)
● People like threat modeling
(be the attacker)
● Threat modeling gives
developers power to
communicate and address
what keeps them up at night
(fail)@_jtmelton
Lessons
Learned

Progress Report
ToDo List
● Learn environment - Done
● Embed into team / culture - Done, solid champions core team
● Translate - Mostly done
● Secure everything - Basics are in place, still plenty of room for
improvement
● Prep for compliance - A little bit done
● Share / leverage successes upstream - Very little upstreamed
@_jtmelton

App Deployment Tool
● Needs a refresh
● Let’s upgrade security at the same time
Containers
● Chance to affect greenfield deployment
● Aim for reasonable default security bump
Runtime Intelligence
● Get runtime feedback about security
● Empower / delegate developers to monitor
Operational Tasks
● Integrate with other teams
● Produce / consume useful data
@_jtmelton

Results
App Deployment Tool (90%)
● Awesome tooling people are awesome
● Platform defaults matter
Containers (75%)
● Set strong, safe defaults for great bump
● Security features not always well-tested
Runtime Intelligence
(10%)
● Introduced the idea and tools
● Needs a “light bulb” moment
Operational Tasks (100%)
● This data is really valuable
● We don’t think about this enough
@_jtmelton

● Vault useful in many
contexts, operational ability
is important*
● Reminder: availability is part
of CIA triad - it is a security
issue
● We need more runtime intel
in our applications (fail)
● AppSec <--> OpSec is an
area ripe for exploration and
exploitation (fail)
● Went to BSIMM (world
changing moment) (fail)
@_jtmelton
Lessons
Learned

@_jtmelton
* http://blog.bronto.com/engineering/tooling-microservices-for-scale-and-
access/
http://blog.bronto.com/engineering/microservices-deployment-security-
flexibility/

Core Security
Libraries
● Stop squashing individual bugs
● Need consistent mechanisms for devs
Source Code Attestation
● Verify steps from commit -> ops
● Auditability
Training Refresh
● Function-specific training
● Continue simplification
CI Upgrade
● Tooling upgrade & apis
● Versioned CI configuration
@_jtmelton

Results
Core Security Library
(50%)
● Built/extended core systems
● Partial custom rules to match
Src Code Attestation (100%)
● Requires integration work
● Audit log is powerful
Training Refresh (25%) ● Customization benefits from modularity
CI Upgrade (75%)
● Config versioning is powerful
● Declarative CI is powerful
@_jtmelton

● ** Killing bug classes is the
useful engineering work
(Fail)
● (git/web) Hooks are great
integration points (Fail)
@_jtmelton
Lessons
Learned

Tool Additions
● Need broader coverage
● Tool vendors always behind
Immutable Infra
● Joint effort with eng
● Increase stability & security
Compliance
● Has to happen
● Supports business
Fast/Slow Checks
● Faster feedback on high confidence issues
● Block certain classes of issues
@_jtmelton

Results
Tool Additions
(100%)
● Lots of tools available these days
● Single purpose tools are nice
Immutable Infra (80%)
● Big eng win
● Big security win
Compliance (100%)
● Sanity check
● Low bar
Fast/Slow Checks
(100%)
● Further left
● Lots of low-hanging fruit here
@_jtmelton

● Completely fixing and
forever preventing an issue
(even a small one) is a win -
combine these to get
momentum
● Create minimum assertions
and raise the bar
● Config mgmt / Infra mgmt are
powerful
● The faster devs see the issue
relative to coding it, the
faster (and better!) the fix
(Fail)
@_jtmelton
Lessons
Learned

Refresh SDLC
● Maturity step
● Chance to move left / increase visibility
App Portfolio
● Do a better job collecting metadata / metrics
● Single, consistent view
Compliance ● More, more, more
Fast/Slow Checks
@_jtmelton

Refresh SDLC (100%)
● Tailor to different stakeholders
● Talk about privacy alongside security
App Portfolio (20%)
● Complex area
● Missing tool support
Compliance (100%) ● More, more, more
Fast/Slow Checks
@_jtmelton

● Privacy is something
everybody cares about (Fail)
● App Portfolio is ripe for a
solution, many people are
struggling in that area
● Hard to secure what you
don’t know about
@_jtmelton
Lessons
Learned

Progress Report
ToDo List
● Learn environment - Done
● Embed into team / culture - Done, solid champions core team
● Translate - Done
● Secure everything – We’re further, but always room for improvement
● Prep for compliance – Oh boy … did we ever
● Share / leverage successes upstream – Good progress
@_jtmelton

TODOs - People
● Work with the best people you
can
● Do small, focused, context-
sensitive training
● Connect with tribal knowledge
owners
● Build a real champions program
● Say “no” rarely
● Know your place and stay
humble - security is not the only
business concern
● Connect with others doing your
job in security at cons, social
media, etc. Ask them questions.
● Talk about security in regular life
(e.g. https://securityplanner.org/)
● Talk about security often (chat,
email, presentations, etc.)
● Talk about privacy
● Talk issues at the highest level
possible - exec buy-in is critical
@_jtmelton

TODOs - Process
● Have office hours, chatops,
email list - be available
● Never start with “no”. Default to
“how can we get to yes”?
● Use the champions program
● Actively build relationships with
other teams
● Collect useful data to improve
and build a data-driven process
● Live and breathe threat modeling
● Inject into the standard SDLC
(not as a blocker though)
● Reqs > Arch > Design > Code
● Use consistent terminology
(words matter)
● Develop method for ranking
apps
● Develop method for ranking
vulns
● Meet customers where they are
(chat, email, wiki, bugtracker,
etc.)@_jtmelton

TODOs - Technology
● Don’t have tech envy
● Isolate security services (e.g.
encryption as a service)
● Exploit CI (fast/slow lanes)
● Squash bug classes, not bugs
(bug of the month/quarter, top
“1”)
● Support / amplify good tech from
devs (containers, cloud, etc.)
● Build a solid app inventory
● Focus on and invest in self-
service
● Spend time on the “big” things
(cloud, 2/MFA, IAM, Authn/z,
crypto)
● Limit crypto primitives (e.g. nacl)
● Support primary tech stacks well
● Work with dev and ops to get
runtime info, and create a
feedback loop
● Build self-defending apps
(appsensor)
@_jtmelton

Some Homework
● “Starting Up Security” by Ryan McGeehan (https://medium.com/starting-up-
security)
● “Preventing Security Bugs Through Software Design” by Christoph Kern
(https://www.youtu.be/ccfEu-Jj0as)
● Measuring End-to-End Security Engineering by Garrett Held
(https://www.youtu.be/MLmQ4uSi4EU)
● “Software Security Metrics” (https://youtu.be/50vOxExpAOU) and “Effective
AppSec Metrics (https://youtu.be/dY8IuQ8rUd4), by Caroline Wong
● Starting a Metrics Program by Marcus Ranum
(https://youtu.be/yW7kSVwucSk)
● Enabling Product Security with Culture and Cloud by Astha Singhal and
Patrick Thomas (https://youtu.be/L1WaMzN4dhY)
@_jtmelton

This I Believe
You can’t do appsec effectively and
not understand code. You should be
able to read & write it. Same with
design & architecture.
@_jtmelton

This I Believe
Security teams don’t scale
effectively, even with significant
automation efforts. Build a
champions program, and move to
self-service.
@_jtmelton

This I Believe
We need more focus on detection
and response, not just prevention.
@_jtmelton

This I Believe
Certain solutions (executive
support/buy-in, 2FA, CI/CD
automation, threat modeling) move
the needle _much_ farther - focus on
those.
@_jtmelton

This I Believe
You cannot protect what you don’t
know about - build an app inventory.
@_jtmelton

Open Source
● We don’t all have huge teams
● Most/all of us use open source
docs, software
● Many of us build useful things
● Some of us can contribute at
work
● Many of us can contribute at
home
● You benefit the community
● You grow
● You build awareness of issues
● You help drive solutions
● You benefit personally and
professionally
@_jtmelton

This I Believe
We all have something useful to
contribute to the community.
@_jtmelton

This I Believe
@_jtmelton
● Docs - https://medium.com/starting-up-security
● Talks - https://www.youtube.com/watch?v=ccfEu-Jj0as
● Scripts - https://github.com/jgamblin/AWSScripts
● Code - https://github.com/jeremylong/DependencyCheck
● Organization - https://github.com/sbilly/awesome-security and
https://github.com/paragonie/awesome-appsec
● Work - https://github.com/nccgroup/Scout2 and
https://www.nccgroup.trust/us/our-research/understanding-and-hardening-
linux-containers/

● https://github.com/manna-
security
● Open Source (Apache 2)
● Alpha release (only 1 rule)
● Please contribute!
@_jtmelton
Manna
Simple static analysis &
automatic remediation

Building an AppSec Program From the Ground Up: An Honest Retrospective

Building an AppSec Program From the Ground Up: An Honest Retrospective

More Related Content

What's hot (20)

Similar to Building an AppSec Program From the Ground Up: An Honest Retrospective (20)

More from jtmelton (8)

Recently uploaded (20)

Building an AppSec Program From the Ground Up: An Honest Retrospective

Editor's Notes