Case Study: SunTrust’s Next Gen QA and Release Services Transformation Journey

Case Study: SunTrust’s Next Gen QA and
Release Services Transformation Journey
Beth Johnson
DevOps: Continuous Delivery
SunTrust
SVP, Release and Test Manager
DO4X131S
#CAWorld

2 © 2015 CA. ALL RIGHTS RESERVED.@CAWORLD #CAWORLD
© 2015 CA. All rights reserved. All trademarks referenced herein belong to their respective companies.
The content provided in this CA World 2015 presentation is intended for informational purposes only and does not form any type
of warranty. The information provided by a CA partner and/or CA customer has not been reviewed for accuracy by CA.
For Informational Purposes Only
Terms of this Presentation

Abstract
Sun Trust’s journey from challenge
identification, charter definition, execution
practices and key metrics and results in their
transformation of traditional QA and Release
functions into a more cohesive, collaborative
and “continuous” model.
Beth Johnson
SunTrust
Senior Vice President,
Release and Test
Manager

Agenda
HISTORICAL CHALLENGES
CHARTER: WHAT WE SET OUT TO DO
EXECUTION: APPROACH, CULTURE, ESSENTIAL TECHNOLOGIES
KEY METRICS & RESULTS TO DATE
WHAT'S NEXT
1
2
3
4
5

About SunTrust
 At SunTrust, success is defined by the impact
we make on the lives of our clients,
our communities and our teammates.
 As one of the nation’s largest and strongest financial services companies, Suntrust
offers a full suite of products and services designed to meet the needs of our
consumer, business, corporate and institutional clients. Lighting the Way to
Financial Well Being is our commitment to helping instill confidence in the
financial circumstances of everyone we interact with.
Sub-head

Historical Challenges
Complexity
LOTS of legacy systems and
platforms
LOTS of dependencies across
differenct systems
LOTS of manual effort required
to adequately test and release
LOTS of time required between
releases
Collaboration
Dev and QA teams not familiar
with each others jobs
App teams unaware of how
their apps/ervices impacted
other teams
Agile going well in some dev
teams, but not being scaled to
QA and Ops teams
Culture
Historical success rate with
“doing things differently”
Mix of GSI teams (Cognizant,
Infosys), consultants and W2
team members
Perception from LOBs as being
slow

Our Focus
SUNTRUST OBJECTIVES
AGILITY SCALABILITYCOST SIMPLICITY
ACCELERATE RENOVATE INNOVATE
Change Management
Tools Adoption
Cost Savings
Automation
Process to adopt Managed services
Shift Left
Domain
Pricing Model
Agile Model
PA
APPROACH to Next Gen QA Model

Transforming Testing Services
 Definition & Communication of a Managed Services Vision
 Transition Change Management from T&M (project based) to a managed
service
 Lunch & Learns and EXPO (internal SunTrust tech conference)
 Domain centric approach
 Simplify and streamline demand management
 Re-introduce Static Testing by partnering with BA team
 New approaches & testing technology (Service Virtualization, App
Dependency Analysis, Cross Browser testing)
Execution highlights over the last 12 months…

Impact to Date in Testing Services…
YTD savings of $6.5M with projected savings of $9.7M USD by November 2015
2 of 9 Can’t Fail Initiatives
implemented successfully
12 out of 25 SoW commitments
implemented and 10 In-Progress
DELIVERY
70% automatable automation and
64% Automated execution with a
cum. effort savings of 63,388 hrs.,
translating to a savings of $140K
AUTOMATION
Requirement Volatility Index (RVI) has
dropped from 82% to <50%
VALUE ADD
Significantly lower Defect Leakage
<1%, compared to SLA targets of 4%
LOW DEFECT LEAKAGE
82% of projects tested by Infosys were
successfully implemented with ZERO
production defects
IMPROVED QUALITY
126 new projects are in Managed
Service model post Oct
MANAGED SERVICE

Transforming Release Services
 Built a Performance Center of Excellence (PCoE)
 Formed a Data Management Office (DMO)
 Expand Service Virtualization (400+ services virtualized today) further left
in SDLC (functional test and early dev and integration test)
 Deployed Test Environment Application Management (TEAM) and
automated much more of the environment staging and setup tasks
Execution highlights over the last 12 months…

Impact to Date in Release Services…
Test Environment Application Management (TEAM)
Data Management Office (DMO)
Performance Center of Excellence (PCoE) Service Virtualization (SV)
 Fulfilled over 4412 data requests.
 Data Reservation process introduced
 DMO assumed additional responsibilities for
OLBM related mining/ conditioning
 4% of Data Requests addressed via
Automation Procedures
 85% more data requests fulfilled successfully
 Completed engineered & regression performance testing
for over 40 projects
 Enabled early performance engineering for ALL critical
projects
 Introduced new types of non functional testing such as
Endurance, Stress, Scalability, Fail-Over
 1500+ hours of production performance triage reduced
by finding issues before PROD
 Reduced the system / third party dependencies
in performance testing
 Over 400 app services are available for use
 Significant speed increase and better quality
with SV being use prior to QA in Dev
 Supported 34 test environment cycles
 Achieved faster cycle run time after reducing known defects &
automating manual environment preparation activities
 18% reduction in defects by automating environment
deployment

Leap Frogging Innovation and Testing Trends
• Defined the process, built the
SharePoint site for idea
gathering and tracking
• Formed the Innovation council
to review and decision ideas
• Participated in 36 hr. digital
challenge and provided about 56
ideas from the team
Innovation Tuesday
• Custom built tool for managing
MS workflow
• Inbuilt exhaustive reporting
• Project Early Warning or ALERT
system to automatically monitor
and report project status
Service Delivery
Platform and Alerts
• Analysis of Defects data of 4
major releases
• Utilized QA Analytics tool to
analyze Application Risk and
Defect patterns
• Predicted risk applications and
vendors for upcoming releases
Predictive Analytics
• Introduced Domain Advisors in
Testing practice to work with
BA’s
• Implemented Static Testing to
measure Requirement Quality
• Defined RQI and RVI to measure
and Influence upstream quality
Domain Approach to
Testing
• Virtualization Journey with CA
partnership
• EXPO to create awareness of MS
• Lunch n Learns to spread
message
Show and Tell
• Improved Daily and Weekly
Status reports
• Enhanced SLA/Metrics report
• Touch Screen dashboard for
Release and Test Reports (POC)
QA Command Center

Progress to Date vs. 5 Year Metrics…
In the last 5 years…
 Reduced the production defect leakage
at to < 1% at enterprise level
 Testing cost per project reduced by more
than 27%
Requirement coverage increased from
<70% to 98%
 Extended testing reach from manual
testing to SOA, IVR, Mobile, Mainframe,
Package based testing
 Extensive usage on advanced tools for
effective testing – CA SV, Test Harness, HP
Service Test, SSIS
 Proactive Proof of Concept (POC) on
latest tools - CrossBrowserTesting,
Sprinter, TOSCA, Turnkey
 6x increase in number of projects served
over the past 4 years

SunTrust Approach to Innovation…
Incremental but with a success at end of each step

What’s next?
2016 plans…
 Industrialization of Testing on Digital Platforms
 Develop comprehensive End to End methodology to support agile
initiatives across SunTrust
 Platform agnostic Automation solutions
 Explore Robotics and Artificial Intelligence in Testing
 Automation of the broader release process

Q & A

Recommended Sessions
SESSION # TITLE DATE/TIME
DO3X118S The why, where and how of Service Virtualization 11/19/2015 at 02:00 pm
DO3X116S
Lloyds Banking Group Leverages Service Virtualization to
Deliver Quality Technology Services
11/19/2015 at 03:00 pm
DO3X114S
Freeing the World from Slow: How Service Virtualization
and the Concept of S.P.E.E.D. Will Steer You Away from
Committing Continuous Delivery Suicide
11/19/2015 at 03:45 pm

Must See Demos
Service
Virtualization
DevOps
Theater 3
DevOps
Simulation
DevOps
Theater 3
SV
Innovations
DevOps
Theater 3
SV Use Cases
DevOps
Theater 3

Follow On Conversations At…
Smart Bar
DevOps
Theater 3 & 4
Tech Talks
DevOps
Theater 3 & 4

For More Information
To learn more, please visit:
http://cainc.to/Nv2VOe
CA World ’15

Case Study: SunTrust’s Next Gen QA and Release Services Transformation Journey

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to Case Study: SunTrust’s Next Gen QA and Release Services Transformation Journey (20)

More from CA Technologies (20)

Recently uploaded (20)

Case Study: SunTrust’s Next Gen QA and Release Services Transformation Journey