Accelerating Your Digital Agenda with Continuous Testing by Diego Lo Giudice

© 2019 FORRESTER. REPRODUCTION PROHIBITED.
Accelerate Your Digital Agenda with
Continuous Testing
Diego Lo Giudice, VP & Principal Analyst serving AD&D
April 2019

2© 2019 FORRESTER. REPRODUCTION PROHIBITED.
We need to test more,
better and faster..
..continuously

3© 2019 Forrester Research, Inc. Reproduction Prohibited
Why?

http://assets.bwbx.io/images/i0dwq8cFhDmE/v1/-1x-1.jpg
Too much speed?
Poor quality?
Or both?

Poor Quality Software ?

4 Seconds

53%
Base: 232 tech professionals at firms that have implemented or are implementing Agile.
Source: Forrester’s Q3 2017 Global Agile Software Application Development Online Survey
Expect Agile to
improve technical
quality

63%
Expect Agile to
improve functional
quality

76%
Expect Agile to
improve customer
experience

better and faster.
Speed Can’t Come
Without Quality !
Quality@speed

What are the
challenges?

Digital
Transformation
Is The Goal For
Every Industry
Base: 354 business and technology leaders
Source: Forrester’s Q1 2018 Digital Process Automation Survey

But,most digital transformation initiatives are struggling
Source: “The Sorry State Of Digital Transformation In 2018” Forrester report.
Base: 1,559 business and technology decision makers

Many Enterprise
Developers
Aspire To Do
DevOps
Yes, our group
actively
embraces
DevOps
21%
Yes, our group actively
embraces DevOps in
conjunction with Agile
32%
Some of our
processes or tools
resemble those used
in DevOps, but we
have not officially
adopted DevOps
25%
No, but adoption
of DevOps is an
internal priority
5%
No, and we
have no plans to
adopt DevOps
11%
Don't
know
6%
Would you characterize your group's current
processes and structure as resembling DevOps?
Base 1358 Enterprise Developers and Development Managers
Source: Forrester Business Technographics Developer Survey, 2018
But only these really
get it!

But Software Delivery Speed Seems Sluggish
Base: 3,228 Global Developers,
Source: Forrester Analytics Business Technographics Global Developer Survey, 2018.
9%
18%
26%
20%
More than once per month
Once per month
Once per quarter
Twice per year
How frequently do you release applications ?

Development Velocity Is Stuck – App Releases
Base (2018): 3,228 Developers
Base: 546-719 Developers who work for a software company, for internal IT, or in technology services
Source: Forrester Business Technographics Developer Survey, 2014-2018
0%
5%
10%
15%
20%
25%
30%
We average one
release a year (or
less frequently)
We average two
releases a year
We average one
release per quarter
We average one
release a month
We average one
release a week
We average a release
a day
We release many
times per day
"How often does your team or teams release applications?"
2018 2017 2016 2015 2014
One release a
month

Testing Is Part Of The Problem
20%
22%
25%
26%
26%
28%
30%
31%
Lack of automation tools and/or infrastructure
Scaling Agile at the enterprise level
Connecting Agile with existing product portfolio management
(PPM)
Lack of dedicated cross functional teams
Lack of Agile testing skills
Lack of Agile skills in downstream or DevOps
Lack of Agile executive leadership
Lack of Agile skills in project management practices
Which of the following are barriers to your organization’s Agile
adoption? (Check all that apply) (not all answers shown)
a

Developers time on testing is closer to 1 hour a day

Test automation plateaus too, and manual testing
resists to go away

As Continuous Delivery Matures Performance Testing
Is The Next Blocker
Performance done at the end is complex
and costly
Centralized performance testing isn’t scalable
Completely disrupts continuous delivery

How can we address
these challenges?

From Digital Business Excellence To SW Delivery
Execution Excellence….Your new mantra…!
Business Must
Master Digital
To Stay In
Business
Software
Powers The
Digital
Experience
Software Execution
Excellence Equals
Business Execution
Excellence

Modernize Testing With Continuous Testing

Sprint definition of "done" includes
your sprint testing goals (Done
done, test data strategy,
automation, cntrl. Gates…)
Define Test Scenarios using a
DSL (e.g: Gherkin)
Include Performance Testing in
Acceptance Criteria
Feature toggles, AB Multivariate…
Prioritize bugs on backlog
PO, Testers and Developers

End to End Testing
Integration Testing
Functional
Testing
Non-Functional
Testing
Unit Testing
Make Sure Its Clear Who Needs To Test What….
Only Developers
Buz mostly manual, Mostly Technical Testers,
Dev Testers (API testing, automation).
Technical Testers (Specialized),
Dev Testers (slowly catching up..shift left)
Mostly Technical Testers, Dev Testers (SVT
tools).
Buz, Mostly Technical Testers

Expert Agile Shops Are Better At It...
• 69% Have a tester embedded in
delivery team
• 36% Practice test-driven
development (TDD)
• 20% Practice Behavior Driven
Development (BDD)
• 58% Report improved functional
quality
• 50% Report improved technical
quality
• 57% Have a tester embedded in
delivery team
• 15% Practice test-driven
development (TDD)
• 12% Practice Behaviour Driven-
Development (BDD)
• 37% Report improved functional
quality
• 33% Report improved technical
quality
Agile Experts Agile Neophytes
Base: 130 tech professionals who have achieved “Expert” status by achieving significant scale and quality of Agile adoption. 102 tech professionals who have achieved
“Neophyte” status by achieving limited scale and quality of Agile adoption.

Design automation of functional
tests (and test data).
Build automation of functional
tests (UI, API).
Build automation for non
functional (SVT, SL PT, Security)
Scriptless, scripting, coding
Exploratory testing (manual)
Technical Testers, Dev-testers,
Business testers

Source: See The Forrester report “Shift Performance Testing Left To Streamline App Delivery”.

Shift Left Performance Testing (SLPT)
Developers
profile code,
smoke test,
leverage
baselines at
unit/
component
level
Product
Owners and
Developers
become
stewards of
Performance
(With Testers)
Overtime E2E
Performance
Testing
Wanes As
SLPT
Becomes The
Norm.
A new set of
lightweight
agile SLPT
tools provide
developers
what they
need to get
the job done

United Airlines experienced
an 80 % Performance
Issues drop in production
with SLPT

Version Control test assets
with source code
Integrate in CI/CD to
automate all test execution
(unit, functional, non-
functional)

Automated integration
testing (Service
Virtualization)
Automated governance for
release automation or
feature deployment

idea Customer
value
control points
versioned
source repository
Code
Environ
Configs
Test assets
Continuous
Integration & Testing
artifact
repository
“built”
artifacts
Backlog
Release
Decision
Release
Automation
vendors
open
source
Developers
EA,
Testers
Ops,
Security
EA,
Developers,
Ops,
QA,
Security
DevOps Automates Testing Governance
CONTROL POINTS IN THE CONTINUOUS DELIVERY PIPELINE

Production monitoring
feeds your testing
Machine Learning predicts
what to test next
Integrated AB testing in
SDLC helps improve CX

A/B Testing improves customer experience

How the use of AI
optimized and
reduced overall
testing time from
years to days for
DELL EMC
26
World Quality Report | 30th October 2018 © 2018 Capgemini. All rights reserved.
Context at DellEMC Servers
Server Configuration Elements
Chassis
Processor
Memory DIMM
Memory Configuration
Hard Disk Drive (HDD)
Non-Volatile Memory (NVM)
Embedded Systems Management
Power Management BIOS
Power Supply
Bezel
Network Daughter Card
RAID Controller
Network Interface Card (NIC)
Host Bus Adapter (HBA)
Additional PCIe Cards
Cooling
465 Trillion
Test Configurations!!
AI Technologies used:
• Machine learning
• Rule-based
system

We need to test faster,
We need to test better,
We need to test smarter!

The Push From Within: A Journey
of Transformation
Claude Jones, Sr. Director of Engineering | Walmart Labs

Accelerating Your Digital Agenda with Continuous Testing by Diego Lo Giudice

More Related Content

What's hot (20)

Similar to Accelerating Your Digital Agenda with Continuous Testing by Diego Lo Giudice (20)

More from Sauce Labs (20)

Recently uploaded (20)

Accelerating Your Digital Agenda with Continuous Testing by Diego Lo Giudice