Scaling Automated Tests: Choosing an Appropriate Subset

DT4
DevOps & Testing
6/8/2017 11:30:00 AM
DT4 Scaling Automated Tests: Choosing
an Appropriate Subset
Presented by:
Manoj Pahuja
GoDaddy
Brought to you by:
350 Corporate Way, Suite 400, Orange Park, FL 32073
888-‐268-‐8770 ·∙ 904-‐278-‐0524 - info@techwell.com - https://www.techwell.com/

Manoj Pahuja
GoDaddy
Test architect at GoDaddy Manoj Pahuja has several years' experience in test
planning, strategy, and automation for web, desktop, and mobile applications. He
has worked on building automation frameworks, created and managed test
management software, and is an active member of the open source community.
Manoj's experience writing automation in VBScript, Java, Ruby, and now NodeJS
has helped him deal with the challenges of automation. A strong believer in
learning while sharing, he co-organizes meetups in the San Francisco Bay Area.

Copyright© 2017 GoDaddy Inc. All Rights Reserved.
Copyright© 2015 GoDaddy Operating Company, LLC. All rights reserved.
Scaling Automated Tests:
Choosing an Appropriate
Subset
June 2017
Daniel Clayton and Manoj Pahuja

Agenda
• Define Continuous Integration and Continuous
Deployment (CI/CD).
• How GoCentral adopted CI/CD.
• Identifying What to test.
• Life cycle of a Test.

Define CI/CD
3
to a developer… automate all things!
to a tester… smaller deployments, reduce risk!
to a manager… release quickly, deliver value!
to a product owner… slice features small enough to prove hypothesis!
Release Quickly with Confidence
What does CI/CD mean…
Continuous Integration – frequently testing and integrating your changes with others
Continuous Deployment/Delivery – frequently delivering your changes to your customers

CI/CD with GoCentral
4
v6
Manual Testing
Manual Deployment
Stress/Uncertainty
0.029 mph
Deployments to Production
v7
Agile
Automation
High Confidence
6.3 mph
368
979
vNEXT
Microservices
Elastic/Resilient
Abstracted Pipelines
45 mph
3179
7,000+
26
2011 2014 2016 2017

CI/CD with GoCentral
5
build
deploy
feature-branch (PR)
s8d7fss.myapp.int.godaddy.com
testing:
➔ smoke
➔ bucket
➔ performance
➔ etc
appname: myapp
ssl: true
slack: ‘#myapp’
node_version: 6.10.2
start_command: node build/server.js
hosts:
- myapp.int
testing:
tests:
- shell: npm run ... smoke-tests
- parallel:
- npm run tests --bucket A
- npm run tests --bucket B
- ...
/cicd.yaml master
build
myapp.int.godaddy.com

Testing Pyramid

Buckets
Smoke
Regression
New
TUT - Failing tests
App Area

Example of buckets and tags
bucket: "[dt_chrome_regression]
[mo-emu_chrome_regression]
[mo-android_chrome_regression_tut]
[mo-ios_safari_regression_tut]
[dt_firefox_regression] [dt_ie_regression] [footer]
[layout]"

Change Detection in CI/CD
appname: myapp
...
change_detection:
BUTTON_A_CHANGED:
- lib/buttonA.js
BUTTON_B_CHANGED:
- lib/buttonB.js
testing:
tests:
- shell: npm run ... smoke-tests
- parallel:
- shell: npm run tests --bucket buttonA
if: BUTTON_A_CHANGED
- shell: npm run tests --bucket buttonB
if: BUTTON_B_CHANGED
- ...
/cicd.yaml
change_detection:
BUTTON_A_CHANGED:
- lib/buttonA/*.js
CORE_CHANGED:
- lib/core/*.js
- shell: npm run tests --bucket buttonA
if: BUTTON_A_CHANGED
- shell: npm run tests --bucket regression
if: CORE_CHANGED

Customizable Test Runner
'multi="spec=- mocha-elk-reporter=-"ui-tester ./test/ui/specs
--config=./test/ui/config
--env ${NODE_ENV}
--server ${SERVER}
--tags ${TEST_TAGS}
--platform ${PLATFORM}
--browser ${TEST_BROWSER:-chrome}
--timeout ${TIMEOUT:-120000}
--tagsType and
--reporter mocha-multi
--project vnext-editor'

Test Run Numbers

Life Cycle of a Test

TMNT - reliability calculators
TMNT-bots Bucket
Service
Profile
Service
mocha-
elk-repo
rter
Architecture

{
"uuid":"image-00005",
"originalTags":"[dt_chrome_regression_tut](reliability: 77.50,
totalRuns: 40) [mo-emu_chrome_regression_tut](reliability: 100.00,
totalRuns: 40) [mo-android_chrome_regression_tut](reliability: 0.00,
totalRuns: 40) [mo-ios_safari_regression_tut](reliability: 0.00,
totalRuns: 40) [image](totalRuns: 0) [about](totalRuns: 0)
[dt_firefox_regression_tut](reliability: 100.00, totalRuns: 40)
[dt_ie_regression_tut](reliability: 15.00, totalRuns: 40)",
"newTags":"[dt_chrome_regression_tut] [mo-emu_chrome_regression]
[mo-android_chrome_regression_tut] [mo-ios_safari_regression_tut]
[image] [about] [dt_firefox_regression] [dt_ie_regression_tut]"
}

{
"uuid":"image-00005",
"originalTags":"[dt_chrome_regression_tut](reliability: 77.50,
totalRuns: 40) [mo-emu_chrome_regression_tut](reliability: 100.00,
totalRuns: 40) [mo-android_chrome_regression_tut](reliability: 0.00,
totalRuns: 40) [mo-ios_safari_regression_tut](reliability: 0.00,
totalRuns: 40) [image](totalRuns: 0) [about](totalRuns: 0)
[dt_firefox_regression_tut](reliability: 100.00, totalRuns: 40)
[dt_ie_regression_tut](reliability: 15.00, totalRuns: 40)",
“newTags”

Take Aways
• Adopt Continuous integration and continuous deployment.
• Following the Testing pyramid to scale
• Ability to selectively run tests.
• Adopt bucketing / categorization of tests.
• Enable Change detection.
• Measure and calculate reliability .

Questions?
Connect with us:
Daniel Clayton: https://www.linkedin.com/in/azwebmaster
Manoj Pahuja: https://www.linkedin.com/in/manojpahuja

Scaling Automated Tests: Choosing an Appropriate Subset

More Related Content

What's hot (20)

Similar to Scaling Automated Tests: Choosing an Appropriate Subset (20)

More from TechWell (20)

Recently uploaded (20)

Scaling Automated Tests: Choosing an Appropriate Subset