Jennifer Scandariato: Transform Your Team from QA to Test Engineering

©2017 iCIMS Inc. All Rights Reserved.
Transform your Team
from QA to Test Engineering
Jennifer Scandariato
Director of Test Engineering, iCIMS
@JScan

©2017 iCIMS Inc. All Rights Reserved.©2017 iCIMS Inc. All Rights Reserved.
Introduction
iCIMS, Inc.
• Director of Test Engineering
• Lead WIT community & initiative
• 21+ years of experience developing technical solutions and
engineering effectiveness
• Passionate supporter of economic empowerment, environmental causes,
and advocate for women in technology
@JScan

Agenda
• The testing landscape at iCIMS
• Challenges around test automation
• The need to transform our QA team
• Our transformation journey
• The results we achieved

Journey at iCIMS
2015
Stabilize and Mature QA
2017
Transformation
(QA -> Test Engineering)
2016
‘Shift Left’ & CoE

SSSSSSSSSSS
Five Stages of Test Maturity For Growth at iCIMS
Test Driven
Development
Centers of
Excellence
Automated
Defined
Initial
• QA team exists
• No formal QA process documented
• Testing being performed ad-hoc
• Formal QA team exists
• Formal test cases documented
• Tests executed manually
Scripts are created and execution in
an automated, batched way
• Known state, expected behavior
• Predictable, Repeatable
• High Performing & Stable
Test First that relies on short development tasks
producing minimum amount of code to pass tests
1
2
3
4
5WORLD CLASS
AD-HOC
BEST
IN CLASS

We are Agile (SCRUM)

iCIMS Test
Strategy
Code quality
analysis
Static code
analysis
Coding best
practices scans
Security
vulnerability
scans
Internationalization
scans
Dynamic code
analysis
Performance
profiling
White box
testing
“True”
unit tests
“Front end”
unit tests
“Junction”
integration
tests
“Rich”
unit tests
Acceptance
testing
Functional
tests
Smoke tests System tests
API tests UI tests
Browser tests
Desktop
browser tests
Mobile
browser tests
Mobile app
tests
/ Localization tests
Accessibility
tests
Big data tests
“System”
integration
tests
Regression
tests
Other
Upgrade tests
Backward
compatibility
tests
Exploratory
tests
Non-functional
tests
Performance
tests
Load tests
Stress tests
Soak tests
Security tests
Dynamic
security tests
Penetration
tests
/ Localization tests
Translation
proof-reading
In-context
linguistic
validation
Visual
inspection
Customer
testing
Beta tests
So, what does our Test Strategy Look Like at iCIMS?
Can be used for build verification (fast running) 2018 focus area

Test Labs – Centers of Excellence (CoE’s)
Performance Security Accessibility Localization

Team Composition & Challenges
Developer
SDET
Developer
Manual
tester
Developer Developer
Our SDET’s were outnumbered!

Test Automation Tools and Technologies @ iCIMS

Our Automation Test Coverage Was Flat
1. Effort - The time to execute
manual test cases is lengthy
2. Quality - We find defects too
late in the SDLC
3. Time to Market - Developers
don’t have enough runway
to fix defects prior to a push
to production

The future of test automation (source: Tricentis)

Now What?
13
?Now what?

Option 1: Hire Help
CONSULTANTS

Option 2: Train Your Team
TRAIN YOUR TEAM

Option 3: Do Nothing
DO NOTHING

0
iCIMS Chose Option 2
17
OUR TRANSFORMATION
JOURNEY
Bootcamp
Conception
Initial communication
Continuing education
Exercise new skills

Phase 1:
Conception

Brainstorming and Convincing…
•We started by brainstorming for ideas on how to
accomplish this transformation
•We collaborated with HR
•I had to convince my leaders this was the right thing to do
•We worked together to identify how to make this successful

Top-Level Challenge & Gaining Buy-in
CFO asks CEO
What happens if we invest
in developing our people
and then they leave us?
CEO
What happens if we
don’t, and they stay?

Establish our goals and define next steps
Learner
User
Practitioner
Expert
SME
Novice
Has no prior experience with programming
and test automation
Has successfully completed in-depth boot-camp focusing on Java Programming,
Selenium and the iCIMS Automation Framework
Can independently execute tests, analyze results, debug
issues and perform minor script maintenance
Can develop new automation scripts based on
existing templates and guidance
Can develop new automation scripts independently
Can design and modify automation
frameworks across multiple
tools / technologiesOUR 2017
GOAL!

Reconcile new job descriptions…
Manual
testers
SDETS
SDET I
SDET II
SDET III
SDET IV
Tester I
Tester II
Tester III
Tester IV
Programming
Exp.
Test Engineers
Formal QA
Exp.
Professional
Years Exp.
HR APPROVED!

Phase 2:
Initial
Communication

Talk to the Team
Present the idea at team town-hall
This is a good thing for their career
Follow up with one on one meetings with each individual
Lay out the goals for the program clearly

Address Concerns and Fears
I hate my new job title!
SDET was so cool 
It feels like I am
getting demoted! Will I lose my job?
How will I compete
with the SDET’s?
KEEP
CALM
AND
REASSURE

Phase 3:
Bootcamp

Prepare for the Boot Camp
Lay down course structure: 100 -> 200 -> 300
(Awareness -> Practitioner -> Certification)
Identify “professors”:
In-house Vs. External
Prepare training material
Leverage existing SDET’s as
“buddies” to the manual testers
Hands-on exercises

Course Structure
100 Test Engineering
Course Duration
Software Quality Assurance Basics 3 hours
iCIMS Test Strategy 2 hours
Accessibility Testing 1 hour
Performance Testing 1 hour
Security Testing 1 hour
Localization Testing 1 hour
Microservices Testing 1 hour
Course Duration
Java Basics 10 hours
Java Advanced I 2 hours
Java Advanced II 2 hours
Selenium User 4 hours
Selenium Practitioner 4 hours
Selenium Expert 4 hours
Course Duration
iCIMS Test Automation Framework 4 hours
Mobile Test Automation using Appium 2 hours
API Test Automation using Rest Assured 2 hours
Security Testing Tools 2 hours
Accessibility Testing Tools 2 hours
Overall training time: 48 hours

Schedule and Execute
• Divided the team into 2 groups
• Assigned a “buddy” for each person
• Scheduled a series of 2 hour sessions:
• 14 hours of Java training
• 12 hours of Selenium training
• Across a 3-month period for each group

Java
Java Basics &
Programming
Concepts
•Eclipse IDE – introduction,
installation & set-up
•Variables
•Fundamental data types
& their operators
•Conditional operators
•String methods
•Conditional statements (if/else, switch)
•Looping statements (for, while)
•Arrays
•Packages
•Exception handling
•Lists
•Read and write files
•Use Java API
Object Oriented
Programming
Concepts
•Create classes and objects
•Class fields
•Methods and constructors
•Method overloading
•Set/get methods
•Class/field/method modifiers
•Xpath and CSS selector syntax
Java Advanced
Concepts I
•Class inheritance
•Base classes
•Method overriding
•Polymorphism
•Interfaces
•Static classes
•Inner classes
•Abstract classes
Java Advanced
Concepts II
•Code refactoring
•Design patterns

Selenium
Selenium User
•Introduction to Selenium
•A brief history of Selenium
•The Selenium suite of tools
•Selenium IDE
•Locators
•Script execution using the Taurus
automation framework
Selenium Practitioner
•WebDriver deep-dive
•Test design patterns
•Basic script creation and maintenance
using Taurus
Selenium Expert
•Special considerations while using
WebDriver
•Selenium internals
•Selenium Grid
•Taurus framework deep-dive
Selenium SME
•Introduction to Appium
•Introduction to Cucumber
•Taurus framework refactoring and
improvements
•Introduction to code coverage

Ask for Feedback

After Group #1 – Lessons Learned
Feedback: Provide dedicated time to practice and do hands-on exercises
Action: We reserved an additional 1 hour after training classes; with instructor
Feedback: Students needed a way to ask for quick help without interrupting work or waiting for the next class
Action: Create ‘closed’ slack channels to encourage collaboration
Feedback: Students required more training on the iCIMS test automation framework
Action: Updated course with additional classes focusing on the iCIMS test automation framework

Graduation Day!
Bootcamps are typically hard work! Celebrating creates comradery, it’s a BIG deal –
these were not programmers nor did everyone have a computer science degree!

Phase 4:
Exercise New Skills

Time
Fear
Framework
Higher expectations
Lack of documentation
Challenges

©2017 iCIMS Inc. All Rights Reserved.©2017 iCIMS Inc. All Rights Reserved.©2017 iCIMS Inc. All Rights Reserved.
Pearls of Wisdom
Provide opportunities to apply the skills learned ‘every day’
Start with simple activities such as script execution and maintenance,
before moving up towards script development
Continue to leverage the more seasoned Test Engineers as “buddies”
Encourage people to make changes to live scripts, and reassure them that
we could always revert changes if required
Implement best practices around code reviews to provide timely feedback
on any script changes

Phase 5:
Continuing
Education

Kaizen = Continuous Improvements
Improve overall documentation of the framework & scripts, jot down FAQ’s, etc.
Improve organization of code within the framework, break down into smaller pieces
where possible
Create “template” scripts that could be used as a starting point to create new scripts
Expand the training to include additional topics such as API testing, mobile testing, etc.
Continue to collect feedback that could be used to improve the transformation process

It’s All About
the Results

Novice
Learner
User
Practitioner
Expert
SME
M A T U R I T Y
0 1 2 3 4 5
Novice Has no prior experience with programming and test automation
Learner Has successfully completed in-depth boot-camp focusing on Java Programming, Selenium and the iCIMS Automation Framework
User Can independently execute tests, analyze results, debug issues and perform minor script maintenance
Practitioner Can develop new automation scripts based on existing templates and guidance
Expert Can develop new automation scripts independently
SME Can design and modify automation frameworks across multiple tools / technologies
Transformation – We are on TRACK!
WE ARE
HERE

Results
• Upskilled Engineers; more hands helping with automation
• Improved Automation Coverage across programs (above 85% for newly adopted Product
Portfolios)
• Better employee engagement; employees thrilled to learn and grow
• Blurred lines between Test Engineers and Developers – our first promotion from TE to Dev
occurred this month!
• Increased test execution productivity
• Total: ~20,000 test cases
• Manual execution: 50-100 test cases per team/per day
• Automated execution: 500 test cases per team/per night
• Better test coverage
• Standardization
• Room for exploration testing
500% Increase

Testimonials
Initially, I created an automated test for the Streaming API, testing the Person profile.
This test included running an IDT, disabling the cool-off to run multiple IDTs, and
receiving the correct JSON for an updated field. Using this test as a template, and
with some help, I adapted it to work with the Job and Company profile types!
-Sarala Vasudevan
“Just wanted to share this exciting info: one of my team’s automation script has
always been failing. I am happy to say that I was able to debug and identify the
issues seen and with little bit of guidance I have fixed the failed tc’s. I will send the
code for review and commit the changes next week.”
-Suril Dalal
“
”
“
”©2017 iCIMS Inc. All Rights Reserved.

Testimonials
This past Friday, I committed my first code! I just wanted to
share this personal and professional accomplishment with you
and the team involved with the java/selenium training. Thank
you to everyone involved in setting up this training program
and to the awesome instructors. Also, I would like to give a
special thanks to Aditya for coaching and code reviewing my
code and Suril and Sarala for periodically checking up on me
and providing me cool tips and tricks. You guys rock!!
–Tony Bomova
“
”©2017 iCIMS Inc. All Rights Reserved.

Add Testimonial Video here!
Click Here for Video

Attendee Takeaways
• Software testing is often accounted for more than 50% of total development
costs – lack of automation is a key root cause for this (source: Journal of
Systems and Software)
• Manual testing is not able to provide the quick and consistent feedback that
DevOps paradigms such as CI/CD are built around
• Specialized skillsets are one of the biggest roadblocks to test automation – the
onus is on test engineering organizations to transform themselves!
• Transformation is challenging, not impossible – all it takes is a positive intent
and belief in your team
• Transformations take time – be patient and support your team all the way

Questions?
Director of Test Engineering, iCIMS
www.icims.com
@JScan @iCIMS

Jennifer Scandariato: Transform Your Team from QA to Test Engineering

More Related Content

What's hot (20)

Similar to Jennifer Scandariato: Transform Your Team from QA to Test Engineering (20)

More from Anna Royzman (20)

Recently uploaded (20)

Jennifer Scandariato: Transform Your Team from QA to Test Engineering