Continuous Validation of 
Load Test Suites
Mark D. Syer, Zhen Ming Jiang, Meiyappan Nagappan, 
Ahmed E. Hassan, Mohamed Nasser and Parminder Flora
mdsyer@cs.queensu.ca
1
2
Failures in ULS systems are typically 
due to performance issues
3
Load testing 
may detect 
failures before 
they occur in 
the field
4
0
200
400
600
800
1000
1200
2005 2006 2007 2008 2009 2010 2011 2012
User Growth Over the Years (in millions)
Facebook Twitter LinkedIn WordPress Tumblr Google+ Pinterest
Field workloads change continuously
13.5M Mobile Users
56M Mobile Users
New HTML5 App
Performance 
analysts can 
compare field 
and load test 
workloads using 
execution logs
6
Comparing 
workloads
is difficult
7
Huge amount of data
Rapidly evolving systems
Workloads are made of hundreds or 
thousands of individual workers
Generate
Signatures
Detect
Outliers
Inspect
Outliers
Our approach identifies events 
that differ between load testing 
and field workloads
9
Test logs
Field logs
10
We generate a signature for each 
unique worker ID
Identify unique user IDs 
00:01, Alice starts a conversation with Bob
00:01, Alice says `hi' to Bob
00:02, Alice says `are you busy?' to Bob
00:11, Bob says `yes' to Alice
00:12, Alice says `ok' to Bob
00:18, Alice ends a conversation with Bob
11
Identify the logs attributable to each 
unique user ID 
00:01, Alice starts a conversation with Bob
00:01, Alice says `hi' to Bob
00:02, Alice says `are you busy?' to Bob
00:11, Bob says `yes' to Alice
00:12, Alice says `ok' to Bob
00:18, Alice ends a conversation with Bob
12
Abstract log lines to events
00:01, Alice starts a conversation with Bob
00:01, Alice says `hi' to Bob
00:02, Alice says `are you busy?' to Bob
00:12, Alice says `ok' to Bob
00:18, Alice ends a conversation with Bob
00:11, Bob says `yes' to Alice
13
Count the events for each use ID
Alice Bob
USER starts a conversation with USER  1 0
USER says MSG to USER  3 1
USER ends a conversation with USER 1 0
14
Detect
Outliers
Inspect
Outliers
We identify and inspect 
outlying signatures
15
16
Can we detect...
17
Issue difference?
Intensity and      
feature differences?
Intensity difference?
Our Approach
18
State‐of‐the‐
Practice
19
State‐of‐the‐practice is to compare 
event occurrence frequencies
Test logs
Field logs
Event Test Field
A 10/s 11/s
B 5/s 7/s
C 1/s 100/s
20
A datanode
fails in the field
Hadoop distributes a workload across 
the nodes of a computing cluster
INFO org.apache.hadoop.hdfs.DFSClient:
Abandoning block blk_id
INFO org.apache.hadoop.hdfs.DFSClient:
Exception in createBlockOutputStream
java.io.IOException: Bad connect ack
with firstBadLink ip_address
WARN org.apache.hadoop.hdfs.DFSClient:
Error Recovery for block blk_id bad
datanode_id ip_address
INFO
org.apache.hadoop.mapred.TaskTracker:
attempt_id progress
INFO org.apache.hadoop.hdfs.DFSClient:
Abandoning block blk_id
INFO org.apache.hadoop.hdfs.DFSClient:
Exception in createBlockOutputStream
java.io.IOException: Bad connect ack
with firstBadLink ip_address
WARN org.apache.hadoop.hdfs.DFSClient:
Error Recovery for block blk_id bad
datanode_id ip_address
INFO
org.apache.hadoop.mapred.TaskTracker:
attempt_id progress
21
INFO org.apache.hadoop.hdfs.DFSClient:
Abandoning block blk_id
INFO org.apache.hadoop.hdfs.DFSClient:
Exception in createBlockOutputStream
java.io.IOException: Bad connect ack
with firstBadLink ip_address
WARN org.apache.hadoop.hdfs.DFSClient:
Error Recovery for block blk_id bad
datanode_id ip_address
INFO
org.apache.hadoop.mapred.TaskTracker:
attempt_id progress
Our approach flags events
with high precision
0
20
40
60
80
100
Issue Feature and Intensity Intensity
Precision
Our Approach State‐of‐the‐Practice
22
Our Approach
23
State‐of‐the‐
Practice
88% 44%
24

More Related Content

PDF
Continuous performance: Load testing for developers with gatling @ JavaOne 2016
PDF
STARWest: Use Jenkins For Continuous 
Load Testing And Mobile Test Automation
ODP
Performance Test Automation With Gatling
PDF
Gatling @ Scala.Io 2013
PDF
Load test REST APIs using gatling
PPTX
Use of Formal Methods at Amazon Web Services
PPTX
When do software issues get reported in large open source software
PDF
When do software issues get reported in large open source software - Rakesh Rana
Continuous performance: Load testing for developers with gatling @ JavaOne 2016
STARWest: Use Jenkins For Continuous 
Load Testing And Mobile Test Automation
Performance Test Automation With Gatling
Gatling @ Scala.Io 2013
Load test REST APIs using gatling
Use of Formal Methods at Amazon Web Services
When do software issues get reported in large open source software
When do software issues get reported in large open source software - Rakesh Rana

Similar to Continuous validation of load test suites (20)

DOC
Sathishkumar_M
DOC
Scalability Manuscript for Star98
DOCX
TimResume
PDF
smartwatch-user-identification
PPTX
Improve the Impact of DevOps
PDF
Deep Dive Into Deep Learning : How AI is Powering the Future of Endpoint Secu...
DOCX
SCGOV Report
PPTX
Testing and debugging Flex/AS3 applications
PDF
Mobile App Testing: The Good, the Bad, and the Ugly
PPTX
Curiosity Software Presents: Isolating blast radiuses for testing - How to no...
PPT
Fast, Strong & Nimble Mobile Performance Testing
PPTX
Troubleshooting App Health and Performance with PCF Metrics 1.2
DOC
Krishnamurthy senoir qa_resume
PPTX
New Tool for Automating Exchange Management Tasks
PDF
The Future of Mobile App Testing - Reliable and Efficient Solutions.pdf
PPT
Advanced sql injection
PPTX
Federal Webinar: Slow is the New Broke: Improving Government Efficiency with ...
PDF
Agile methods cost of quality
PDF
Agile Methods Cost of Quality: Benefits of Testing Early & Often
ODP
Mobile App Security Testing -2
Sathishkumar_M
Scalability Manuscript for Star98
TimResume
smartwatch-user-identification
Improve the Impact of DevOps
Deep Dive Into Deep Learning : How AI is Powering the Future of Endpoint Secu...
SCGOV Report
Testing and debugging Flex/AS3 applications
Mobile App Testing: The Good, the Bad, and the Ugly
Curiosity Software Presents: Isolating blast radiuses for testing - How to no...
Fast, Strong & Nimble Mobile Performance Testing
Troubleshooting App Health and Performance with PCF Metrics 1.2
Krishnamurthy senoir qa_resume
New Tool for Automating Exchange Management Tasks
The Future of Mobile App Testing - Reliable and Efficient Solutions.pdf
Advanced sql injection
Federal Webinar: Slow is the New Broke: Improving Government Efficiency with ...
Agile methods cost of quality
Agile Methods Cost of Quality: Benefits of Testing Early & Often
Mobile App Security Testing -2
Ad

More from SAIL_QU (20)

PDF
Studying the Integration Practices and the Evolution of Ad Libraries in the G...
PDF
Studying the Dialogue Between Users and Developers of Free Apps in the Google...
PPTX
Improving the testing efficiency of selenium-based load tests
PDF
Studying User-Developer Interactions Through the Distribution and Reviewing M...
PDF
Studying online distribution platforms for games through the mining of data f...
PPTX
Understanding the Factors for Fast Answers in Technical Q&A Websites: An Empi...
PDF
Investigating the Challenges in Selenium Usage and Improving the Testing Effi...
PDF
Mining Development Knowledge to Understand and Support Software Logging Pract...
PPTX
Which Log Level Should Developers Choose For a New Logging Statement?
PPTX
Towards Just-in-Time Suggestions for Log Changes
PDF
The Impact of Task Granularity on Co-evolution Analyses
PPTX
A Framework for Evaluating the Results of the SZZ Approach for Identifying Bu...
PPTX
How are Discussions Associated with Bug Reworking? An Empirical Study on Open...
PPTX
A Study of the Relation of Mobile Device Attributes with the User-Perceived Q...
PDF
A Large-Scale Study of the Impact of Feature Selection Techniques on Defect C...
PPTX
Studying the Dialogue Between Users and Developers of Free Apps in the Google...
PDF
What Do Programmers Know about Software Energy Consumption?
PPTX
Threshold for Size and Complexity Metrics: A Case Study from the Perspective ...
PDF
Revisiting the Experimental Design Choices for Approaches for the Automated R...
PPTX
Measuring Program Comprehension: A Large-Scale Field Study with Professionals
Studying the Integration Practices and the Evolution of Ad Libraries in the G...
Studying the Dialogue Between Users and Developers of Free Apps in the Google...
Improving the testing efficiency of selenium-based load tests
Studying User-Developer Interactions Through the Distribution and Reviewing M...
Studying online distribution platforms for games through the mining of data f...
Understanding the Factors for Fast Answers in Technical Q&A Websites: An Empi...
Investigating the Challenges in Selenium Usage and Improving the Testing Effi...
Mining Development Knowledge to Understand and Support Software Logging Pract...
Which Log Level Should Developers Choose For a New Logging Statement?
Towards Just-in-Time Suggestions for Log Changes
The Impact of Task Granularity on Co-evolution Analyses
A Framework for Evaluating the Results of the SZZ Approach for Identifying Bu...
How are Discussions Associated with Bug Reworking? An Empirical Study on Open...
A Study of the Relation of Mobile Device Attributes with the User-Perceived Q...
A Large-Scale Study of the Impact of Feature Selection Techniques on Defect C...
Studying the Dialogue Between Users and Developers of Free Apps in the Google...
What Do Programmers Know about Software Energy Consumption?
Threshold for Size and Complexity Metrics: A Case Study from the Perspective ...
Revisiting the Experimental Design Choices for Approaches for the Automated R...
Measuring Program Comprehension: A Large-Scale Field Study with Professionals
Ad

Continuous validation of load test suites