SlideShare a Scribd company logo
Automated Unit Test Generation

Vassil Popovski, VMware
Automated unit test generation presentation  -  Java2ays 2013
Automated unit test generation presentation  -  Java2ays 2013
The Cost of Software Bugs?
$59.9 B per year in USA (NIST, 2002)*
$312 B per year worldwide (Cambridge University, 2012)**

(*) http://www.nist.gov/director/planning/upload/report02-3.pdf
(**) http://www.roguewave.com/DesktopModules/Bring2mind/DMX/Download.aspx?
entryid=1606&command=core_download&PortalId=0&TabId=607
The Cost of Software Bugs?

$312,000,000,000
The Cost of Software Bugs?
Total market capitalization for all 3
companies(*):
!

$177.36 B
(*) Google finance - as of Nov 2nd, 2013
The Cost of World Hunger?

$40B per year(*)

(*) According to United Nations
Automated unit test generation presentation  -  Java2ays 2013
The Cost of Fixing Bugs?

http://www.amazon.com/Assessments-Benchmarks-Addison-Wesley-Information-Technology/dp/
0201485427?ie=UTF8&s=books&qid=1209056706&sr=1-1
Software Testing Pyramid
E2E/GUI testing:

1-5%

Integration testing: 4-15%
Unit Testing:

http://blogs.agilefaqs.com/2011/02/01/inverting-the-testing-pyramid/

80-90%
Techniques/Tools

The goal: High (close to 100% code coverage*)
1. Random data generation
2. Search based
3. Symbolic/Concolic based
Anatomy of a Unit Test
Arrange
Act
Assert
Anatomy of a Unit Test
Arrange Data
Arrange Mocks/Stubs
Act
Assert
Anatomy of a Unit Test
Evaluation of Tools

Arrange/
Generate Data
Arrange/
Generate
Mocks/Stubs
Act/Invoke
methods
Assert/Verify
the results
Techniques/Tools

x != 100

x>=z

x == 100

x<z
Techniques/Tools
Random data generation:
!

x != 100

x == 100

Feed x,y with random data
x>=z

x<z
Demo
Evaluation of Tools
junit-quickcheck
feed4JUnit
Arrange/
Generate Data
Arrange/
Generate
Mocks/Stubs
Act/Invoke
methods
Assert/Verify
the results

yes - some
no
no
no
Techniques/Tools
1. Random Data Generation:
• junit-quickcheck (Java)
https://github.com/pholser/junit-quickcheck/
• Feed4JUnit & Feed4TestNG + Benerator (java)
http://databene.org/feed4-tools.html
http://databene.org/databene-benerator.html
• Randoop (Java + .NET)
https://code.google.com/p/randoop/
• T2 (Java)
https://code.google.com/p/t2framework/
Techniques/Tools
Search based approach:
!

x != 100

x == 100

1. Start with random/initial
data (x=1, y=1)
x>=z

x<z
Techniques/Tools
x=1, y=1
Search based approach:
!

x != 100

x == 100

1. Start with random/initial
data (x=1, y=1)
x>=z

x<z
Techniques/Tools
x=1, y=1
Search based approach:
!

x != 100

x == 100

1. Start with random/initial
data (x=1, y=1)
x>=z

x<z
Techniques/Tools
x=100, y=1
Search based approach:
!

1. Start with random/initial
data (x=1, y=1)
2. Based on coverage
information - select new
value(s) (x==100)

x != 100

x>=z

x == 100

x<z
Techniques/Tools
x=100, y=1
Search based approach:
x != 100

!

1. Start with random/initial
data (x=1, y=1)
2. Based on coverage
information - select new
value(s) (x==100)
!

x>=z

x == 100

x<z
Demo
Automated unit test generation presentation  -  Java2ays 2013
Automated unit test generation presentation  -  Java2ays 2013
Evaluation of Tools
junit-quickcheck
feed4JUnit
Arrange/
Generate Data
Arrange/
Generate
Mocks/Stubs
Act/Invoke
methods
Assert/Verify
the results

yes - some

EvoSuite

yes

no

no

no

yes

no

yes
Techniques/Tools
2. Search Based:
• EvoSuite (Java)
http://www.evosuite.org/
• Testful (Java)
https://code.google.com/p/testful/
Techniques/Tools
x=X, y=Y, z=2*Y

Symbolic based approach:
!

x != 100

x == 100

1. Evaluate path conditions (PC)
PC: X!=100, Y
PC: X == 100, Y
on every branch
2. For each leaf - calculate X & Y
x>=z
x<z
using a constraints solver:
1. X!=100, Y
=> (x=1, y=1)
2. X=100, X>=2*Y =>(x=100, y=1)
3. X=100, X<2*Y =>(x=100, y=51)

PC: X == 100, PC: X == 100,
X>=2*Y
X<2*Y
Techniques/Tools
Concolic based
approach:

x != 100

x == 100

!

Optimization of symbolic
execution - but use real
values for some of the
branches

x>=z

x<z
Demo
Evaluation of Tools
junit-quickcheck
feed4JUnit
Arrange/
Generate Data
Arrange/
Generate
Mocks/Stubs
Act/Invoke
methods
Assert/Verify
the results

yes - some

EvoSuite

PEX (+ Moles)

yes

yes

no

no

no

yes

yes

no

yes

yes

yes - some
Techniques/Tools
3. Symbolic/Concolic based:
• Microsoft Pex (.NET)
http://research.microsoft.com/en-us/projects/pex/
• jCUTE (Java)
http://osl.cs.uiuc.edu/software/jcute/
• jalangi (Java Script)
https://github.com/SRA-SiliconValley/jalangi
• AgitarOne (Java, commercial)
http://www.agitar.com/solutions/products/
agitarone.html

• Jtest (Java, commercial)
http://www.parasoft.com/jsp/products/jtest.jsp
Best Usage of Auto Generated Unit
Tests
1.Regression testing of legacy code
!

2.To support your unit testing / TDD
Q&A
Thank you
!

Vassil Popovski
!

Sr. Director, R&D, VMware
vpopovski@vmware.com

More Related Content

PDF
Anomaly detection
PDF
A Survey on Dynamic Symbolic Execution for Automatic Test Generation
PDF
Automated Unit Test Generation - ISTA 2013
PPTX
Testing ASP.NET - Progressive.NET
PDF
Using Robots for App Testing
PDF
Software testing
PDF
Whatever it takes - Fixing SQLIA and XSS in the process
PDF
ML基本からResNetまで
Anomaly detection
A Survey on Dynamic Symbolic Execution for Automatic Test Generation
Automated Unit Test Generation - ISTA 2013
Testing ASP.NET - Progressive.NET
Using Robots for App Testing
Software testing
Whatever it takes - Fixing SQLIA and XSS in the process
ML基本からResNetまで

Similar to Automated unit test generation presentation - Java2ays 2013 (20)

PDF
Learning from Computer Simulation to Tackle Real-World Problems
PDF
Testing and Building Android
PDF
Android Building, Testing and reversing
PDF
Pretty pictures - Brandon Satrom
PDF
Django tricks (2)
PPTX
Youtube big data analysis using hadoop,pig,hive
PDF
AUTOCODECOVERGEN: PROTOTYPE OF DATA DRIVEN UNIT TEST GENRATION TOOL THAT GUAR...
PDF
Unit Testing on Android - Droidcon Berlin 2015
ODP
Good Practices On Test Automation
DOCX
Lab Deliverable for Lab nYour NameDateTitle Creating, Using, Remo.docx
PDF
DSR Testing (Part 2)
ODP
The Art Of Debugging
PPTX
Beyond the basics of SonarQube: improve your Java(Script) code even further
PPTX
AI Development with H2O.ai
PDF
Examview testmanager userguide 8.1
PDF
SOFTWARE ENGINEERING UNIT 6 Ch 13
PPT
Pragmatic Parallels: Java and JavaScript
PDF
Use Machine Learning to Get the Most out of Your Big Data Clusters
PDF
멀티플랫폼 앱 개발과 테스팅
PDF
jQuery in the [Aol.] Enterprise
Learning from Computer Simulation to Tackle Real-World Problems
Testing and Building Android
Android Building, Testing and reversing
Pretty pictures - Brandon Satrom
Django tricks (2)
Youtube big data analysis using hadoop,pig,hive
AUTOCODECOVERGEN: PROTOTYPE OF DATA DRIVEN UNIT TEST GENRATION TOOL THAT GUAR...
Unit Testing on Android - Droidcon Berlin 2015
Good Practices On Test Automation
Lab Deliverable for Lab nYour NameDateTitle Creating, Using, Remo.docx
DSR Testing (Part 2)
The Art Of Debugging
Beyond the basics of SonarQube: improve your Java(Script) code even further
AI Development with H2O.ai
Examview testmanager userguide 8.1
SOFTWARE ENGINEERING UNIT 6 Ch 13
Pragmatic Parallels: Java and JavaScript
Use Machine Learning to Get the Most out of Your Big Data Clusters
멀티플랫폼 앱 개발과 테스팅
jQuery in the [Aol.] Enterprise
Ad

Recently uploaded (20)

PPTX
Understanding_Digital_Forensics_Presentation.pptx
PPTX
Detection-First SIEM: Rule Types, Dashboards, and Threat-Informed Strategy
PDF
Dropbox Q2 2025 Financial Results & Investor Presentation
PDF
Blue Purple Modern Animated Computer Science Presentation.pdf.pdf
PDF
Electronic commerce courselecture one. Pdf
PDF
Network Security Unit 5.pdf for BCA BBA.
PDF
Approach and Philosophy of On baking technology
PDF
Advanced IT Governance
PPT
Teaching material agriculture food technology
PPTX
MYSQL Presentation for SQL database connectivity
PDF
KodekX | Application Modernization Development
PDF
[발표본] 너의 과제는 클라우드에 있어_KTDS_김동현_20250524.pdf
PDF
Build a system with the filesystem maintained by OSTree @ COSCUP 2025
PDF
TokAI - TikTok AI Agent : The First AI Application That Analyzes 10,000+ Vira...
PDF
GamePlan Trading System Review: Professional Trader's Honest Take
PPTX
Big Data Technologies - Introduction.pptx
PPTX
breach-and-attack-simulation-cybersecurity-india-chennai-defenderrabbit-2025....
PDF
Diabetes mellitus diagnosis method based random forest with bat algorithm
PPT
“AI and Expert System Decision Support & Business Intelligence Systems”
PDF
Per capita expenditure prediction using model stacking based on satellite ima...
Understanding_Digital_Forensics_Presentation.pptx
Detection-First SIEM: Rule Types, Dashboards, and Threat-Informed Strategy
Dropbox Q2 2025 Financial Results & Investor Presentation
Blue Purple Modern Animated Computer Science Presentation.pdf.pdf
Electronic commerce courselecture one. Pdf
Network Security Unit 5.pdf for BCA BBA.
Approach and Philosophy of On baking technology
Advanced IT Governance
Teaching material agriculture food technology
MYSQL Presentation for SQL database connectivity
KodekX | Application Modernization Development
[발표본] 너의 과제는 클라우드에 있어_KTDS_김동현_20250524.pdf
Build a system with the filesystem maintained by OSTree @ COSCUP 2025
TokAI - TikTok AI Agent : The First AI Application That Analyzes 10,000+ Vira...
GamePlan Trading System Review: Professional Trader's Honest Take
Big Data Technologies - Introduction.pptx
breach-and-attack-simulation-cybersecurity-india-chennai-defenderrabbit-2025....
Diabetes mellitus diagnosis method based random forest with bat algorithm
“AI and Expert System Decision Support & Business Intelligence Systems”
Per capita expenditure prediction using model stacking based on satellite ima...
Ad

Automated unit test generation presentation - Java2ays 2013