![Background – Test Suite Maintenance
April 12, 2018 3
• Test suites are not static entities[1]
– Developers must keep maintaining test suites
• Adding new tests, modifying existing tests, or delete
unnecessary tests
Test suite maintenance is necessary to build reliable software
Old
Test Suite
New
Tests
Developer
add
New
Test Suite
Ex. adding
new tests
Old
Test Suite
Developer
modify
New
Test Suite
Ex. modifying
existing tests
Existing
Tests
Modified
Tests
[1] L. Pinto et al. “Understanding Myths and Realities of Test-Suite Evolution”, FSE 2012](https://image.slidesharecdn.com/sacpresentation-180416012734/85/SAC-2018-AutoPUT-An-Automated-Technique-for-Retrofitting-Closed-Unit-Tests-into-Parameterized-Unit-Tests-3-320.jpg)
![Background – Test Maintainability
April 12, 2018 4
• Developers need to understand purpose of tests[2]
– Good readability helps developers to maintain test suites[3]
– Simplifying tests increases understandability of tests[4]
Test maintainability ≒ Test understandability
Test Suite
maintain
read Developers
New
Test Suite
Understandability ↑
Test Suite
maintain
read Developers
New
Test Suite
Understanding ↑ Understanding ↓
Understandability ↓
[2] E. Daka et al. “Modeling Readability to Improve Unit Tests”, FSE 2015
[3] B. Zhang et al. “Towards Automatically Generating Descriptive Names for Unit Tests”, ASE 2016
[4] Y. Lei et al. “Minimization of Randomized Unit Test Cases”, ISSRE 2005](https://image.slidesharecdn.com/sacpresentation-180416012734/85/SAC-2018-AutoPUT-An-Automated-Technique-for-Retrofitting-Closed-Unit-Tests-into-Parameterized-Unit-Tests-4-320.jpg)
![Background – Parameterized Unit Test (PUT)
April 12, 2018 5
Unit tests allowing parameters•
Enable to separate parameters and procedures–
Less code and independent of concrete parameters•
Easier to understand than closed unit tests (CUTs)[5]
However, PUTs are rare in the real world projects
[5] G. Fraser et al. “Generating Parameterized Unit Tests”, ISSTA 2011
// no parameters -> CUT
@Test public void testAdd1() {
int expected = 3;
int actual = Adder.add(1, 2);
assertEquals(expected, actual);
}
@Test public void testAdd2() {
int expected = 0;
int actual = Adder.add(1, -1);
assertEquals(expected, actual);
}
// PUT
@Theory public void testAdd(int[] params) {
int expected = params[0];
int actual =
Adder.add(params[1], params[2]);
assertEquals(expected, actual);
}
parameterize](https://image.slidesharecdn.com/sacpresentation-180416012734/85/SAC-2018-AutoPUT-An-Automated-Technique-for-Retrofitting-Closed-Unit-Tests-into-Parameterized-Unit-Tests-5-320.jpg)
![Related Work – Parameterized Unit Tests
April 12, 2018 6
• Generating parameters for PUTs by symbolic execution[6]
Not full solution for practicality problem
PUTs
prepare Unit
Meister
input generate
Developers
Parameters
[6] N. Tillman et al. “Parameterized Unit Tests”, FSE 2005
High coverage
• Parameterizing auto-generated CUTs[5]
EvoStuite
generate
CUTs
parameterized
unit tests
input parameterize
PUTs
Less code than CUTs
Retrofitting existing unit tests is more difficult[5]
• Semi-automated retrofitting CUTs into PUTs[7]
CUTs
read
Pex
promote
parameters retrofit
PUTs
Semi
PUTs
input
Developers
[7] S. Thummalapenta et al. “Retrofitting Unit Tests for Parameterized Unit Testing”, FASE 2011
Who
detects?](https://image.slidesharecdn.com/sacpresentation-180416012734/85/SAC-2018-AutoPUT-An-Automated-Technique-for-Retrofitting-Closed-Unit-Tests-into-Parameterized-Unit-Tests-6-320.jpg)
![Challenges – Fully Automated Retrofitting CUTs
April 12, 2018 7
Developers do not have
sufficient resources to retrofit CUTs
2 challenges for fully automated retrofitting existing CUTs
CUTsCUTs
CUTsCUTs
About 25% of unit tests are easy
to be parameterized[8]
Project #CUT #PUT
Codec 643 24
Compress 867 11
Math 4100 1
But...
[8] D. Saff et al. “Theories in practice: Easy-to-write specifications that catch bugs”, MIT CSAIL Technical Report 2008
Fully automated retrofitting can solve practicality problem
Project
Repository
CUTs to be
parameterized
1. detect 2. generate
PUTs
check
Developers Common procedure and different parameters](https://image.slidesharecdn.com/sacpresentation-180416012734/85/SAC-2018-AutoPUT-An-Automated-Technique-for-Retrofitting-Closed-Unit-Tests-into-Parameterized-Unit-Tests-7-320.jpg)
![Approach – Detecting CUTs to be Parameterized
April 12, 2018 9
• Using *AST-based code-clone detection technique
– CUTs to be parameterized = kind of code clones[8]
– Effective detecting by static program analysis
• Ignoring values of literal nodes, and considering names of
identifiers
Accurately detecting in practical amount of time
*AST = Abstract Syntax Tree
add
compare
add
different procedure
@Test
public void testAdd1 () {
int expected = 3;
int actual = Adder.add(2, 1);
assertEquals(expected, actual);
}
@Test
public void testAdd2() {
int expected = 1;
int actual = Adder.add(-1, 2);
assertEquals(expected, actual);
}
@Test
public void testSub1() {
int expected = 1;
int actual = Subber.sub(2, 1);
assertEquals(expected, actual);
}
sub
AST](https://image.slidesharecdn.com/sacpresentation-180416012734/85/SAC-2018-AutoPUT-An-Automated-Technique-for-Retrofitting-Closed-Unit-Tests-into-Parameterized-Unit-Tests-9-320.jpg)
![Approach – Requirements for Generated PUTs
April 12, 2018 11
• Retrofitting CUTs often induce degradation[7]
Guaranteeing that degradation does not occur
CUTs
retrofit
PUTs
× Compile Error
× Test Failure
× Decrease in Coverage
• Confirming whether degradation does not occur
– If degradation occurs, AutoPUT excludes the PUT
Generated
PUTs
success? decreases?
Original
CUTs
no
Coverage
Measuring
&
Comparison
yes
yes
Compile
&
Test Execution
PUTs
No degradation
Discard Discard](https://image.slidesharecdn.com/sacpresentation-180416012734/85/SAC-2018-AutoPUT-An-Automated-Technique-for-Retrofitting-Closed-Unit-Tests-into-Parameterized-Unit-Tests-11-320.jpg)
![Evaluation - Results and Discussions (RQ1)
April 12, 2018 13
• Results
– SCC[9]: the-state-of-the-art tool for detecting code-clones
AutoPUT scored higher or the same score than SCC
in a practical amount of time
AutoPUT SCC4 SCC5 SCC6
Precision Recall Precision Recall Precision Recall Precision Recall
Avg. Score 72.5% 55.2% 1.3% 61.5% 2.4% 55.2% 3.8% 44.0%
Total Time 9.3 sec. N/A N/A N/A
[9] H. Sajnani et al. “SourcererCC: Scaling code clone detection to big-code.”,ICSE 2016
• Discussions
– Higher precision score than SCC* and the same recall
score as SCC5
– Spent 9.3 sec. for 7 projects → sufficiently practical](https://image.slidesharecdn.com/sacpresentation-180416012734/85/SAC-2018-AutoPUT-An-Automated-Technique-for-Retrofitting-Closed-Unit-Tests-into-Parameterized-Unit-Tests-13-320.jpg)
![Evaluation - Results and Discussions (RQ2)
April 12, 2018 14
• Results
AutoPUT generated 204 PUTs, reducing the
number of statements by 64.0%, in 8.5 hours
# of Generated PUTs # of Converted CUTs Reduction Ratio Time
204 675 64.0% 8.5 hrs
– Reduction ratio[5]: to
– Time: including coverage measuring time
#statements of PUTs #statements of CUT s
• Discussions
– 204 generated PUTs: 95.3% of PUT candidates
• 4.7% inducing degradation: implementation issue (e.g., Enum-typed
parameters)
– 64.0% reduction of statements that developers must maintain
– Spent 8.5 hours for all PUTs → sufficiently feasible (in a day)](https://image.slidesharecdn.com/sacpresentation-180416012734/85/SAC-2018-AutoPUT-An-Automated-Technique-for-Retrofitting-Closed-Unit-Tests-into-Parameterized-Unit-Tests-14-320.jpg)
SAC 2018: "AutoPUT: An Automated Technique for Retrofitting Closed Unit Tests into Parameterized Unit Tests"
- 1. AutoPUT: An Automated Technique for Retrofitting Closed Unit Tests into Parameterized Unit Tests Keita Tsukamoto†, Yuta Maezawa†, Shinichi Honiden‡ April 12, 2018 1 † Udzuki Inc., Japan ‡ Waseda University, Japan The 33rd ACM/SIGAPP Symposium On Applied Computing Pau, France
- 2. Outline • Background – Test Suite Maintenance – Parameterized Unit Test • Challenges: Fully Automated Parameterizing • Approach: AutoPUT – Detecting CUTs to be Parameterized – Generating PUTs from detected CUTs • Evaluation • Conclusion April 12, 2018 2
- 3. Background – Test Suite Maintenance April 12, 2018 3 • Test suites are not static entities[1] – Developers must keep maintaining test suites • Adding new tests, modifying existing tests, or delete unnecessary tests Test suite maintenance is necessary to build reliable software Old Test Suite New Tests Developer add New Test Suite Ex. adding new tests Old Test Suite Developer modify New Test Suite Ex. modifying existing tests Existing Tests Modified Tests [1] L. Pinto et al. “Understanding Myths and Realities of Test-Suite Evolution”, FSE 2012
- 4. Background – Test Maintainability April 12, 2018 4 • Developers need to understand purpose of tests[2] – Good readability helps developers to maintain test suites[3] – Simplifying tests increases understandability of tests[4] Test maintainability ≒ Test understandability Test Suite maintain read Developers New Test Suite Understandability ↑ Test Suite maintain read Developers New Test Suite Understanding ↑ Understanding ↓ Understandability ↓ [2] E. Daka et al. “Modeling Readability to Improve Unit Tests”, FSE 2015 [3] B. Zhang et al. “Towards Automatically Generating Descriptive Names for Unit Tests”, ASE 2016 [4] Y. Lei et al. “Minimization of Randomized Unit Test Cases”, ISSRE 2005
- 5. Background – Parameterized Unit Test (PUT) April 12, 2018 5 Unit tests allowing parameters• Enable to separate parameters and procedures– Less code and independent of concrete parameters• Easier to understand than closed unit tests (CUTs)[5] However, PUTs are rare in the real world projects [5] G. Fraser et al. “Generating Parameterized Unit Tests”, ISSTA 2011 // no parameters -> CUT @Test public void testAdd1() { int expected = 3; int actual = Adder.add(1, 2); assertEquals(expected, actual); } @Test public void testAdd2() { int expected = 0; int actual = Adder.add(1, -1); assertEquals(expected, actual); } // PUT @Theory public void testAdd(int[] params) { int expected = params[0]; int actual = Adder.add(params[1], params[2]); assertEquals(expected, actual); } parameterize
- 6. Related Work – Parameterized Unit Tests April 12, 2018 6 • Generating parameters for PUTs by symbolic execution[6] Not full solution for practicality problem PUTs prepare Unit Meister input generate Developers Parameters [6] N. Tillman et al. “Parameterized Unit Tests”, FSE 2005 High coverage • Parameterizing auto-generated CUTs[5] EvoStuite generate CUTs parameterized unit tests input parameterize PUTs Less code than CUTs Retrofitting existing unit tests is more difficult[5] • Semi-automated retrofitting CUTs into PUTs[7] CUTs read Pex promote parameters retrofit PUTs Semi PUTs input Developers [7] S. Thummalapenta et al. “Retrofitting Unit Tests for Parameterized Unit Testing”, FASE 2011 Who detects?
- 7. Challenges – Fully Automated Retrofitting CUTs April 12, 2018 7 Developers do not have sufficient resources to retrofit CUTs 2 challenges for fully automated retrofitting existing CUTs CUTsCUTs CUTsCUTs About 25% of unit tests are easy to be parameterized[8] Project #CUT #PUT Codec 643 24 Compress 867 11 Math 4100 1 But... [8] D. Saff et al. “Theories in practice: Easy-to-write specifications that catch bugs”, MIT CSAIL Technical Report 2008 Fully automated retrofitting can solve practicality problem Project Repository CUTs to be parameterized 1. detect 2. generate PUTs check Developers Common procedure and different parameters
- 8. Approach – Automated Technique for Retrofitting CUTs into PUTs (AutoPUT) April 12, 2018 8 • Addressing two challenges – Detecting CUTs to be parameterized → Detector – Generating PUTs from detected CUTs → Generator AutoPUT can help developers maintain test suites Developers 1. Detector 2. Generator @Test Project Repository No AutoPUT Degraded? Coverage Measuring @Theory @DataPoints @Theory @DataPoints
- 9. Approach – Detecting CUTs to be Parameterized April 12, 2018 9 • Using *AST-based code-clone detection technique – CUTs to be parameterized = kind of code clones[8] – Effective detecting by static program analysis • Ignoring values of literal nodes, and considering names of identifiers Accurately detecting in practical amount of time *AST = Abstract Syntax Tree add compare add different procedure @Test public void testAdd1 () { int expected = 3; int actual = Adder.add(2, 1); assertEquals(expected, actual); } @Test public void testAdd2() { int expected = 1; int actual = Adder.add(-1, 2); assertEquals(expected, actual); } @Test public void testSub1() { int expected = 1; int actual = Subber.sub(2, 1); assertEquals(expected, actual); } sub AST
- 10. Approach – Generating PUTs from detected CUTs April 12, 2018 10 • Extracting parameters from AST of each CUT – Parameter nodes = literal nodes, or constant value nodes • Parameter nodes with inconsistent values in all CUTs → parameters • Replacing parameter nodes with abstract variables – Judge each parameter is related to expected outputs or not Automated generating PUTs in practical amount of time input expected Related to expected outputs → replaced with `expected` proceduresparameters abstract variable node CUTA CUTB
- 11. Approach – Requirements for Generated PUTs April 12, 2018 11 • Retrofitting CUTs often induce degradation[7] Guaranteeing that degradation does not occur CUTs retrofit PUTs × Compile Error × Test Failure × Decrease in Coverage • Confirming whether degradation does not occur – If degradation occurs, AutoPUT excludes the PUT Generated PUTs success? decreases? Original CUTs no Coverage Measuring & Comparison yes yes Compile & Test Execution PUTs No degradation Discard Discard
- 12. Evaluation - Experimental Setting • Subjects April 12, 2018 12 Project # of Class # of Test Class # of Existing CUTs # of Existing PUTs BCEL 378 63 112 1 Codec 63 56 643 24 Compress 195 127 867 11 CSV 11 19 286 2 Digester 169 103 218 2 FileUpload 839 19 66 8 Math 847 522 4100 1 • Research Questions 1. Can AutoPUT accurately detect CUTs to be parameterized in a practical amount of time? 2. Can AutoPUT effectively generate PUTs in a feasible amount of time?
- 13. Evaluation - Results and Discussions (RQ1) April 12, 2018 13 • Results – SCC[9]: the-state-of-the-art tool for detecting code-clones AutoPUT scored higher or the same score than SCC in a practical amount of time AutoPUT SCC4 SCC5 SCC6 Precision Recall Precision Recall Precision Recall Precision Recall Avg. Score 72.5% 55.2% 1.3% 61.5% 2.4% 55.2% 3.8% 44.0% Total Time 9.3 sec. N/A N/A N/A [9] H. Sajnani et al. “SourcererCC: Scaling code clone detection to big-code.”,ICSE 2016 • Discussions – Higher precision score than SCC* and the same recall score as SCC5 – Spent 9.3 sec. for 7 projects → sufficiently practical
- 14. Evaluation - Results and Discussions (RQ2) April 12, 2018 14 • Results AutoPUT generated 204 PUTs, reducing the number of statements by 64.0%, in 8.5 hours # of Generated PUTs # of Converted CUTs Reduction Ratio Time 204 675 64.0% 8.5 hrs – Reduction ratio[5]: to – Time: including coverage measuring time #statements of PUTs #statements of CUT s • Discussions – 204 generated PUTs: 95.3% of PUT candidates • 4.7% inducing degradation: implementation issue (e.g., Enum-typed parameters) – 64.0% reduction of statements that developers must maintain – Spent 8.5 hours for all PUTs → sufficiently feasible (in a day)
- 15. Conclusion April 12, 2018 15 • Background – PUT: easier to understand than CUT • Challenge – Fully automated CUT-PUT retrofitting • Approach: AutoPUT – Detecting CUTs with similar structure – Identifying parameters and generate PUTs • Experimental Results – Generated 204 PUTs in 8.5 Hours AutoPUT can help developers to maintain test suites
- 16. Future Work – Extending AutoPUT April 12, 2018 16 • Investigating meaningfulness of generated PUTs – Current: Reducing #CUTs – Future: To be accepted by actual developers • Retrofitting PUTs to CUTs – Current: Unkind error messages from PUTs – Future: To handle software failures easily • Combining with symbolic execution – Current: Guaranteeing no degradation – Future: To achieve higher coverage Plan to make AutoPUT more practical and applicable