Test Dependencies and the Future of Build Acceleration
1 like1,211 views
The document discusses build acceleration in software testing, highlighting the dominance of testing in build times and the impacts of test dependencies on execution efficiency. It introduces the VMVM approach for unit test virtualization to isolate tests and minimize overhead, showcasing experiments that compare its performance to traditional methods. The findings reveal considerable speedup in testing while ensuring fault isolation, but also raise concerns about how to address test dependencies safely.
![@_jon_bell_Future of Build Acceleration
Test Independence
• We typically assume that tests are order-
independent
• Might rely on developers to completely reset the
system under test between tests
• Who tests the tests?
• Dangerous: If wrong, can have false positives or
false negatives (Muşlu [FSE ’11], Zhang [ISSTA
’14])](https://image.slidesharecdn.com/futureofbuildacceleration-150910225038-lva1-app6892/85/Test-Dependencies-and-the-Future-of-Build-Acceleration-9-320.jpg)
![@_jon_bell_
Smarter Test Isolation
for Faster Testing
“Unit Test Virtualization with VMVM”
[Bell and Kaiser at ICSE ’14; Distinguished Paper Award]
Forkm
e
on
Github](https://image.slidesharecdn.com/futureofbuildacceleration-150910225038-lva1-app6892/85/Test-Dependencies-and-the-Future-of-Build-Acceleration-12-320.jpg)
![@_jon_bell_Future of Build Acceleration
RQ1: VMVM vs Test
Minimization
• Study design follows Zhang [ISSRE ‘11]’s
evaluation of four minimization approaches
• Compare to the minimization technique with least
impact on fault finding ability, Harrold [TOSEM
‘93]'s technique
• Study performed on the popular Software
Infrastructure Repository dataset](https://image.slidesharecdn.com/futureofbuildacceleration-150910225038-lva1-app6892/85/Test-Dependencies-and-the-Future-of-Build-Acceleration-22-320.jpg)
![@_jon_bell_Future of Build Acceleration
Regression Test Selection
Test 1 Test 2 Test 3
Test 8 Test 9 Test 10
Test 4 Test 5 Test 6 Test 7
Test 11 Test 12 Test 13 Test 14
Gligoric et al. [ISSTA ’15], Orso et al. [FSE ’04], Harrold et al. [OOPSLA ’01]
Changeset
Tests not relevant to changeset: skipped](https://image.slidesharecdn.com/futureofbuildacceleration-150910225038-lva1-app6892/85/Test-Dependencies-and-the-Future-of-Build-Acceleration-33-320.jpg)
![@_jon_bell_Future of Build Acceleration
Test Suite Minimization
Test 1 Test 2 Test 3
Test 8 Test 9 Test 10
Test 4 Test 5 Test 6 Test 7
Test 11 Test 12 Test 13 Test 14
< /> Code
Hao et al. [ICSE ’12]; Orso et al. [ICSE ’09]; Jeffrey et al. [TSE ’07]; Tallam et
al. [PASTE ’05]; Jones et al. [TOSEM ’03]; Harrold et al. [TOSEM ’93]; Chen et
al. [IST ’98]; Wong et al. [ICSE ’95] and more
Redundant tests: removed](https://image.slidesharecdn.com/futureofbuildacceleration-150910225038-lva1-app6892/85/Test-Dependencies-and-the-Future-of-Build-Acceleration-34-320.jpg)
![@_jon_bell_Future of Build Acceleration
Test Dependencies:
A Clear and Present Danger
• Really exist in practice (Zhang et al. found 96, Luo
et al. found 14)
• Hard to specify - if we could specify, would be safe
to accelerate
• Can’t arbitrarily isolate (and it adds overhead!)
• Existing technique to detect: combinatorially run
tests [Zhang, et al ’14]](https://image.slidesharecdn.com/futureofbuildacceleration-150910225038-lva1-app6892/85/Test-Dependencies-and-the-Future-of-Build-Acceleration-41-320.jpg)
Test Dependencies and the Future of Build Acceleration
- 1. Test Dependencies and the Future of Build Acceleration Jonathan Bell (@_jon_bell_) Columbia University
- 2. @_jon_bell_Future of Build Acceleration Simplified Software Lifecycle Make changes to code Build & test Commit How long is too long of a build? 1 day? 6 hours? 10 minutes?
- 3. @_jon_bell_Future of Build Acceleration Simplified Software Lifecycle • Compile sources • Generate documentation • Run tests • Package Make changes to code Build & test Commit
- 4. @_jon_bell_Future of Build Acceleration Testing Dominates Build Times 20% 38% 41% 351 projects from GitHub Testing Other Compiling
- 5. @_jon_bell_Future of Build Acceleration Testing Dominates Build Times 14% 26% 60% Projects taking > 10 minutes to build (69) Testing Other Compiling
- 6. @_jon_bell_Future of Build Acceleration Testing Dominates Build Times 2%8% 90% Projects taking > 1 hour to build (8) Testing OtherCompiling
- 7. Faster tests = Faster builds
- 8. @_jon_bell_Future of Build Acceleration JUnit Test Execution Start JVM Execute Test Terminate App Begin Test Start Test Suite 1.4 sec (combined) For EVERY test!Up to 4,153%, avg 618% Overhead of restarting the JVM? Unit tests as fast as 3-5 ms JVM startup time is fairly constant (1.4 sec) *From our study of 20 popular FOSS apps
- 9. @_jon_bell_Future of Build Acceleration Test Independence • We typically assume that tests are order- independent • Might rely on developers to completely reset the system under test between tests • Who tests the tests? • Dangerous: If wrong, can have false positives or false negatives (Muşlu [FSE ’11], Zhang [ISSTA ’14])
- 10. @_jon_bell_Future of Build Acceleration Test Independence /** If true, cookie values are allowed to contain an equals character without being quoted. */ public static boolean ALLOW_EQUALS_IN_VALUE = Boolean.valueOf(System.getProperty("org.apache.tomcat. util.http.ServerCookie.ALLOW_EQUALS_IN_VALUE","false")) .booleanValue(); This field is set once, when the class that owns it is initialized This field’s value is dependent on an external property
- 11. @_jon_bell_Future of Build Acceleration A Tale of Two Tests TestAllowEqualsInValue TestDontAllowEqualsInValue Sets environmental variable to true Start Tomcat, run test public static boolean ALLOW_EQUALS_IN_VALUE = Boolean.valueOf( System.getProperty(“org.apache.tomcat.util.http.ServerCookie. ALLOW_EQUALS_IN_VALUE","false")).booleanValue(); Sets environmental variable to false Start Tomcat, run test But our static field is stuck! TestAllowEqualsInValue TestDontAllowEqualsInValue
- 12. @_jon_bell_ Smarter Test Isolation for Faster Testing “Unit Test Virtualization with VMVM” [Bell and Kaiser at ICSE ’14; Distinguished Paper Award] Forkm e on Github
- 13. @_jon_bell_Future of Build Acceleration How do Tests Leak Data? Java is memory-managed, and object oriented Test Runner Instance Test Case 1 references Test Case 2 references Accessible Objects references Accessible Objects references Accessible Objects references Test Case n references We think in terms of object graphs No cross-talk No cross-talk
- 14. @_jon_bell_Future of Build Acceleration How do Tests Leak Data? Java is memory-managed, and object oriented We think in terms of object graphs
- 15. @_jon_bell_Future of Build Acceleration How do Tests Leak Data? Java is memory-managed, and object oriented We think in terms of object graphs Class A Static Fields Class B Static Fields Static fields: owned by a class, NOT by an instance These are leakage points references references
- 16. @_jon_bell_Future of Build Acceleration Isolating Side Effects Class A Static Fields Class B Static Fields Class C Static Fields Test 1 Test 2 Writes Reads Reads Static Fields Writes
- 17. @_jon_bell_Future of Build Acceleration Isolating Side Effects Class A Static Fields Class B Static Fields Class C Static Fields Test 1 Test 2 Writes Reads Reads Writes *Interception* Static Fields So, don’t touch them! These classes had no possible conflicts Key Insight: No need to re-initialize the entire application in order to isolate tests
- 18. @_jon_bell_Future of Build Acceleration VMVM: Unit Test Virtualization • Isolates in-memory side effects, just like restarting JVM • Integrates easily with ant, maven, junit • Implemented completely with application byte code instrumentation • No changes to JVM, no access to source code required
- 19. @_jon_bell_Future of Build Acceleration Efficient Reinitialization • Does not require any modifications to the JVM and runs on commodity JVMs • The JVM calls a special method, <clinit> to initialize a class • We do the same, entirely in Java • Add guards to trigger this process • Register a hook with test runner to tell us when a new test starts
- 20. @_jon_bell_Future of Build Acceleration VMVM: Unit Test Virtualization if(CookiesSupport.ALLOW_EQUALS_IN_VALUE) //... else //... if(CookiesSupport.ALLOW_EQUALS_IN_VALUE) //... else //... VMVM adds guards to reinitialize classes if(ShouldReInit(CookiesSupport.class) CookiesSupport.REINIT();
- 21. @_jon_bell_Future of Build Acceleration Experiments • RQ1: How does VMVM compare to Test Suite Minimization? • RQ2: What are the performance gains of VMVM? • RQ3: Does VMVM impact fault finding ability?
- 22. @_jon_bell_Future of Build Acceleration RQ1: VMVM vs Test Minimization • Study design follows Zhang [ISSRE ‘11]’s evaluation of four minimization approaches • Compare to the minimization technique with least impact on fault finding ability, Harrold [TOSEM ‘93]'s technique • Study performed on the popular Software Infrastructure Repository dataset
- 23. @_jon_bell_Future of Build Acceleration 0%! 10%! 20%! 30%! 40%! 50%! 60%! 70%! 80%! 90%! Antv1!Antv2!Antv3!Antv4!Antv5!Antv6!Antv7!Antv8! JM eterv1! JM eterv2! JM eterv3! JM eterv4! JM eterv5! jtopas v1! jtopas v2! jtopas v3! xm l-sec v1! xm l-sec v2! xm l-sec v3! ReductioninTestingTime! Application! Test Suite Minimization! VMVM! Combined! 13% 46% 49% RQ1: VMVM vs Test Minimization Larger is better
- 24. @_jon_bell_Future of Build Acceleration RQ2: Broader Evaluation • Previous study: well-studied suite of 4 projects, which average 37,000 LoC and 51 test classes • This study: manually collected repository of 20 projects, average 475,000 LoC and 56 test classes • Range from 5,000 LoC - 5,692,450 LoC; 3 - 292 test classes; 3.5-15 years in age
- 25. @_jon_bell_Future of Build Acceleration RQ2: Broader Evaluation 0%! 20%! 40%! 60%! 80%! 100%! upm! JTor! Openfire! Trove for Java! FreeRapid Downloader! JAXX! Commons Validator! Commons Codec! Closure Compiler! betterFORM! Apache Ivy! mkgmap! gedcom4j! btrace! Apache River! Commons IO! Jetty! Apache Tomcat! Apache Nutch! Bristlecone! Relative Speedup! Max: 97% Average: 62% Larger is better
- 26. @_jon_bell_Future of Build Acceleration Factors that impact reduction • Looked for relationships between number of tests, lines of code, age of project, total testing time, time per test, and VMVM’s speedup • Result: Only average time per test is correlated with VMVM’s speedup (in fact, quite strongly; p < 0.0001)
- 27. @_jon_bell_Future of Build Acceleration RQ3: Impact on Fault Finding • No impact on fault finding from seeded faults (SIR) • Does VMVM correctly isolate tests though? • Compared false positives and negatives between un- isolated execution, traditionally isolated execution, and VMVM-isolated execution for these 20 complex applications • Result: False positives occur when not isolated. VMVM shows no false positives or false negatives.
- 28. @_jon_bell_ How do we make it faster? Java VMVM Unit Tests
- 29. @_jon_bell_ How do we make it faster? Java VMVM Unit Tests
- 30. @_jon_bell_Future of Build Acceleration Testing is Embarrassingly Parallel Project Raw ,me (minutes) 8 Worker Speedup 24 Worker Speedup Internal CI 20.50 2.5x 1.8x Mule ESB 150.92 6.4x 10.9x Jenkins 2.33 2.2x 2.3x OpenWebBeans 0.54 1.9x 2.1x Cut from 2.5 hours to 14 minutes
- 31. @_jon_bell_Future of Build Acceleration Feedback from Developers about VMVM • “It’s great! It cuts our 45 minute tests in half!” • “It’s useless! We don’t isolate our tests! Our tests take 24 hours so isolating them would make them take days!” • Remember: Although our study showed many isolate their tests, not all do!
- 32. What happens if you don’t isolate?
- 33. @_jon_bell_Future of Build Acceleration Regression Test Selection Test 1 Test 2 Test 3 Test 8 Test 9 Test 10 Test 4 Test 5 Test 6 Test 7 Test 11 Test 12 Test 13 Test 14 Gligoric et al. [ISSTA ’15], Orso et al. [FSE ’04], Harrold et al. [OOPSLA ’01] Changeset Tests not relevant to changeset: skipped
- 34. @_jon_bell_Future of Build Acceleration Test Suite Minimization Test 1 Test 2 Test 3 Test 8 Test 9 Test 10 Test 4 Test 5 Test 6 Test 7 Test 11 Test 12 Test 13 Test 14 < /> Code Hao et al. [ICSE ’12]; Orso et al. [ICSE ’09]; Jeffrey et al. [TSE ’07]; Tallam et al. [PASTE ’05]; Jones et al. [TOSEM ’03]; Harrold et al. [TOSEM ’93]; Chen et al. [IST ’98]; Wong et al. [ICSE ’95] and more Redundant tests: removed
- 35. @_jon_bell_Future of Build Acceleration Test Parallelization Test 1 Test 2 Test 3 Test 8 Test 9 Test 10 Test 4 Test 5 Test 6 Test 7 Test 11 Test 12 Test 13 Test 14
- 36. @_jon_bell_Future of Build Acceleration Test Parallelization Test 1 Test 2 Test 3 Test 8 Test 9 Test 10 Test 4 Test 5 Test 6 Test 7 Test 11 Test 12 Test 13 Test 14
- 37. @_jon_bell_Future of Build Acceleration Controlled Regression Testing Assumption Tests </> Code External Factors External Factors External Factors External FactorsExternal Factors External Factors External Factors External Factors Test 1Test 2Test 3
- 38. Not sound in practice
- 39. @_jon_bell_Future of Build Acceleration Test Dependencies Test 1 Test 2 Test 3 Test 4Test 1 Test 2 Shared File Value: A Write, Value “A” Test 4 Read Write, Value “B” Value: B Test 3 Read
- 40. @_jon_bell_Future of Build Acceleration Test Dependencies Test 1 Test 2 Test 3Test 4Test 1 Test 2 Test 3 Shared File Value: A Write, Value “A” Test 4 Write, Value “B” Read, Expect Value “A” Value: B A manifest test dependency Read
- 41. @_jon_bell_Future of Build Acceleration Test Dependencies: A Clear and Present Danger • Really exist in practice (Zhang et al. found 96, Luo et al. found 14) • Hard to specify - if we could specify, would be safe to accelerate • Can’t arbitrarily isolate (and it adds overhead!) • Existing technique to detect: combinatorially run tests [Zhang, et al ’14]
- 42. @_jon_bell_Future of Build Acceleration Brute Force Dependency Detection Test 1 Test 2 Test 3 Test 4Test 1 Test 2 Test 4Test 3
- 43. @_jon_bell_Future of Build Acceleration Brute Force Dependency Detection Test 1 Test 2 Test 4 Test 3Test 1 Test 2 Test 3Test 4
- 44. @_jon_bell_Future of Build Acceleration Brute Force Dependency Detection Test 2 Test 1 Test 3 Test 4Test 2 Test 1 Test 4Test 3
- 45. @_jon_bell_Future of Build Acceleration Brute Force Dependency Detection Test 4 Test 2 Test 3 Test 1Test 4 Test 2 Test 1Test 3
- 46. @_jon_bell_Future of Build Acceleration Brute Force Dependency Detection Test 1 Test 3 Test 2 Test 4Test 1 Test 3 Test 4Test 2
- 47. @_jon_bell_Future of Build Acceleration Brute Force Dependency Detection • Looked at feasibility on 10 large open source test suites • Exhaustive approach: > 10300 years to find all dependencies • Pairwise approach: Average 31,882 executions of the entire test suite to find (incomplete) dependencies • Problem: How do we safely accelerate test suites in the presence of unknown dependencies?
- 48. @_jon_bell_Future of Build Acceleration Manifest Test Dependencies • Definition: a data dependence between tests T1, T2 that results in the outcome of T2 changing • All manifest dependencies are data dependencies • Not all data dependencies are manifest dependencies
- 49. @_jon_bell_Future of Build Acceleration Data Dependencies Test 1 Test 2 Test 3 Test 4Test 1 Test 2 Shared File Write, Value “A” Test 4 Read Write, Value “B” Test 3 Read Present Dependencies: Test 1 must run before 2 and 3 Test 4 must run after 2 and 3
- 50. Key Insight: Dependencies don’t need to be precise, but must be sound
- 51. @_jon_bell_Future of Build Acceleration Intuition Test 1 Test 2 Test 3 Test 8 Test 9 Test 10 Test 15 Test 4 Test 5 Test 6 Test 7 Test 11 Test 12 Test 13 Test 14 Test 1 Test 2 Test 3 Test 4 Test 5 Test 6 Test 7 Test 8 Test 9 Test 10 Test 11 Test 12 Test 13 Test 14 Test 15 Idle extra capacity
- 52. @_jon_bell_Future of Build Acceleration Intuition Test 1 Test 2 Test 3 Test 8 Test 9 Test 10 Test 15 Test 4 Test 5 Test 6 Test 7 Test 11 Test 12 Test 13 Test 14 Test 1 Test 2 Test 3 Test 8 Test 9 Test 10 Test 15 Test 4 Test 5 Test 6 Test 7 Test 11 Test 12 Test 13 Test 14 Idle extra capacity A lot of dependencies, but still a 2x speedup
- 53. @_jon_bell_Future of Build Acceleration Efficient Dependency Detection for Safe Java Test Acceleration Jonathan Bell, Gail Kaiser, Eric Melski and Mohan Dattatreya Columbia University & Electric Cloud, Inc
- 54. @_jon_bell_Future of Build Acceleration ElectricTest - Detecting Data Dependencies in Java • Tracks in-memory dependencies (JVMTI plugin) • Tracks file and network dependencies (IO-Trace agent) • Implemented entirely within the Oracle or OpenJDK JVM, no specialized drivers, etc required • Captures stack traces when dependencies occur to support debugging • Generates dependency trees to enable sound test acceleration
- 55. @_jon_bell_Future of Build Acceleration Identifying Heap Dependencies After each test, garbage collect; traverse heap to map objects back to static fields. Class A W1 W1 W1 W1 W1 W1 W1 W1 W1 static field static field static field staticfield End of test 1
- 56. @_jon_bell_Future of Build Acceleration Identifying Heap Dependencies During test execution, monitor accesses to existing objects Class A W1 W1 W1 W1 W1 W1 W1 W1 W1 static field static field static field staticfield W2 W2 W1 Write! Write! Read! During Test 2 Dependency!
- 57. @_jon_bell_Future of Build Acceleration Identifying External Dependencies Application under test Network Filesystem Log remote host address Log path
- 58. ElectricTest enables sound exploitation of existing test acceleration techniques
- 59. @_jon_bell_Future of Build Acceleration Safe Test Parallelization Test 1 Test 2 Test 3 Test 8 Test 9 Test 10 Test 15 Test 4 Test 5 Test 6 Test 7 Test 11 Test 12 Test 13 Test 14 Test 1 Test 2 Test 3 Test 4 Test 5 Test 6 Test 7 Test 8 Test 9 Test 10 Test 11 Test 12 Test 13 Test 14 Test 15
- 60. @_jon_bell_Future of Build Acceleration Safe Test Parallelization Test 1 Test 2 Test 3 Test 8 Test 9 Test 10 Test 15 Test 4 Test 5 Test 6 Test 7 Test 11 Test 12 Test 13Test 14 Test 1 Test 2 Test 3 Test 8 Test 9 Test 10 Test 15 Test 4 Test 5 Test 6 Test 7 Test 11 Test 12 Test 13Test 14
- 61. @_jon_bell_Future of Build Acceleration Safe Test Selection Test 15 Single test selected to be executed
- 62. @_jon_bell_Future of Build Acceleration Safe Test Selection Test 15Test 1 Test 2 Test 3 Single test selected to be executed with its dependencies
- 63. @_jon_bell_Future of Build Acceleration Understanding Dependencies • What should a developer do about test dependencies? • Might be intentional (e.g. cache shared state) • Might be unintentional but OK (e.g. loggers) • Might be unintentional and bad (e.g. bug)
- 64. @_jon_bell_Future of Build Acceleration Assisting Debugging Debugging information reported by the previous technique Test 3 Test 1 Depends on
- 65. @_jon_bell_Future of Build Acceleration Assisting Debugging Exception in thread "main" edu.columbia.cs.psl.testdepends.DependencyException: Static Field ClassA.FieldA member was previously written by Test 1, read here. at edu.columbia.cs.psl.testdepends.test.Example$NestedExample.dragons(Example.java:20) at edu.columbia.cs.psl.testdepends.test.Example.moreMagic(Example.java:12) at edu.columbia.cs.psl.testdepends.test.Example.magic(Example.java:8) at edu.columbia.cs.psl.testdepends.test.Example.main(Example.java:15) Really helpful Test that wrote value Stack trace shows use Value that is read
- 66. @_jon_bell_Future of Build Acceleration Evaluation • RQ1: Recall (accuracy) • RQ2: Runtime overhead • RQ3: Impact on acceleration
- 67. @_jon_bell_Future of Build Acceleration RQ1: Recall Dependencies Detected ElectricTest Shared Ground Truth ElectricTest Resource Locations Project Writers Readers App Library Joda 2 15 121 39 12 XMLSecurity 4 3 103 3 15 Crystal 18 15 39 4 19 Synoptic 1 10 117 3 14
- 68. @_jon_bell_Future of Build Acceleration RQ2: Overhead • Selected 10 projects with > 10 minutes of tests • Also included projects studied by Zhang et al, averaging < 10 seconds of testing • Previous exhaustive approach slowdown: >10300X • Previous heuristic approach slowdown: 31,882X • ElectricTest slowdown: 36X (885X faster than previous approach)
- 69. @_jon_bell_Future of Build Acceleration 0X 1,000X 2,000X 3,000X 4,000X 5,000X 6,000X 7,000X 8,000X 9,000X 10,000X mongo%java%driver- tachyon- spring%data%mongodb- xml-security- ne8y- je8y.project- crystal- crunch- camel- :tan- synop:c- hazelcast- mule- joda%:me- ElectricTest Slowdown Pairwise Slowdown *418,000X RQ2: Overhead On average, ElectricTest is 885X faster than running all tests pairwise Slowdown relative to a single test suite execution (lower is better)
- 70. @_jon_bell_Future of Build Acceleration 0X 50X 100X 150X 200X 250X 300X mongo%java%driver- tachyon- spring%data%mongodb- xml-security- ne8y- je8y.project- crystal- crunch- camel- :tan- synop:c- hazelcast- mule- joda%:me- RQ2: Overhead Average 36X A lot of fast running tests: Runtime dominated by pauses between tests (gc) Slowdown relative to a single test suite execution (lower is better)
- 71. @_jon_bell_Future of Build Acceleration 0X 5X 10X 15X 20X 25X 30X camel& crunch& hazelcast& je/y.project& mongo5java5driver& mule& ne/y& spring5data5mongodb& tachyon& :tan& Safe Unsafe Speedup (higher is better) RQ3: Impact on Acceleration Average (Unsafe) 19x Average (Safe) 7x
- 72. @_jon_bell_Future of Build Acceleration
- 73. Test Dependencies and the Future of Build Acceleration Jonathan Bell Columbia University jbell@cs.columbia.edu http://jonbell.net/