![INITIAL DL EXPERIMENTS
• Preprocess testOutput based on own listed feature key words index.
34
Padding preprocessed testOutput for Deep Learning
model:
“[4, 7, 10, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0…]”
Label: “1”
Original testOutput:
“Running test TestIBMJlmLocal_0: ERROR code 3, FAILED test 1,
running second test, exception ValueError, exit code 1”
Label: “FAILED”
Preprocessed testOutput:
“[4,
Label:
7, 10]”
“1”
Dealing with Verification Data Overload](https://image.slidesharecdn.com/dealingwithverificationdataoverloadtestingunited2019-200110005033/85/DealingwithVerificationDataOverload-34-320.jpg)
DealingwithVerificationDataOverload
- 1. SHELLEY LAMBERT LAN XIA IBM RUNTIME TECHNOLOGIES NOV 2019 @SHELLEYMLAMBERT @BONJOURLAN DEALING WITH VERIFICATION DATA OVERLOAD Dealing with Verification Data Overload
- 2. • THE SCOPE • TEST FRAMEWORK (TESTKITGEN) • JENKINS BUILDS • TEST RESULT SUMMARY SERVICE (TRSS) • DATA REFINERY EXPERIMENTS • PLANS FORWARD AGENDA Dealing with Verification Data Overload
- 3. THE SCOPE AdoptOpenJDK • ENSURING FREE AND VERIFIED JAVA™ FOR THE COMMUNITY • PROJECTS: ECLIPSE OMR, ECLIPSE OPENJ9, ADOPTOPENJDK • 6+ JENKINS SERVERS Dealing with Verification Data Overload
- 4. Dealing with Verification Data Overload
- 5. DEGREES OF FREEDOM OpenJ9 Hotspot SAP Corretto 8 9 10 11 12 + openjdk functional perf system external RI 13 JDK Implementations Platforms JDK Versions Test Categories osx osx aix win xlinux plinux 6 versions aarch 58 impl_platform 250000 tests 87,000,000 Tests Impl_platform x testLevels x testGroups x versions 58 x 2 x 3 x 6 x 10M = 20G+ test output per nightly build Plus PR builds, promotion builds and personal builds Dealing with Verification Data Overload
- 6. GATHER GREAT TESTS 6 functional openjdk perfexternalsystem Dealing with Verification Data Overload
- 7. GATHER GREAT TESTS 7 functional openjdk perfexternalsystem testNG, cmdlinetester STF junit & others Assorted benchmarks Jtreg, testNG Dealing with Verification Data Overload
- 8. ADOPTOPENJDK QUALITY ASSURANCE (AQA) 8Dealing with Verification Data Overload • “Make quality certain to happen” • Testing a wide criteria representing actual business requirements to identify binaries ready for production usage Today Roadmap Functional correctness Security OpenJDK regression (open) Passes known vulnerability tests Oracle JCK (closed) Functional correctness OpenJDK regression Builder-specific testing (unknown) Eclipse functional Application & framework tests Performance Published metrics Achieves minimum throughput scores Scalability & durability Load & stress testing
- 9. AQA MANIFESTO • open & transparent • diverse & robust set of test suites • evolution alongside implementations • continual investment • process to modify • codecov & other metrics • comparative analysis • portable • tag & publish 9Dealing with Verification Data Overload
- 10. INNOVATE AND COLLABORATE • Reactive systems • Latitude • Flexible • Common • Standardized • Simple 10Dealing with Verification Data Overload
- 11. GRANULARITY • Specific Testcases/groups • Different levels • Different versions • Different implementations • Different iterations • Different features • With/Without native test image • … 11 sanity/extended/special 8/11/13/14… openj9/hotspot/ibm/corretto/sap 1/2/n… AOT/JITAAS functional/system/openjdk/perf/external Dealing with Verification Data Overload
- 12. CONSOLIDATE AND CURATE 12 functional openjdk perfexternalsystem TestKitGen testNG, cmdlinetester STF junit & others Assorted benchmarks Jtreg, testNG Dealing with Verification Data Overload
- 13. TEST FRAMEWORK (TESTKITGEN) 13Dealing with Verification Data Overload
- 14. GROUPING & GRANULARITY • group=openjdk • levels=sanity.openjdk, extended.openjdk, special.openjdk • targets=tests in playlist file • jdk_awt, jdk_math, jdk_lang, etc. • jdk_custom=CUSTOM_TARGET env var • set to individual directories or classes 14 openjdk sanity.openjdk jdk_math Java/math/BigDecimal/NegateTests.java Dealing with Verification Data Overload
- 15. ADOPTOPENJDK CI PIPELINE Run in parallel Build Deploy openjdk functional system perf external Test sanity.system lambdaLoadTest extended.system special.system mathLoadTest … daaLoadTest Dealing with Verification Data Overload
- 16. JENKINS SCRIPTS FOR TESTING 16 • Repo: https://github.com/AdoptOpenJDK/openjdk-tests • One script (JenkinsfileBase) for all test builds: • Nightly/release • Pull Request • Promotion • Personal/Grinder Dealing with Verification Data Overload
- 17. SURVEY OF TESTS: CI.ADOPTOPENJDK.NET 17 Categorize test builds based on JDK Version, JDK Impl, test category and platform Dealing with Verification Data Overload
- 18. TAP & JUNIT PLUGIN 18 Standardize output Dealing with Verification Data Overload
- 19. ARCHIVE DATA • Archive test data from failed tests onto Jenkins master or Artifactory: • Test logs/output files • Diagnostic files (core/trace/javacore files) • TAP file, JUnit *.jtr, *.xml files • Test repo SHA 19 Minimize stored artifacts Dealing with Verification Data Overload
- 20. ADOPTOPENJDK TESTING vm docker physical machine Jenkins Server artifactory How to monitor the result??? Dealing with Verification Data Overload
- 21. TEST RESULT SUMMARY SERVICE (TRSS) • Monitors multiple Jenkins servers • Personalized Dashboard • Provide filtering, sorting, comparing and searching feature • Provide history for triaging and performance trends • Available at https://trss.adoptopenjdk.net/ • Git repo: https://github.com/AdoptOpenJDK/openjdk-test-tools 21Dealing with Verification Data Overload
- 22. TRSS OVERVIEW 22 frontend backend backend backend TRSS Server Client Multiple Jenkins Servers MongoDB Dealing with Verification Data Overload
- 23. TRSS: PERSONALIZED DASHBOARD 23 Personalize views, only what you need Dealing with Verification Data Overload
- 24. TRSS: PERSONALIZED DASHBOARD 24Dealing with Verification Data Overload Grid view for test builds summary
- 25. TRSS: MONITOR JENKINS PIPELINE BUILDS 25 Aggregate results from top-level pipelines Dealing with Verification Data Overload
- 26. TRSS: TEST BUILDS RESULT 26 Test Result Summary Dealing with Verification Data Overload
- 27. TRSS: TESTS RESULT 27 Filter and sort on test name, duration, and test results Dealing with Verification Data Overload
- 28. TRSS: TEST RESULT ACROSS ALL BUILDS 28 Compare test results for all JDK impls, JDK versions and platforms Dealing with Verification Data Overload
- 29. TRSS: SEARCH TEST 29Dealing with Verification Data Overload Search test output among different test builds
- 30. TRSS: TEST COMPARE 30 Diff test results among two different builds Dealing with Verification Data Overload
- 31. TRSS: PERF DASHBOARD 31 Visualize data graphically Dealing with Verification Data Overload
- 32. LET US COUNT THE WAYS • categorize • standardize • minimize • personalize • aggregate • summarize • filter • sort • compare • search • diff • visualize • model Dealing with Verification Data Overload
- 33. WHAT IS DEEP LEARNING? Deep learning is a subset of ML algorithms distinguished by: • Loosely based on structure and function of the brain, use artificial neural networks (ANN) • Multiple layers of processing units, “neurons”, output of a layer is input to another layer • Modes of learning, supervised (regression, classification) or unsupervised (pattern analysis) 33Dealing with Verification Data Overload
- 34. INITIAL DL EXPERIMENTS • Preprocess testOutput based on own listed feature key words index. 34 Padding preprocessed testOutput for Deep Learning model: “[4, 7, 10, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0…]” Label: “1” Original testOutput: “Running test TestIBMJlmLocal_0: ERROR code 3, FAILED test 1, running second test, exception ValueError, exit code 1” Label: “FAILED” Preprocessed testOutput: “[4, Label: 7, 10]” “1” Dealing with Verification Data Overload
- 35. INITIAL DL EXPERIMENTS - Training Accuracy and Validation Accuracy, our evaluation result is 98.897% with 7800 training data (3900 failed and 3900 passed) and 7800 test data - Training Loss and Validation Loss 35Dealing with Verification Data Overload
- 36. MODEL BUILDING 36 JVM version Variants used Things we know (input layer) Failure expression Platform JVM Impl Machine ‘age’ Failure age PR list Defect category Things we want to know (output layer) Bug prediction scores Best next action Rate value of test SHAs Dealing with Verification Data Overload
- 37. Deep Learning for Fuzzing Java Compilers 38Dealing with Verification Data Overload
- 38. Deep Learning for Fuzzing Java Compilers 39Dealing with Verification Data Overload • Integrated DeepSmith with TKG and TRSS • Easily run thousands of DeepSmith tests in Jenkins with different JDK versions / impls / JVM options • Compare and monitor test outputs using TRSS
- 39. PLANS FORWARD • Test smarter (as test volume increases) • Change-based testing • Bug prediction service • Enhance TRSS with analytics services • Build skills and continue model/deploy, observe & measure • Innovate/Collaborate • AI-driven fuzz testing (with Professor Hugh Leather) • Test generation service (application of CTD) • Leverage & deploy useful models in open projects 40Dealing with Verification Data Overload
- 40. @ShelleyMLambert @bonjourlan AdoptOpenJDK: https://adoptopenjdk.net/ Git repo: https://github.com/AdoptOpenJDK/openjdk-tests https://github.com/AdoptOpenJDK/openjdk-test-tools https://github.com/AdoptOpenJDK/TKG https://github.com/eclipse/openj9 THANK YOU! Dealing with Verification Data Overload
- 41. QUESTIONS ?
Editor's Notes
- #8: - test frameworks are different - no summary page
- #9: - Test source open, executed in the open, results openly available - Cover different test categories (functional/regression, security, performance, scalability, load/stress), implementations, versions, platforms - Evolution: - refine automation and tools - remove friction and reduce the process - review existing/new tests on regular basis - community awareness of addition/modification/deletion - codecov, heatmap and bug prediction - Easily run on local and Jenkins - Audit trail
- #10: - Test source open, executed in the open, results openly available - Cover different test categories (functional/regression, security, performance, scalability, load/stress), implementations, versions, platforms - Evolution: - refine automation and tools - remove friction and reduce the process - review existing/new tests on regular basis - community awareness of addition/modification/deletion - codecov, heatmap and bug prediction - Easily run on local and Jenkins - Audit trail
- #11: We need a reactive system which works well in a changing world. latitude to cover tests that may use significant different test frameworks flexible to specify and test against different requirements a common way to easily add, group, exclude and execute tests test results to have same look & feel Simple to be integrated with any build system
- #12: Slice and dice Different levels of granularity
- #13: Group, categorize, tag
- #19: Leverage 3rd party Jenkins plugins
- #20: Artifactory open source project binary repositories
- #35: TensorFlow Keras Sequential model https://keras.io/models/sequential/ https://www.tensorflow.org/tutorials/keras