LCE13: Test and Validation Summit: The future of testing at Linaro
1 like3,347 views
The Linaro Test and Validation Summit held in Dublin in July 2013 focused on improving testing practices within Linaro engineering teams. Discussions included the use of Continuous Integration (CI), automated testing frameworks like Jenkins and LAVA, and challenges in validating outputs across hardware platforms. Key areas for enhancement identified were better documentation, more infrastructure for testing and analysis, and the necessity of integrating tests effectively into workflows.
![● "Master image" is not useful
○ LAVA assumes are powered on and running master
image (or will reboot into master image)
○ assumptions about SD card existence, partitioning...
○ assumptions about shell prompts
linaro-test [rc=0] #
○ etc. etc.
● Goal: LAVA directly controls bootloader
○ netboot: get kernel + DTB + initrd via TFTP
○ extension via board-specific bootloader scripting
Tyler's new "bootloader" device support in LAVA
appears to have mostly solved this !!
Issues: LAVA design](https://image.slidesharecdn.com/03summitsessions3and4final1-140321073854-phpapp02/85/LCE13-Test-and-Validation-Summit-The-future-of-testing-at-Linaro-33-320.jpg)
LCE13: Test and Validation Summit: The future of testing at Linaro
- 1. Linaro Test and Validation Summit Linaro Engineering Teams LCE13 - Dublin, July 2013 How Do We Better Test our Engineering 2nd Half
- 2. PART 1: Linaro Platform Overview LAVA Citius,Altius,Fortius "Faster,Higher,Stronger" Builds&CI BuildYourCodeWhenYou areNotLooking QAServices Coverallbases PART 2: Linaro Engineering KernelDevelopers &Maintainers LandingTeams LinaroGroups LEG/LNG (DotheyuseCI? Manual orAutomatedTesting?)*Howdotheyvalidate/verifytheiroutput? *What/howdotheydevelop? &easiertouse
- 3. Linaro Test and Validation Summit Mike Holmes LCE13 - Dublin, July 2013 LNG Engineering
- 4. ● Create: Two Linux kernels (with and without RT) and a yocto filesystem. ● Benchmark: RT + KVM + HugePage + Dataplane APIs Required to test kernel and userspace performance, some tests may be run in both spaces. ● Platforms: Arndale, AM335x Starter Kit ? (LSI & TI boards in future ?) QEMU - versatile Express ? LNG outputs
- 5. ● Our code is validated using CI, and performance trends monitored. ● Our output is verified on one general purpose ARM platform and against two SoC vendor platforms via a configurable switch to allow for dedicated links between nodes under test. ● Using open source software, we use one realistic network application, a general purpose benchmark and five feature specific test suites. LNG outputs are verified by
- 6. Automated testing is done using ● custom scripts run via jenkins & lava executing ○ (RT) LTP (Real Time Test Tree) ○ (RT) Cyclictest ○ (RT) Hackbench ○ (KVM) virt-test ○ (Hugepage) sysbench OLTP ○ (KVM, Hugepage, RT) openvswitch (Kernel and userspace) ○ (KVM, Hugepage, RT) netperf ○ Traffic test cases via pcap files and tcpreplay LNG uses these tools
- 7. ● We test against three branches ○ linux-lng-tip (development) ○ linux-lng-lsk (bug fixes to stable) ○ linux-lng-lsk-RT (bug fixes to stable RT variant) ● LNG specific CFG fragments ○ KVM (or will this be in a lsk kernel per default?) ○ PREEMPT_RT ○ NO_HZ_FULL (or will this be in a lsk kernel per default?) ○ HUGEPAGE (is that a CFG option?) LNG Kernel branches / configuration
- 8. ● Some of the SoC vendors hardware has up to 16 x 10Gb links, generating this much traffic is non trivial. ● Tests equipment such as IXIA traffic generators are expensive. ● Test equipment needs to be remotely switched between the different hardware under test in an automated way ● Scheduling test runs that take days and requires specific equipment to be dedicated to the task. LNG unique challenges
- 9. ● Multiple nodes may be needed to test traffic interoperability. ● It is not feasible to replicate the test environment at every developer's desk. ● the applied RT patch even when disabled, alters the execution paths ● Some test run for 24 hours or more LNG unique challenges
- 10. Questions ○ LAVA is(isn't) working for us ■ Interactive shells in the LAVA environment would speed debugging given that testing can only be performed with the test equipment in the lab ■ Multinode testing, with the reservation and configuration of network switches is required. ■ Long term trends in performance data need to analysed and compared for regression analysis, triggering alerts for deviations. ○ Further thoughts on Friday ○ https://lce-13.zerista. com/event/member/79674 LNG Q&A
- 11. Linaro Test and Validation Summit Scott Bambrough LCE13 - Dublin, July 2013 Landing Team Engineering
- 12. ● Bootloaders ● Linux kernels based on mainline or current RC's ● Linux kernels based on LSK (expected) ● Ubuntu member builds ● Android member builds ● ALIP member build Some outputs are public, others confidential. LT Outputs
- 13. ● Kernel code is validated using CI in the Linaro LAVA Lab, on various member hardware devices and ARM fast models. ● Our kernel code is also validated in member LAVA labs on both current and next gen hardware. ● Our builds at present are a sanity tested by the LT's but most testing is done by piggybacking on QA or automated testing set up by the platform team. Verification of LT outputs
- 14. ● Currently run only basic compile boot test + default CI tests (LTP, powermgmt) ● This needs to change, we want/need to do more ● We need more SoC level tests, having LT's aware of how to produce tests to run in LAVA will become more important LT and kernel tests
- 15. 1. Much better LAVA documentation 2. Document the tests themselves 3. Infrastructure for testing 4. Infrastructure for better analysis of results LT & Member Services Needs
- 16. ● Deployment Guide ○ what are the hardware requirements for a LAB ○ what are the infrastructure requirements for a LAB ○ hardware setup, software installation instructions ● Administrator's Guide ○ basically how Dave Piggot does his job ○ after initial setup, day to day ops and maintenance Better Documentation
- 17. ● Test Developer's Guide ○ how to integrate tests to be run in lava-test-shell (lava glue) ○ recommendations on how best to write tests for lava- test-shell ● User's Guide for lava-test-shell ○ for developers to use lava-test-shell ○ section devoted to using lava-test-shell in workflow of kernel developer? Better Documentation
- 18. ● Impossible to answer the question: What tests are available in LAVA? ● http://lava-test.readthedocs.org/en/latest/index.html ○ not sufficient, not up to date ○ problem isn't LAVA team, Linaro needs an acceptance policy on what a test has available before being used in LAVA ● would like to see meta-data in test documentation that can be used in test reports ○ in a format that can be used in report generation Document the tests
- 19. ● Buddy systems ○ TI LT developed tests that require access to reference material for comparison ■ video frame captures ■ audio filed ○ TI LT audio/video tests required external box to capture hdmi/audio output ○ Need to do more of this type of automated testing to verify that lower level functions work correctly at BSP level ○ GStreamer insanity test suite requires access to multimedia content Infrastructure for Testing
- 20. ● Web dashboard won't cut it ● need to separate analysis from display ○ rather do an analysis, then decide how to display ● why infrastructure? ○ think there should be a level of reuse for components used to do analysis ○ think these should be separate from LAVA ○ think of this a more of a data mining operation Infrastructure for Analysis
- 21. example: ● generate test report as PDF ○ perform tests, generate a report ○ include metadata regarding tests ■ metadata from test documentation? example: ● test report comparing: ○ current member BSP kernel ○ current LT kernel based on mainline ● evidence of quality/stability of LT/mainline kernel ● could be used to convince product teams Infrastructure for Analysis
- 22. example: ● regression analysis of kernel changes ○ perform tests one day, make changes, test next ○ did any test results change? ■ yes, send report of changes via email example: ● generate test report as PDF ○ perform tests, generate a report ○ include metadata regarding tests ■ metadata from test documentation? Infrastructure for Analysis
- 23. example: ● test report comparing: ○ current member BSP kernel ○ current LT kernel based on mainline ● evidence of quality/stability of LT/mainline kernel ● could be used to convince product teams Infrastructure for Analysis
- 24. Linaro Test and Validation Summit Kevin Hilman LCE13 - Dublin, July 2013 Kernel Developer/Maintainer
- 25. Most kernel development is done with little or no automation ● build: local, custom build scripts ● boot: manual boot testing on local hardware ● debug: custom unit-test scripts, manual verification of results ● publish: to public mailing lists ● merged: into maintainer trees, linux-next ● test: manual test of maintainer trees, linux-next ○ but many (most?) developers don't do this Current workflow: development
- 26. ● Code review on mailing list ● build/boot testing by maintainers ● build testing in linux-next (manual) ○ several developers do manual build tests of their pet platforms in linux-next and report failures ● Intel's 0-day tester (automated, but closed) ○ regular, automatic build tests ○ multi-arch build tests ○ boot tests (x86) ○ automatic git bisect for failures ○ very fast results ○ detailed email reports ○ extremely useful Current workflow: validation
- 27. This model is "good enough" for most developers and maintainers, so... Why should we use Jenkins/LAVA? Linaro test/validation will have to be ● at least as easy to use (locally and remotely) ● output/results more useful ● faster ○ build time ○ diagnostic time Current workflow: "good enough"
- 28. ● Local testing: aid in build, boot, test cycle ○ local LAVA install, using local boards ○ reduce duplication of custom scripts/setup ○ encourage writing LAVA-ready tests ○ easy to switch between local, and remote LAVA lab ● Remote CI: broader coverage ○ "I'm about ready to push this, I wonder if broke any other platforms..." ○ automatic, fast (ish) response Potential Usage models
- 29. ● Has to be easy to install ○ packaged (deb, rpm) ○ or git repo for development (bzr is ......) ● Has to fit into existing developer work flow ○ LAVA does not exclusively own hardware ○ developers have non-Linaro platforms ○ command-line driven ○ must co-exist with existing interactive use of boards ■ existing Apache setup ■ existing TFTP setup ■ existing, customized bootloaders ■ ... Local testing: LAVA
- 30. ● Broad testing ● multi-arch (not just ARM) ● ARM: all defconfigs (not just Linaro boards) ○ also: allnoconfig, allmodconfig, randconfig, ... ● Continuous builds ○ Linus' tree, linux-next, arm-soc/for-next, ... ○ developers can submit their own branches ● On-demand builds ○ register a tree/branch ○ push triggers a build ● fast, automatic reporting of failures ○ without manual monitoring/clicking through jenkins Remote CI
- 31. Tracking build breakage in upstream trees ● when did build start breaking ● what are the exact build error messages (without Jenkins click fest) ● which commit (probably) broke the build ○ automated bisect Useful output: build testing
- 32. Where is the line between Jenkins and LAVA? ● Jenkins == build, LAVA == test? ● when a LAVA test fails how do I know... ○ was this a new/updated test? ○ was this a new/updated kernel? ○ if so, can I get to the Jenkins build? In less than 10 clicks? Issues: Big picture
- 33. ● "Master image" is not useful ○ LAVA assumes are powered on and running master image (or will reboot into master image) ○ assumptions about SD card existence, partitioning... ○ assumptions about shell prompts linaro-test [rc=0] # ○ etc. etc. ● Goal: LAVA directly controls bootloader ○ netboot: get kernel + DTB + initrd via TFTP ○ extension via board-specific bootloader scripting Tyler's new "bootloader" device support in LAVA appears to have mostly solved this !! Issues: LAVA design
- 34. ● Terminology learning curve ○ dispatcher, scheduler, dashboard ○ device, device-type ○ What is a bundle? ○ WTF is a bundle stream? ○ Documentation... not helpful (enough said) ● Navigation ○ click intensive ○ how to get from a log to the test results? or... ○ from a test back to the boot log? ○ what about build log (Jenkins?) ○ can I navigate from Jenkins log to the LAVA test? Issues: LAVA usability
- 35. Kernel + modules: omap2plus_defconfig ● 1 minute ○ hackbox.linaro.org (-j48: 12 x 3.5GHz Xeon, 24G) ● 1.5 minutes ○ khilman local (-j24: 6 x 3.3GHz i7, 16G RAM) ● 8 minutes ○ Macbook Air (-j8: 2 x 1.8GHz i7, 4G) ● 14 minutes ○ Thinkpad T61 (-j4: 2 x Core2Duo, 4G RAM) ● 16 minutes ○ Linaro Jenkins (-j8: EC2 node, built in tmpfs) ● 17 minutes ○ ARM chromebook (-j4: 2 x 1.7 GHz A15, 2G RAM) Issues: Jenkins performance
- 36. Linaro Test and Validation Summit Grant Likely LCE13 - Dublin, July 2013 LEG Engineering