LCE13: Test and Validation Summit: Evolution of Testing in Linaro (II)
0 likes3,016 views
This document summarizes discussions from the Linaro Test and Validation Summit in Dublin regarding improving engineering testing at Linaro. It describes Linaro's continuous integration (CI) process and tools like LAVA, and plans to enhance testing capabilities. Key points include: 1. Fathi Boudra provided an overview of Linaro's CI loop using tools like Jenkins and LAVA, and future plans to improve the CI process and LAVA. 2. Milosz Wasilewski discussed QA services tasks like manual testing and dashboard monitoring, and ideas to streamline these processes. 3. Antonio Terceiro outlined improvements to LAVA like added monitoring, packaging, documentation and a test helper tool. He
LCE13: Test and Validation Summit: Evolution of Testing in Linaro (II)
- 1. Linaro Test and Validation Summit Linaro Engineering Teams LCE13 - Dublin, July 2013 How Do We Better Test our Engineering
- 2. PART 1: Linaro Platform Overview LAVA Citius,Altius,Fortius "Faster,Higher,Stronger" Builds&CI BuildYourCodeWhenYou areNotLooking QAServices Coverallbases PART 2: Linaro Engineering KernelDevelopers &Maintainers LandingTeams LinaroGroups LEG/LNG (DotheyuseCI? Manual orAutomatedTesting?)*Howdotheyvalidate/verifytheiroutput? *What/howdotheydevelop? &easiertouse
- 3. Agenda (Tuesday, 9am-1pm Time Topic Speaker 9:00 Introduction (Dev, Test, Loop) Alan Bennett 9:15 Overview of the CI loop (25 min) Fathi Boudra 9:40 QA services (20 min) Milosz Wasilewski 10:00 Recent LAVA updates (45 min) Antonio Terceiro 10:45 BREAK 11:00 LNG Mike Holmes 11:30 Landing Teams Scott Bambrough 12:00 KWG PM Kevin Hilman 12:30 LEG Grant Likely PlatformUpdates technicaldetails
- 4. Preface Why is the Quality of Linaro Engineering so important?
- 5. Preface Why is the Quality of Linaro Engineering so important?
- 6. ● applying continuous quality control ● frequent integration of small pieces of software ● rapid feedback ● Extreme programming (XP) ○ minimize integration problems ● Shared code repositories ● daily commits ● automated build systems ● extensive unit tests ● testing in cloned production environments Highlights of Continuous Integration Preface Continuous Integration
- 8. Linaro Test and Validation Summit Fathi Boudra Builds and Baselines LCE13 - Dublin, July 2013 How Do We Better Test our Engineering
- 9. ● CI Present ○ Anatomy of CI loop ● CI Future ○ What is on the CI roadmap Overview
- 10. ● Get the source ○ Source code is under SCM ■ Git (git.linaro.org) ■ Bazaar (bazaar.launchpad.net) ● Build the code ○ Use a build system ■ Jenkins (ci.linaro.org and android-build.linaro. org) ■ LAVA (yes, LAVA can be used!!!) ● Publish the build results ○ Build artifacts are available (snapshots.linaro.org) Anatomy of CI loop
- 11. ● Submit the results for testing ○ LAVA (validation.linaro.org) ● Get the tests results ○ E-mail notifications with filters (validation.linaro. org/lava-server/dashboard/filters) ○ LAVA dashboard (validation.linaro.org/lava- server/dashboard) Anatomy of CI loop
- 12. ● Different type of jobs ○ Kernel CI ○ Engineering builds ○ Components ● Build triggers ○ manual, periodically, URL trigger, post-commit ● Do the build ○ shell script(s) ■ can be maintained under SCM (linux-preempt-rt) ○ Groovy script(s) ● Publish ○ to snapshots.linaro.org ○ to package repositories (PPA, other) Build jobs in depth
- 13. ● Submit to LAVA ○ Generate a LAVA job file (json) ○ test definitions are pulled from SCM (git.linaro. org/gitweb?p=qa/test-definitions.git) ● Misc ○ Jenkins can run unit tests (e.g qemu-ltp job) ■ junit ■ xunit ○ CI helpers ■ post-build-lava ■ post-build-ppa ■ Linaro CI build tools Build jobs in detail
- 14. ● LAVA CI Runtime ○ LAVA as a build system ● LAVA Publishing API ○ LAVA ability to publish artifacts on remote host ● Build time optimization ○ persistent slaves ○ mirrors and caching ● Better documentation CI Future
- 16. Linaro Test and Validation Summit Milosz Wasilewski QA Services LCE13 - Dublin, July 2013 How Do We Better Test our Engineering
- 17. QA Services Tasks: ● manual testing ● dashboard monitoring ● reporting ● porting tests to LAVA
- 18. Manual Testing Current approach: ● test results are not very detailed ● no connection between test case description and result sheet ● results stored in google spreadsheet ● bug linking done manually (makes it hard to extract the list of 'known issues')
- 19. Future: ● store test cases in some better suited place than wiki ● preserve test case change history ● store manual test results along automatic ones (in LAVA) ● have ability to link bugs from various tracking systems to failed cases (in LAVA) ● generate reports easily (known issues, fixed problems, etc.) ○ might be done using LAVA if there is an easy way to extract testing results (for example REST API) Manual Testing
- 20. ● Monitoring dashboard ○ adding bugs ○ debugging failed runs ● Creating custom dashboards ○ Dashboard from filter ○ No need to edit python code to create/edit dashboard ○ Private/public dashboards ○ Dashboard email notification (falls in the concept of filter-as-dashboard approach) Dashboards
- 21. ● Use only binaries that were already automatically tested ● Don't repeat automated tests in manual run (we have to be confident that automated results are reliable) Release workflow
- 22. LAVA: Faster, Higher, Stronger (& easier to use) Antonio Terceiro LAVA LCE13 - Dublin, July 2013 Test and Validation Summit
- 23. ● Improvements ● New testing capabilities ● Engineering Progress Overview ● What are we missing? ○ Open Discussion ○ We want to hear from you Overview
- 24. ● ~90 ARM devices ● ~300 ARM CPUs ● ~150 jobs submitted per day ● ~99% reliability Context (0): the size of LAVA, today
- 25. ● LAVA started as an in-house solution ● Open source since day 1 ● Other organizations (incl. Linaro members) interested in running their own LAVA lab We need to go from an in-house service to a solid product Context (1)
- 26. ● No bootloader testing ● Tests only involve single devices We need to provide features to support new demands in test and validation Context (2)
- 27. Improvements
- 28. ● Queue size monitored with munin ● Nagios monitoring all sorts of things (e.g. temperature on Calxeda highbank nodes) ● Health check failures Monitoring
- 29. Easing LAVA installation ● Effort on proper upstream packaging so that packages for any (reasonable) OS can be easily made ● WIP on Debian and Fedora packaging $ apt-get install lava-server $ yum install lava-server Packaging enhancements
- 30. Easing LAVA learning ● Documentation is ○ scattered ○ outdated ○ confusing Documentation overhaul is in the LAVA roadmap. Documentation overhaul
- 31. Easing LAVA usage ATM a lava-test-shell job requires ● 1 JSON job file ● 1 YAML test definition file ● + the test code itself $ sudo apt-get install lava-tool $ lava script submit mytestscript.sh $ lava job list LAVA test suite helper tool
- 32. Getting more out of LAVA data More information out of LAVA data ● Improvements in test results visualization in the LAVA dashboard
- 34. LAVA is too hard to develop ● Too many separate components ○ Also a mess for bug/project management ● Requires almost a full deployment for development ● Consolidated client components (3 to 1) ● Will consolidate server components (3+ to 1) Developer-friendliness
- 36. ● LAVA Multi-purpose Probe ● 1 base design, 5 boards now ● USB serial connection(s) to the host ● management of other connections to/from devices under test LMP
- 37. ● prototype sets manufactured and under test ● Use cases: ethernet hotplug, SATA hotplug, HDMI hotplug and EDID faking, USB OTG testing, USB mux (sort of), lsgpio, audio hotplug, SD-Mux for bootloader testing LMP (2)
- 39. LMP (3) - how it works (e.g. SD-MUX) DUT SDC1 Host LMP USB serialUSB MSD
- 40. Multi-node testing (1) ● Schedule jobs across multiple target devices ○ Client-server, peer-to-peer and other scenarios ● Combine multiple results into a single result ● LAVA will provide a generic interface, test writers can program any tests they need. ○ (special hardware setups possible but need to be handled case-by-case) Other sessions: ● LAVA multi-node testing on Thursday ● LNG multi-node use-cases on Friday
- 41. Multi-node testing (2) ● Logistics challenge! ● We might end up needing 20 of every device type in the lab ● Need to manage the needed growth in the lab in a sensible way
- 42. Other projects ● Lightweight interface for kernel developers ● Boot from test UEFI on all Versatile Express boards ● Support for new member boards
- 44. In Progress ● LAVA LMP ● Multi-node testing ● Helper tool ● Test result visualization improvements ● Lightweigth interface for kernel devs ● UEFI on V. Express ● Support for new member boards In Progress X Planned Planned (for soon) ● Server components consolidation ● QA improvements ● Doc overhaul
- 46. Open Discussion
- 47. ● What is your experience getting started with LAVA? ● What would have made your experience easier? ● Any suggestions to the LAVA team? Let us know! ● Feedback about the image reports revamp? Seed Questions