Firefox OS real-phone automation lab: goals, challenges, and successes
Download as PPTX, PDF4 likes1,205 views
The document outlines the goals, challenges, and accomplishments involved in building a real-phone test-automation lab for Firefox OS. Key goals include increasing capacity for builds, improving reliability, and facilitating device troubleshooting, while major challenges encompass maintaining existing infrastructure amidst aggressive build schedules, managing dual SIM configurations, and ensuring robust connectivity. The outlined solutions involve unique device management strategies and enhancing the Jenkins configuration to support multiple nodes efficiently.
![Goals:
● (much) greater capacity
o keep up with b2g-inbound builds for UI + perf (esp. the latter), as much
as possible
● maximal use of resources (devices, limited personnel, etc.)
● multi-node adb mapping/support
● high[er|est] reliability
● easier device troubleshooting/administration
● break new ground and help inform the testing world](https://image.slidesharecdn.com/firefoxosreal-phoneautomationlab-141121115939-conversion-gate01/85/Firefox-OS-real-phone-automation-lab-goals-challenges-and-successes-2-320.jpg)
![What (was?) is Left?
● MozPool[1] for managing devices, including:
o taking devices offline
o power measurement through in-house, custom-3D-printed ammeters
o remote-reboot capability through power harness
● Puppetization for managing node configurations
[1] https://github.com/mozilla/mozpool](https://image.slidesharecdn.com/firefoxosreal-phoneautomationlab-141121115939-conversion-gate01/85/Firefox-OS-real-phone-automation-lab-goals-challenges-and-successes-20-320.jpg)
Firefox OS real-phone automation lab: goals, challenges, and successes
- 1. Building a 40+ real-phone test-automation lab for Firefox OS: Goals, Challenges, Accomplishments San Francisco Selenium Meetup Group November 19, 2014 Stephen Donner sdonner@mozilla.com
- 2. Goals: ● (much) greater capacity o keep up with b2g-inbound builds for UI + perf (esp. the latter), as much as possible ● maximal use of resources (devices, limited personnel, etc.) ● multi-node adb mapping/support ● high[er|est] reliability ● easier device troubleshooting/administration ● break new ground and help inform the testing world
- 3. From this... (which itself is “test rig 3.0” -- if you can believe that!)
- 4. 1st Buildout
- 5. To this...
- 6. 2nd Buildout
- 9. Challenges: (1/4) ● balancing aggressive buildout schedule and challenge with current-rig’s maintenance (a.k.a. “changing a plane’s engine at 30,000 feet”) ● near-constant task (re)prioritization (a.k.a “new capabilities, new requests/demands”) ● real carriers/networks (AT&T/T-Mobile), real problems: o spam calls/voicemail o Amber alerts o low signal strength/data-connection resets slow throughput DNS-lookup failures o logistics of provisioning 40 SIMs ● phone flashing (en masse): o base (OEM/vendor/reference) build non-engineering ● vs. o daily builds (release candidates, nightlies, etc.) o aggressive power-management features
- 10. Challenges: (2/4) ● multi-node adb mapping o solution had to work with both UI and perf tests o also required: Jenkins-config changes (job/system) physical reconfiguration ● and powered hubs ● active SIMs vs. inactive SIMs o reduce interference/cost/maintenance cost ● DSDS (Dual SIM, Dual Standby) ● SecOps/NetOps requirements o new office, new problems (new policies) basically, new office == slackware of offices ;-) port-by-port, host-by-host firewall pass-through o VPN/Jenkins (CI) access granted on a user-by-user basis
- 11. Challenges: (3/4) ● RF (Radio Frequency) o Wi-Fi ateam vs. Mozilla Mobile ● capacity + speed + reach o channels, bands, new router vendor o FM tuner interference requires cabling magic ● ...and perhaps a low-power, local FM transmitter? o more devices, moar interference? ● physical space/configuration o shelving space o power-strip capacity
- 12. Challenges: (4/4) ● remote teams requested live Air Mozilla streams of the phones, i.e. “what in the world is the state of that phone in?” o prototype works, but doesn’t immediately scale bandwidth (# streams, codecs/encoders) informs # of encoder boxes/# of channels o live (remote) demo ● “keeping the light on” o camera test failures due to low light ...but intermittent
- 14. Multi-node adb Mapping Background: ● Jenkins allows many "nodes" to run on 1 host ● ‘node’ meaning ‘agent’ ● ADB permits multiple sessions on 1 host ● $ adb -s <serial_num> <command> Problems: ● Some vendors don't provide a unique serial ● Sometimes OEM (base build) may have same serial number hard-coded in all devices! o You might not have any control over this :-( o There are tools for unpacking Android base images and rewriting those serial numbers: https://github.com/AndroidRoot/BootTools Solution: ● Create 1 unique node per mobile device ● This = 4 devices to one slave host, 1 per USB port ● Manage scripts - modify our scripts to pass in device serial numbers (to all adb commands)
- 15. Currently*, we have: ● ~30 active Flame nodes (phones) o 15 dedicated to UI / 12 perf o 26 potentially ● attached to 13 active Mac Minis
- 20. What (was?) is Left? ● MozPool[1] for managing devices, including: o taking devices offline o power measurement through in-house, custom-3D-printed ammeters o remote-reboot capability through power harness ● Puppetization for managing node configurations [1] https://github.com/mozilla/mozpool
- 22. Credit Where Credit is Due ● Malini Das ● Raymond Etornam ● Jonathan Griffin ● Andrew Halberstadt ● Dave Hunt ● Johan Lorenzo ● Richard Milewski ● Richard Pappalardo ● Clint Talbert ● Dylan Wong ● Rob Wood o … and everyone else I forgot!
- 23. Contact ● Mailing list: mozwebqa@mozilla.org ● Team page: https://quality.mozilla.org/teams/web-qa/ ● IRC: #mozwebqa on irc.mozilla.org