Crowd-sourced Automated Firefox UI Testing
- 1. Crowd-sourced Automated Firefox UI Testing Henrik Skupin <hskupin@mozilla.com> Lead of QATest Automation quality.mozilla.org qa automation Thursday, November 4, 2010
- 2. Who is Mozilla QA? Thursday, November 4, 2010
- 3. WHO IS MOZILLA QA Group of about 30 employees / contractors Are you serious? No! Thursday, November 4, 2010
- 4. WHO IS MOZILLA QA 1000s of community members across the globe who are partners to the Mozilla QA core team Thursday, November 4, 2010
- 5. Mozilla Scale 400+ Developers 10,000+ Nightly Testers 2,500,000 Beta Testers 400,000,000 Users 400,000,000 350,000,000 300,000,000 250,000,000 200,000,000 150,000,000 100,000,000 50,000,0000 Thursday, November 4, 2010
- 6. What are our Areas of Work? Thursday, November 4, 2010
- 7. • Testing new features in beta versions and nightly builds • i.e. Switch toTab,Add-ons Manager, ... • Testing release builds before they get shipped • Smoketests across all platforms • ~ 300 basic functional tests (BFT) • Software update tests WHAT ARE OUR AREAS OF WORK? Thursday, November 4, 2010
- 8. A Brief History of Automation in Mozilla QA Thursday, November 4, 2010
- 9. A BRIEF HISTORY OF AUTOMATION IN MOZILLA QA • No automated functional tests existent by early 2009 • 100% manual testing of new Firefox features • 100% manual testing of release builds (limited to a single platform) Huge amount of manual resources necessary! Thursday, November 4, 2010
- 10. A BRIEF HISTORY OF AUTOMATION IN MOZILLA QA So what has been changed so far? Thursday, November 4, 2010
- 11. A BRIEF HISTORY OF AUTOMATION IN MOZILLA QA • Integration of MozMill (based on WindMill) for functional tests • Automation of basic tests (30% done - ~100 tests) • Automation of update tests (nearly done - Only 1single person needed to run 192 different update paths across 5 platforms and 4 P1 locales in under 2 hours) Perfect time to open up testing for everyone Thursday, November 4, 2010
- 12. Crowd SourcedTesting Thursday, November 4, 2010
- 13. CROWD SOURCEDTESTING Jeff Howe: "It’s not outsourcing; it’s crowdsourcing." Thursday, November 4, 2010
- 14. CROWD SOURCEDTESTING Crowdsourcing relies on people, and thus works when participants are treated as people Thursday, November 4, 2010
- 15. CROWD SOURCEDTESTING What can be achieved? • Support testers, localizers, and add-on authors to allow them to focus on their main tasks • Constantly testing builds across platforms and locales • Assembling test results from a large number of different environments (settings and performance) How can this be done? Thursday, November 4, 2010
- 16. MozMill Crowd Extension Thursday, November 4, 2010
- 17. MOZMILL CROWD EXTENSION Pre-requisites: • Trivial setup (“One click install”) • Uses MozMill to run tests in any version of Firefox • Support for different test-runs (basic tests, add-ons, ...) • Reporting test results to a central database Under Construction Thursday, November 4, 2010
- 18. MOZMILL CROWD EXTENSION What remains: • How reliable are our tests in the crowd • Fix focus issues to allow background tests (webdriver) • Analyzing results based on the environment • ... Thursday, November 4, 2010
- 19. DEMO What will be shown? • Setup of the test environment by using MozMill-Crowd • Test-run for a subset of general Firefox tests • Analyzing test results • Test-run for the GoogleToolbar extension Thursday, November 4, 2010
- 20. REFERENCES QATest Automation team page: http://quality.mozilla.org/teams/automation/ MozmillTests Repository: https://developer.mozilla.org/en/Mozmill_Tests Mozmill Code: https://github.com/mozautomation/mozmill Mozmill Crowd Extension Project: https://wiki.mozilla.org/QA/Mozmill_Test_Automation/ Crowd_Testing Thursday, November 4, 2010
- 21. QUESTIONS & ANSWERS Questions? Thursday, November 4, 2010
- 22. Thanks Thursday, November 4, 2010