selenimu commiter day 2017: Methods to Sustain Long Term Operations of E2E Automated Tests
6 likes4,645 views
The document outlines methods for sustaining long-term operations of end-to-end automated tests used in a browser-based game platform developed by Dena Co., Ltd. Key strategies include periodic execution of test suites, establishing processes to exclude flaky test cases, enhancing test debuggability, and decoupling tests from the system under test. These practices aim to improve test stability, reduce maintenance costs, and facilitate efficient software development.
![Copyright © DeNA Co.,Ltd. All Rights Reserved.
3. Improve test debuggability
■ Live preview and video records for UI tests
■ zalenium
⁃ h]ps://github.com/zalaxndo/zalenium
⁃ developed by zaland
⁃ A selenium grid extension](https://image.slidesharecdn.com/2017-0701-komoda-selenimu-commiter-day-170702064927/85/selenimu-commiter-day-2017-Methods-to-Sustain-Long-Term-Operations-of-E2E-Automated-Tests-12-320.jpg)
![Copyright © DeNA Co.,Ltd. All Rights Reserved.
4. Make tests loosely coupled with SUT (system under test)
■ Goal
⁃ make tests hard to break
⁃ make tests robust to changes of SUT
■ Web API tests are relaNvely easy to maintain. Why ?
⁃ clear interface between tests and SUT
■ UI tests are not. Why ?
⁃ hard-coded page link informaNon
⁃ hard-coded DOM id or other a]ributes values
■ Can we make UI tests loosely coupled with SUT as API tests ?
⁃ 2 ongoing projects will be shown](https://image.slidesharecdn.com/2017-0701-komoda-selenimu-commiter-day-170702064927/85/selenimu-commiter-day-2017-Methods-to-Sustain-Long-Term-Operations-of-E2E-Automated-Tests-13-320.jpg)
![Copyright © DeNA Co.,Ltd. All Rights Reserved.
4.b Input topic iden5fica5on with machine learning
■ Goal
⁃ remove hard-coded DOM a]ributes values from UI test codes
■ use natural language processing technique to handle input forms
⁃ based on a research paper
「Using SemanNc Similarity for Input Topic IdenNficaNon in Crawling-based Web ApplicaNon TesNng」,
Jun-Wei Lin, Farn Wang
• h]ps://arxiv.org/abs/1608.06549
■ our prototype shows promising results
⁃ can esNmate 80% of input forms
⁃ experiments for login and user register forms
■ can be used with the previous site crawler](https://image.slidesharecdn.com/2017-0701-komoda-selenimu-commiter-day-170702064927/85/selenimu-commiter-day-2017-Methods-to-Sustain-Long-Term-Operations-of-E2E-Automated-Tests-15-320.jpg)
selenimu commiter day 2017: Methods to Sustain Long Term Operations of E2E Automated Tests
- 2. Copyright © DeNA Co.,Ltd. All Rights Reserved. Bio ■ Toshiya Komoda ⁃ Server side engineer ⁃ Test Engineer ■ developed game pla?orms, using automated tests ■ currently working in SWET group ⁃ SWET - SoFware Engineer in Test ⁃ dedicated team for test engineering
- 3. Copyright © DeNA Co.,Ltd. All Rights Reserved. mobage game plaMorm ■ Mainly a browser game pla?orms ⁃ tens of millions users play games on the pla?orm ■ A variety of services for 3rd party developers ⁃ OpenID Connect ⁃ payment service ⁃ Community service • friend, chat ⁃ and more
- 4. Copyright © DeNA Co.,Ltd. All Rights Reserved. History of mobage game plaMorms ■ Business trend shiFs to ⁃ browser to naNve ⁃ pla?orm to game publisher ■ SNll ⁃ a lot of users play games on our pla?orms ⁃ the profit is huge ■ Our goal ⁃ maintain high availability and strict backward compaNbility ⁃ with minimum maintenance cost automated E2E test suites has been key to achieve this goal
- 5. Copyright © DeNA Co.,Ltd. All Rights Reserved. Some sta5cs ■ 5 ⁃ years operaNons ■ 22 ⁃ test suites ⁃ grouped based on the funcNonality ■ 700 ⁃ browser test cases ■ 60000 ⁃ web API test cases ■ enable developers to release server codes without manual QA ⁃ reduce lead 5me ⁃ reduce QA Cost
- 6. Copyright © DeNA Co.,Ltd. All Rights Reserved. Life cycle of test suites ■ Development Phase ⁃ build E2E tests as QA for service development ■ Healthy Phase ⁃ all test suites are stable enough ■ Broken Phase ⁃ environmental changes, test data corrupNon ⁃ increase of the raNo of unstable tests… ■ Dead Phase ⁃ no one can pass the E2E tests ⁃ QA cost and lead time increase a lot
- 7. Copyright © DeNA Co.,Ltd. All Rights Reserved. Life cycle of test suites ■ Development Phase ⁃ build E2E tests as QA for service development ■ Healthy Phase ⁃ all test suites are stable enough ■ Broken Phase ⁃ environmental changes, test data corrupNon ⁃ increase of the raNo of unstable tests… ■ Dead Phase ⁃ no one can pass the E2E tests ⁃ QA cost and lead time increase a lot Gain > maintenance cost
- 10. Copyright © DeNA Co.,Ltd. All Rights Reserved. 2. Define clear process to exclude test cases ■ Goal ⁃ Control cost and coverage trade-off ■ Allow to exclude and reNre test cases ⁃ highly flaky test cases ⁃ broken test cases that are hard to fix ■ Example process ⁃ idenNfy the reason of the test failure • off course, if there is a bug in producNon service, fix it. ⁃ if the reason is not a bug in producNon service ⁃ make JIRA Nckets and exclude the test cases
- 11. Copyright © DeNA Co.,Ltd. All Rights Reserved. 3. Improve test debuggability ■ Goal ⁃ minimize test maintenance cost ⁃ maximize the number of engineers who can see tests ■ Test language ⁃ should be familiar language to service developers ■ Test Infrastructure should provide ⁃ performance ⁃ retry mechanism ⁃ precise debug log
- 12. Copyright © DeNA Co.,Ltd. All Rights Reserved. 3. Improve test debuggability ■ Live preview and video records for UI tests ■ zalenium ⁃ h]ps://github.com/zalaxndo/zalenium ⁃ developed by zaland ⁃ A selenium grid extension
- 13. Copyright © DeNA Co.,Ltd. All Rights Reserved. 4. Make tests loosely coupled with SUT (system under test) ■ Goal ⁃ make tests hard to break ⁃ make tests robust to changes of SUT ■ Web API tests are relaNvely easy to maintain. Why ? ⁃ clear interface between tests and SUT ■ UI tests are not. Why ? ⁃ hard-coded page link informaNon ⁃ hard-coded DOM id or other a]ributes values ■ Can we make UI tests loosely coupled with SUT as API tests ? ⁃ 2 ongoing projects will be shown
- 14. Copyright © DeNA Co.,Ltd. All Rights Reserved. 4.a Visual regression tes5ng with site crawler ■ Goal ⁃ remove hard-coded link informaNon from UI test codes ■ use crawler techniques in regression tests ⁃ detect links in web pages automaNcally ⁃ take screenshots of all pages in the site 2-up swipe
- 15. Copyright © DeNA Co.,Ltd. All Rights Reserved. 4.b Input topic iden5fica5on with machine learning ■ Goal ⁃ remove hard-coded DOM a]ributes values from UI test codes ■ use natural language processing technique to handle input forms ⁃ based on a research paper 「Using SemanNc Similarity for Input Topic IdenNficaNon in Crawling-based Web ApplicaNon TesNng」, Jun-Wei Lin, Farn Wang • h]ps://arxiv.org/abs/1608.06549 ■ our prototype shows promising results ⁃ can esNmate 80% of input forms ⁃ experiments for login and user register forms ■ can be used with the previous site crawler
- 16. Copyright © DeNA Co.,Ltd. All Rights Reserved. Summary: Methods to Sustain Long Term Opera5ons of E2E Automated Tests 1. Periodic execuNon of all test suites 2. Define clear process to exclude test cases 3. Improve test debuggability 4. make tests loosely coupled with SUT (system under test)