Test automation lesson
- 1. Test Automation Lesson Version 1.0 (9th July 2019) Sadaaki Emura
- 2. Agenda 1. Purpose of Test Automation 2. Test automation tool 3. Test automation process (End to End test) 4. Test automation pyramid
- 3. Scope System under test is Web Application (Browser , Native app) Target test level is System Test (End to End)
- 4. Purpose of test automation
- 5. Purpose of Test Automation What test automation does • Set up test preconditions (build server, docker, set up app, create test data etc) • Execute tests • Validation (compare actual outcomes to expected outcomes) @いらすとや
- 6. Purpose of Test Automation What effect comes • Improve test efficiency • Provide wider function coverage (improve test coverage) • Reduce the total test cost • Speed up test execution , quick feedback , increase the test cycle @いらすとや
- 7. Advantages & Disadvantage Test Automation Advantage • Run test repeatedly, faster • Can do test that cannot be done by manual (complex, wide coverage, repeat) • Reduce human operation error, improve consistency of tests • Effective and efficient use of testing resource Disadvantage • Additional cost for scripting and maintenance • Require additional technologies , skills • Need maintenance ( update script, run stable ) • Test can be more complex • Additional investigation for errors by automation Limitation • Not all manual tests can be automated • Can only check machine-understandable results • can only check actual results with test oracle prepared (compare) • Cannot do exploratory testing
- 9. Tool depend on Test level @wikipedia • Requirements and architecture phase • Check documentation format, proofreading • Ex. RedPen, ClearDoc • Implementation • Statistic analysis • check coding rule, bug, security issue • Ex. Coverity , Adlint • Dynamic analysis (Unit test) • Check program specification by unit level, white-box • Ex. Junit, PHPinit • Integration test and verification • Check interface level (API) • Ex. SoapUI • System Verification and Validation • Check End to End level, load test, security test • Ex. Selenium, Appium, Ranorex, UFT
- 10. End to End test automation tool How to recognize object License Technical skill Ranorex Id, XPath Not free Middle level UFT Id, XPath Not free Middle level Selenium IDE Id, XPath Free Low level Eggplant Image Not free Middle level Selenium + Webdriver Id, XPath Free High level
- 11. How to recognize objects Class , Id Xpath
- 12. Test automation process (End to End test)
- 13. Test Automation process 1. Test scope 2. Test design 3. Coding design 4. Setup scripting environment (System Under Test) 5. Script code 6. Check test automation 7. Run test automation, evolve into Continuous Integration 8. Maintain test automation
- 14. Test Automation process 1.Test scope Decide which area , function should be automated • Collaborate with manual testing • Consider cost and building time Sample standard • No scope automation cannot do test • No scope the functions scripting is complex • No scope scenario reproduce is difficult • No scope test is seldom used • No scope the functions specification is changed frequently
- 15. Test Automation process 2.Test design Test automation is same as app. That’s why test automation specification is needed. Test design is same as this specification. • Basically same as manual test design / scenario • Clarify step, what is validated and what is expectation for machine to understand • Clarify what is not automated (constraint) in this step to cover it by manual testing
- 16. Test Automation process 3.Coding design Consider architecture same as app programing • Simple code, coding rule • Maintainability (library , object oriented re-usable) • Performance ( minimum steps ) • Data driven ( data and script are separated ) • Keyword driven ( encapsulation steps ) • Consistency ( initialize, tear down ) • Sensitive and robust
- 17. Data driven Capture 1. Input ‘testuser001’ in userid 2. Input ‘abcdefg’ in password 3. Click ‘login’ Data driven 1. Input $userid in userid 2. Input $password in password 3. Click ‘login’ Data source userid , password testuser001 , abcdefg testuser002 , hijklmn Separate step and data Reusability Maintainability
- 18. Sensitive and robust How volume do test automation validate? Sensitive • Compare many information to validate • Coding cost increase • Easy to fail by little change UI Robust • Limit little information to compare • Coding cost is a small • Difficult to fail by little change UI • Miss bug often
- 19. Test Automation process 4.Setup scripting environment (System Under Test) build test environment • Script test code based on this environment • If this system does not work, it’s difficult to script code • If system UI include DOM level was changed , code might be remade 5.Script code • Recognize object by id, XPath, image etc. • Build step • Set validation that compare actual with expectation
- 20. When can we start scripting ? Ex. Waterfall model requireme nt Design coding test release Class, id 設計する？ Build environment When is system stable ? Actually we can start scripting
- 21. Test Automation process 6.Check test automation • Run script and check if test automation is correct • Basically if app has bug, test automation might not be correct even if it works successfully • Need to do failed test pattern • Check if report is enough 7.Run test automation, evolve into Continuous Integration • Do test • Use for contentious integration
- 22. Test Automation process 8.Maintain test automation • Increase coverage • Up-to-date app specification • Up-to-date test automation tool version, test device version • Monitor test results from CI • Monitor performance , tune up script • Check flaky test script, investigate and fix it • Delete unused , un-useful test script • Reduce task dependency in CI • Visualize metrics ( success/failed , performance ) • Keep good data ( delete unnecessary data )
- 24. Test Automation pyramid (c) OnTestAutomation Top Layer : User interface tests • Scripting test and execution takes long time • Environment is unstable Base Layer : Unit tests • Script test and execute quickly • Difficult to find integration bugs • Analyze and fix Bug are easy