Automation from start to finish
- 1. Testing Automation From Start to Finish Liang Gao
- 2. Testing Automation Attribute: 贵 expensive 2 This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
- 3. Testing Automation Attribute: 冰山 iceberg 3 This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
- 4. Agenda ∗ Realize when you need automation ∗ Always remember the goal ∗ Find the way to control your product ∗ API design ∗ Define the development process ∗ Script Quality ∗ Do I need an automation framework? ∗ Define the execution process ∗ Team management ∗ Equipment management 4 This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
- 5. Automation is useful for…. ∗ Frequently functionality regression testing ∗ Sample based performance testing ∗ Repetitive work for system and solution testing 5 This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
- 6. Frequently functionality regression testing ∗ Faster turnaround ∗ Reduce human errors ∗ Cost saving 6 This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
- 7. Sample based performance testing 7 This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
- 8. system and solution testing ∗ Semi-automation ∗ Configure 1000 VPN tunnels ∗ Create 10000 routing table entries ∗ ….. 8 This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
- 9. Agenda ∗ Realize when you need automation ∗ Always remember the goal ∗ Find the way to control your product ∗ API design ∗ Define the development process ∗ Script Quality ∗ Do I need an automation framework? ∗ Define the execution process ∗ Team management ∗ Equipment management 9 This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
- 10. The Goal ∗ No matter manual or auto Accurate Testing Report 10 This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
- 11. Agenda ∗ Realize when you need automation ∗ Always remember the goal ∗ Find the way to control your product ∗ API design ∗ Define the development process ∗ Script Quality ∗ Do I need an automation framework? ∗ Define the execution process ∗ Team management ∗ Equipment management 11 This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
- 12. Remote Control Your Products Personal Computer Web Browser Truck Tablet Video Printer Camera Telecommuter House with Web Server Router Video Refrigerator Phone Television Woman - End User Scanner Television - Closed-circuit Weather Station Woman - Set Top Box Sitting (STB) 12 Tunnel This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
- 13. Remote Control Your Testing ∗ 2 ways of software manual testing (or Device Control) ∗ Mouse and Eye (expectation) ∗ Keyboard and Eye (expectation) ∗ Automation is to repeat ∗ Mouse and Eye – GUI automation ∗ Keyboard and Eye – CLI automation 13 This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
- 14. GUI Automation ∗ Windows based GUI automation ∗ AutoIT ∗ Browser based GUI automation ∗ QTP ∗ WinRunner ∗ Silktest ∗ AutoIT ∗ Selenium 14 This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
- 15. CLI Automation ∗ Expect ∗ TCL Expect ∗ Perl Expect ∗ Python Expect 15 This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
- 16. Embedded devices State of the Union ∗ Most network devices (network equipment) ∗ Management interface: Browser or console ∗ Mobile devices ∗ Keyboard simulation ∗ Serial port, USB port or Bluetooth 16 This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
- 17. A Sigma Approach (in progress) Keyboard simulation, predefined key sequence (XML), render engine and Screen capture Result Verification (manual, only once) Replay keyboard simulation, Screen capture and Image comparison 17 This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
- 18. Pick the Tool is just beginning 万里长征第一步 18 This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
- 19. Agenda ∗ Realize when you need automation ∗ Always remember the goal ∗ Find the way to control your product ∗ API design ∗ Define the development process ∗ Script Quality ∗ Do I need an automation framework? ∗ Define the execution process ∗ Team management ∗ Equipment management 19 This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
- 20. API Design ∗ Lessons learned: API (Library) design is key for ∗ Productivity ∗ Modularity ∗ Maintainability ∗ Expansion ∗ Don’t start with programming, start with API design ∗ 磨刀不误砍柴工 – It is true… 20 This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
- 21. API Design ∗ Design pattern or other OO technique apply ∗ Need senior software architect blessing ∗ More design brain storming sessions, more reviews, the better. ∗ Spend more time on the API design, it will pay off in the long run 21 This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
- 22. API Principle Scripts API Control Library 22 This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
- 23. API Design - Principle •Connect to Device Device_id = connect_to_device(); ∗ Script only contains testing actions, no software •Issue XXX command to send Status = Send_SMS(device_id, $string); independent info SMS messages Assert($status); •Verify messages have been Status = Get_SMS(sent, $string) sent successfully Assert($status) •Verify messages saved in the SMS sent folder Status = delete_SMS(sent,$string); •Delete message Assert($status) •Verify message has been deleted 23 This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
- 24. API User Manual ∗ Most important document for script development API 名称 描述 输入参数 参数 1 参数 2 参数 3 返回值 使用举例 This material is for training purpose only. No duplication without consent of Sigma-RT is permitted. 24
- 25. Agenda ∗ Realize when you need automation ∗ Always remember the goal ∗ Find the way to control your product ∗ API design ∗ Define the development process ∗ Script Quality ∗ Do I need an automation framework? ∗ Define the execution process ∗ Team management ∗ Equipment management 25 This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
- 26. Script Development Process ×Ô¶¯»¯½Å±¾¿ª·¢Á÷ ³Ì²Î¿¼ 26 This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
- 27. Define Script Quality ∗ All common development coding standard applies ∗ Version control ∗ Standard template ∗ Standard header ∗ 20% of the code is comments ∗ Well indented ∗ Use API and functions ∗ Enough error checking 27 This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
- 28. Agenda ∗ Realize when you need automation ∗ Always remember the goal ∗ Find the way to control your product ∗ API design ∗ Define the development process ∗ Script Quality ∗ Do I need an automation framework? ∗ Define the execution process ∗ Team management 28 This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
- 29. Define Script Quality ∗ Testing script has it own quality too ∗ Exactly match the manual testing steps ∗ Just by reading logs, we know exactly what scripts are trying to do ∗ Device and hardware/software environment independent ∗ Enough debugging message when test case failed ∗ Clean out environment before exit or before execution 29 This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
- 30. Script Quality Checklist Sigma Script Quality Others Script Quality Checklist Checklist 30 This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
- 31. Agenda ∗ Realize when you need automation ∗ Always remember the goal ∗ Find the way to control your product ∗ API design ∗ Define the development process ∗ Script Quality ∗ Do I need an automation framework? ∗ Define the execution process ∗ Team management 31 This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
- 32. Automation Framework ？ ∗ Always remember our goal ∗ You need at least an execution engine for ∗ Error resuming batch run ∗ Report generation 32 This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
- 33. Automation Framework ∗ Want to design and implement an Automation Framework? ∗ Checkout <<Automation Framework: Design and Implementation>> 33 This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
- 34. Agenda ∗ Realize when you need automation ∗ Always remember the goal ∗ Find the way to control your product ∗ API design ∗ Define the development process ∗ Script Quality ∗ Do I need an automation framework? ∗ Define the execution process ∗ Team management 34 This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
- 35. Execution Process Always remember the goal 35 This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
- 36. Execution Process ∗ Image upgrade process ∗ Parallel execution ∗ Total run time ∗ Debugging time ∗ Report collection and generation 36 This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
- 37. Image upgrade process ∗ Where to get the image ∗ Can you automatically get the image and upgrade to your product ∗ Call this a “clean” procedure 37 This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
- 38. Parallel execution ∗ To save total run time – sometimes even automation is too long ∗ This is where the script need to be environment independent ∗ Be aware on the equipment cost issue ∗ How many parallel scripts can an engineer run and manage (also equipments) 38 This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
- 39. Run Time ∗ Total run time per engineer ∗ Total Debugging time per engineer ∗ Total reporting time per engineer 39 This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
- 40. Report collection and generation ∗ If it is parallel, need a process to parallel report collection and generation ∗ Testing Information Database? ∗ Web Interface? 40 This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
- 41. Agenda ∗ Realize when you need automation ∗ Always remember the goal ∗ Find the way to control your product ∗ API design ∗ Define the development process ∗ Script Quality ∗ Do I need an automation framework? ∗ Define the execution process ∗ Team management 41 This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
- 42. Team Management ∗ Who write the scripts? ∗ Who write the framework ∗ Who execute the scripts ∗ Are they happy 42 This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
- 43. Who Write Scripts ∗ Whoever write the test plan, write the test scripts ∗ Knows test case the best ∗ Not professional coder ∗ Hard for project management ∗ Professional scripter ∗ Need to understand the test case first ∗ High productivity ∗ Good Quality 43 This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
- 44. Who Write Framework ∗ Or any other testing related tools ∗ If a tester develop a great testing tool, who maintains it? ∗ Who to coordinate the common share library development ∗ Who to develop testing tools if there is a request ∗ Most of the time developer does this. ∗ If have resources, think about a Tools Team 44 This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
- 45. Who Execute the Scripts ∗ Whoever write the scripts, execute the scripts ∗ Low productivity ∗ Bad script portability ∗ Unpredictable schedule (bad for scheduled release) ∗ Dedicated resources (professional executer) ∗ Streamline resources ∗ High productivity ∗ Fixed total run time ∗ Most modern companies do it this way 45 This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
- 46. Interaction process ∗ Test cases writer and professional scripter ∗ Scripter and executer 46 This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
- 47. Case Writer and Scripter ∗ Repeatable test cases 47 This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
- 48. Scripter and Runner ∗ Script integration process ∗ Newly develop scripts submit to runner ∗ Use “State” to manage ∗ Script maintenance process ∗ New image change ∗ Use “State” to manage 48 This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
- 49. Team Building ∗ Manual tester, scripter are doing creative things ∗ Script runner may be board ∗ Role rotation ∗ Clear role for a clear career path 49 This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
- 50. Streamlining the team 脚本开发 脚本开发 脚本开发 脚本开发 脚本执行 脚本执行 脚本执行 50 This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
- 51. The Goal ∗ No matter manual or auto Accurate Testing Report 51 This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
Editor's Notes
- #21: Use Huawei high level wrapper (10 times productivity increase) as example
- #22: Use Huawei high level wrapper (10 times productivity increase) as example
- #24: Use Huawei high level wrapper (10 times productivity increase) as example
- #25: Use Huawei high level wrapper (10 times productivity increase) as example