C:\documents and settings\selvam.mc\my documents\automation testing process
Download as PPTX, PDF1 like551 views
This document discusses challenges with large-scale test automation and provides recommendations to address those challenges. It recommends that automation be method-centric rather than tool-centric, focusing on maintainability, reusability, and scalability. Test design should optimize these factors through practices like separating test objectives from test cases. Automation tools should support extensibility, validation, and a keyword-driven approach. The document advocates for just-in-time automation and obtaining an efficient test volume exceeding 50% coverage to reduce the cost of automated tests.
C:\documents and settings\selvam.mc\my documents\automation testing process
- 1. Large - Scale Test Automation ProductionDate :09th July’2010Place :Chennai Olympia Tech Park, IndiaAnbhu Selvam.MC…MS(CS).MBA.(Senior Software Quality Engineer, EDS. India)
- 2. Common ChallengesAutomation Technology–Multiplatform (cross-platform) support•Ability to interface with multiple platforms•Ease of extensibility to new platforms•Reusability of test suites/scripts across multiple platforms–Output/Object validation support•High rate-of-change–Interface changes–Functionality changes
- 3. Large-scale Automations Method-CentricAutomation is Method-Centric—it is more than about the technology; it is about the effective application of technology–Maintainability—handling high rate-of-change–Reusability—ability to reuse common test components without having to program/code–Scalability—ability to automate large volume of tests through reusability (and team-based cost efficiency)–Visibility•Tests auditable by management and non-programmer staff•Productivity is structurally measurable
- 4. Efficiency is Key!Efficiency Key Contributors•Test designDefinition: A test (case) creation activity with intent to serve one or both of the following objectives: 1.Exposing bugs/errors (or show otherwise, it works as intended), and 2.Optimizing maintainability and scalability (especially for automation-readiness).•Test automation (large-scale)–Method-centric, not tool-centric–Team-based, not individual coders
- 5. Automation Strategy Must Include Efficient Practices–Methodology•High maintainability (low maintenance cost)•High reusability–Technology•High extensibility•Object validation capability•People and process–High scalability (large-volume and team-based methodology)–Just-in-time automation—test engineers are skilled at test design and automation can be done before code-complete and/or concurrent to exploratory testing–High manageability (high visibility)
- 6. "Agile" System DevelopmentEstablish objectives (global)•Develop tests•Develop the system –in iterations, one subsystem at the time–test-driven•ReleaseAgile ―Home Ground*‖•Low mission-critical•Senior developers•Requirements change often•Small number of developers•Culture that thrives on chaos
- 7. ReservedRole of AutomationEffective automation can allow tests to support agile system development•Automation should not dominate. Don't make it into an "agile automation" project•A keyword driven method is, in my view, essential for successful test development w/ automation in agile development environment–Automation is separate from test development–Usually doesn’t need user interaction (only some with testers)–Automation focuses on action-keywords, not on tests
- 8. DEVELOPMENTStakeholders Testers Agile Test Development ProcessAUTOMATIONAUTOMATIONAUTOMATIONSystem Development Level –II Development
- 9. RecommendationsHave a global test-design scheme–Keep it short and simple: focused on breakdown of the tests–This global test design can (and usually should) be revisited throughout the project cycle•Separate (1) test objectives and (2) test cases–Test objectives are easier to create/describe than test cases•Use your stakeholders (specialists, power-users, etc.) wisely–Focus the efforts on the relevant tests–Don’t bother them more than needed•Avoid involving them in low-and medium-level tests–Observe various types of input: •How does this work?•What do we need to test?•What is interesting to test/How can we break the system?•Keep automation separate–Test developers are not consumed by automation–Use keyword-driven approach (e.g., Action Based Testing)
- 10. EAS: Extended Automation Support
- 11. 6000 3000250200 22 20Application (High-level)ActionsTESTSBuilt-in/System(Low-level)ActionsScalability Illustration3000
- 12. Division of Work
- 13. Key Takeaways1. Fully understand automation cost-of-ownership2.Don’t underestimate the challenge of keeping maintenance costs low3.You need to get efficient—optimize your volume of test to exceed 50% coverage4.Efficiency is key, and it will come from excellent test design and automation methodology (e.g., action-driven), and a well architected framework technology5.Minimize programming tests6.High scalability comes from high reusability of common ―actions‖ and team-based staffing model 7.High maintainability comes from keeping maintenance activities at the lowest level8.High visibility in your automation program to give you control and measurability, which ultimately leads to manageability9.Practice just-in-time automation10.Have a global test-design scheme that separate test objectives from test cases; and automation from test development
- 14. Automation Cost of Ownership&The Need for Large-Volume Automation
- 15. Automation Cost of Ownership1.Technology = Tool Licensing + Development/Customization Cost2.Production Cost = *One-time Cost + **Recurring Cost*NEW–One-time Cost of Design, Implementation & Execution–Test Case or Test Script Creation »Test line creation »Sequencing test logic or order of test lines (test steps) »Testing and debugging test script –Function or Keyword Creation –Interface Capturing or Mapping –Test Dataset Creation **EXISTING—Recurring Cost Test Execution & Maintenance Cost –Monitoring test suit execution –Investigating and troubleshooting false negatives –Maintaining Test Case or Test Script»Test line modification »Updating test logic or re-sequencing order of test lines (test steps) »Testing and debugging updated test script »Function or Keyword Modification »Interface Recapturing or Remapping »Test Dataset Modification
- 16. The Cost of AutomationThe Cost of Owning an Automated TestCost of an Automated Test*= **Cost of Ownership / Volume (#) of Tests* All test cases are not equal—a clear and structured definition of a test case is required. Furthermore, this cost will be a moving target over time** Cost of Ownership = Technology Cost + Production Cost
- 17. The Need for Large Volume Cost of an Automated Test*=**Cost of Ownership/Volume (#) of Tests You need to get efficient by optimizing the Volume of Tests! * All test cases are not equal—a clear and structured definition of a test case is required. Furthermore, this cost will be a moving target over time ** Cost of Ownership = Technology Cost + Production Cost
- 18. Rules of Automation Built-to-Last1.Tests are treated as product asset, along with the source code.2.Tests, good or bad are dependent on the design.3.Tests, manual or automated must be optimized for visibility, reusability, scalability and maintainability.4.Tests must be automation-ready.5.Tests, if they are worth automating, should follow the 5% rule:No more than 5% of all tests should be executed manuallyNo more than 5% of all efforts around testing should involve automating the testsNo more than 5% of coded test scripts against non-coded test scripts.
- 19. Action Based Testing Example
- 20. Example with Test Architect™ (TA)AT Computer Mobile APITCP/IPWirelessAT Agent
- 22. Action Definition for ―send SMS message and check‖ action
- 23. TA Connects to Window Mobile via TCP/IPTCP/IPWireless
- 24. Interface Definition for ―message editor‖ Entity
- 25. Questions ?