How to Build a Fully Open Source Test Automation Framework
- 1. T3 Session 10/27/2016 10:15:00 AM How to Build a Fully Open Source Test Automation Framework Presented by: Matt Joste Ryerson University Brought to you by: 350 Corporate Way, Suite 400, Orange Park, FL 32073 888-‐268-‐8770 ·∙ 904-‐278-‐0524 - info@techwell.com - http://www.starcanada.techwell.com/
- 2. Matt Joste Ryerson University A software quality assurance and testing specialist for Ryerson University in downtown Toronto, Matt Joste has more than fifteen years of experience in software quality assurance including two years as a ScrumMaster. Passionate about helping teams become as agile and self-sufficient as possible, Matt has journeyed through the finance, banking, security, media, and now education sectors.
- 3. 10/17/2016 1 Presentation: How to Build a Fully Open Source Test Automation Framework Presenter: Matt Joste Date: October 27th 2016 ‐ Software > 15 years ‐ Industries: Banking, Finance, Software Security, Media ‐ Mostly QA, apart from a stint as a Scrum Master ‐ Currently QA & Testing Specialist @ Ryerson
- 4. 10/17/2016 2 Software Test Automation In software testing, test automation is the use of special software (separate from the software being tested) to control the execution of tests and the comparison of actual outcomes with predicted outcomes Kolawa, Adam; Huizinga, Dorota (2007). Automated Defect Prevention: Best Practices in Software Management. Wiley‐IEEE Computer Society Press. p. 74. ISBN 0‐470‐04212‐5. Why Automate? • Half the labor expended to develop a working program is typically spent on testing activities‐ (Boris Beizer, Software Testing Techniques,1990. + IDT Survey responses) • Automation can increase the depth and breadth of testing. Specific benefits include:* • Improved build verification testing(smoke testing) • Improved regression testing • Multiplatform compatibility and configuration testing • Improved execution of mundane tests • Improved focus on advanced test issues • Testing beyond manual: e.g. security or memory leak testing Abilit t d d f t i t tl• Ability to reproduce defects consistently • Extending the hours of operation *Automated Software Testing(Dustin et al)
- 5. 10/17/2016 3 Agile Software Development Agile software development is a group of software development methods in which requirements and solutions evolve through collaboration between self- organizing, cross-functional teams. It promotes adaptive planning, evolutionary development, early delivery, continuous improvement, and encourages rapid and flexible response to change "What is Agile Software Development?". Agile Alliance. 8 June 2013 Benefits of Agile Software Development Benefits Obtained from Implementing Agile ‐VersionOne ‘State of Agile Summary 2011’
- 7. 10/17/2016 5 Automation Tools Overview Open Source vs Proprietary 1. Support levels V i d b t d d t l( )Varied, but depends on tool(o) vs Guaranteed(for a price) 2. Cost of tools Free vs hundreds to thousands 3. Cost of personnel Open Source/in house could be much more expensive 4. Vendor lock‐in Systems and 'Vendorscript' *Lessons learned in Software Testing(Kaner et al) Available Automation Tools(selection) Selenium(o), QTP(p), AutoITi. a. GUI Testing b. Load Testing Jmeter(o), LoadRunner(p)i. Appium(o), MobileAnywhere(p),i. c. Mobile Testing STAF/STAX(o) i. d. Desktop App testing OWASP ZAP(o), BurpSuite(p), WebScarabi. e. Security Testing FireEyes(o), Compliance Sheriff(p)i. f. Accessibility Testing FireEyes(o), Compliance Sheriff(p)i. Jenkins CI(o), Quality Centre(p), Robot Framework(o), Excel(p) i. g. Test Case Management, Execution and Reporting FireEye/ACTS from NIST(o)i. h. Test Data Generation RegEx Coach(o), Fiddler(o), FireRobot(o)i. i. Automation Support
- 8. 10/17/2016 6 Cloud‐based Services Browserstack Sauce Labs Browserstack Keynote DeviceAnywhereBrowserstack Keynote DeviceAnywhere Appthwack BrowserSwarm Automation Frameworks • Create Structure for Automation Efforts(format, execution, reporting) • Allow for Abstraction of Test Cases( e.g. Keyword‐driven Testing) • Facilitate Growth of Test Repositories • Enable Many Contributors • Should Support Integration of Multiple Tools • Should Support Testing without having to install Framework on Every Computer
- 9. 10/17/2016 7 Pros and Cons of Automation Pros Repeatable consistentRepeatable, consistent Fast(once implemented) Possible(e.g. Load tests) Vital to realize Agile and Continuous Delivery Less tedious Cons Time consuming(3‐10 times manual testing)Time consuming(3 10 times manual testing) More skills required Expensive tools or personnel or both Maintenance Slower to start Doesn't fully replace manual testing How We Automate @ Ryerson • Open Source • Small QA team • Leverage developers • Build toward Agililty
- 10. 10/17/2016 8 The Ryerson CCS Automation Framework Robot Framework Test Example RF Test Case Folder
- 11. 10/17/2016 9 h k f h !Thanks for watching! https://www.linkedin.com/pub/matt‐joste/6/abb/b89 mjoste@ryerson.ca