Meet Big Agile: Testing on Large-Scale Projects
- 1. W16 Agile Testing 5/7/2014 3:00:00 PM Meet Big Agile: Testing on Large-Scale Projects Presented by: Geoff Meyer Dell, Inc. Brought to you by: 340 Corporate Way, Suite 300, Orange Park, FL 32073 888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com
- 2. Geoff Meyer Dell, Inc. A test architect in the Dell Enterprise Solutions Group, Geoff Meyer has more than twenty-eight years of experience as a software developer, manager, business analyst and test architect. Geoff helps serve the software development practices of more than 800 development, test, and UX engineers across four Global Design Centers. Geoff is an active member in and contributor to the Agile Austin community.
- 3. 4/26/2014 1 Agile Testing @ Large Geoff Meyer, geoffrey_meyer@dell.com May 2014 Last updated: 25 February, 2014 Courtesy: Gulliver's Travels (2010) Session Objectives • Challenges to expect with Agile practices in large-scale, multi-team projects • Brief overview of Industry Models • Real-world examples − from a large organization perspective • Organizational strategies & considerations • Test automation considerations • An In-depth Tools Discussion • Focused on how to develop Test Automation IS IS NOT
- 4. 4/26/2014 2 Geoff Meyer • Dell Inc, 1998 – present – Test Architect › Agile Test & Automation Strategy – Agile Steering Committee Co-Chair – Global Projects › 15 Scrum teams (2 regions) › 11 Scrum teams (4 regions) › 9 scrum teams (2 regions) › 7 Scrum teams (4 regions) • NCR Corp. 1984 – 1998 – SW developer, Project Lead, SW Manager • B.S. Computer Science, San Diego State University • Masters Engineering Management - NTU 3 4 Agenda • The Dell Context and • Early Agile Testing Challenges • In Search of… Solutions for Scale • Adaptations for Testing at Scale • Automating in an Agile world • Lessons of Large Agile @
- 5. 4/26/2014 3 5 The Context at Dell ESG http://amazngwallpapers.blogspot.com Dell Enterprise Solutions Group Global Design and Development using Agile Scrum 6 Noida Design Center Bangalore Design Center Austin Design Center Silicon Valley Design Center
- 6. 4/26/2014 4 Dell Enterprise Solutions Software Products • Server Systems Management • Converged Infrastructure Systems Management • Private Cloud Systems Management • Console Plug-ins (i.e. for SCCM, vCenter…) 7 Common Product Characteristics: • Embedded Software • Large hardware support test matrix • Software is installed in the Data Center • Enterprise update cycles ~6 months • Products must function even as underlying HW, FW, BIOS, and Drivers are upgraded - SUSTAINING Storage Network Compute The Early Challenges 8 ? http://cleaneatingchelsey.com/
- 7. 4/26/2014 5 Culture Transition • Early commitment on Requirements • Insufficient Detail in Requirements • Development vs. Test • A different “School” of Test1 • Fail Early, Inspect & Adapt 91 – Scott Barber “Approaches to Software Testing: An Introduction “ “ “Everybody, every team, every platform, every division, every component is there not for individual competitive profit or recognition, but for contribution to the system as a whole” W. Edwards Deming Signs of Trouble • Thursday’s Build • Planning more stories into sprint than team’s velocity • High % of carry-overs • Inability to Automate Acceptance Tests within Sprint • “Engineering” Stories Large-Scale • Cross-team interdependencies block sprint progress • Large HW configuration validation consumes test resources • No single team had visibility to the entire product • Insufficient Product Owner bandwidth http://www.huffingtonpost.com/2012/03/21/truck-stuck-thomas- circle_n_1369545.html?p=truck+and+underpass&vm=r
- 8. 4/26/2014 6 Test Automation Pitfalls • Minimal guidance to Test beyond “Go forth and automate” • Black Box Test Mentality • Automation was interpreted by many as “Automate the UI” 11 • Insufficient SW engineering skills across Test teams • Architecture(s) not optimized for Automatability 12 In Search of… Solutions For Scale
- 9. 4/26/2014 7 Leverage the Community 13 Books Conferences Professional Groups Consultants Dallas Agile Leaders’ roundtable Scaled Agile Framework™ Big Picture © 2008- 2013 Scaled Agile, Inc and Leffingwell, All rights reserved
- 10. 4/26/2014 8 Agile Testing Quadrants 15 4 - http://lisacrispin.com/wordpress/2011/11/08/using-the-agile-testing-quadrants/ 4 ALM Tooling 16 • Multi-team, geographically distributed projects create a necessity for ALM Tooling –Requirements Management –User Story Management –Test Case Management –Defect Management –Test Script Repository
- 11. 4/26/2014 9 Automation Tooling 17 • IDE • UI Tools • Scripting Languages • Services • Frameworks 18 Agile @ Dell Adaptations
- 12. 4/26/2014 10 Agile Testing @ PG Enterprise Solutions Group How did we go about it? 19 Process/Project Culture/Organization X-Large project (15) 2011 2014 UI Automation “Automate-First” Service-level automation ALM using TFS In-sprint Automation Accelerate Automation Skillset Maximize Utilization Automation Architects Automated Setup & Config 4 Geography project (11) 2012 TDD & Automated UT Non-Functional Scale, Longevity 2010 2013 Test into Scrum 2009 Collaborative Seating Rally Agile @ Dell Core Activities • Pre-Sprint activities: – Staffing – Training - Project tools and processes – CI/Build environment – Automation Framework and BVT • Establish Project-wide ‘Done’ criteria • End-to-end, short duration User Stories • Test Automation is included User Story acceptance criteria • Refresher Workshops for new projects 20
- 13. 4/26/2014 11 Week 1 Week 2 2 week Sprint Lifecycle Example Manual Regression Automated Regression Automated Regression Automated Regression Sprint Planning Automated Regression Automated Regression Sprint Review Retro spective Demo Accept User Story Demo Accept User Story Demo Accept User Story Demo Accept User Story Demo Accept User Story Product Owner (or Proxy) Scrum Master Development Test Tech Pubs UI 22 In-Sprint: Testing • Design/Develop Test cases and Review with team • Develop test automation and maintain in TFS • Execute test cases (manual and automated) • Immediate Feedback (IM, email, in-person) with team of sightings… do not automatically enter defects • Verify fixes for defects; re-test if needed • Maintain test case status (in MTM) • Cross-team coordination including Weekly SoS Confidential
- 14. 4/26/2014 12 Agile @ Dell Roles/Responsibilities 23 Product Owner Product Owner Proxy 1 per Scrum Scrum Master 1 per Scrum Development 4-5 per Scrum Test 3:1 Tech Pubs 1 per 3 scrum team UI 1 per Scrum Scrum teams • Small teams • Co-located • Multi-disciplined • Feature-based • Distributed Projects • Scrum of Scrums • Teams formed from functional silos … Release Manager SW Architect Test Architect Automation Architect Cross-team Test roles 24 • Test Architect – Defines Test Strategy across all Scrum teams – Establishes non-Functional Test approach – Influences consistent Application Architectures • Automation Architect – Leads Analysis and POC activities for new automation needs – Ensures automation consistency for BVT and nightly regression suites – Safeguards against automation tool proliferation
- 15. 4/26/2014 13 EXAMPLE - Project-wide “Done” criteria 25 Code – Junit Tests – SONAR – Code Reviews completed – All User Story tasks completed – Demo and accepted by Product Owner – All known error conditions caught and handled appropriately – Code is Globalized Test – Acceptance Tests Reviewed with scrum team – Acceptance Tests documented in Tool – All planned Acceptance Tests for User Story has been attempted – Acceptance Tests executed against Test system configuration – Defects that are “must fix” before releasing must be fixed prior to story acceptance – Service-level automated tests – Service-level automated Tests are globalized Large-Scale Complexities 26 Characteristic Description Large-Scale Project consists of more than 4 Scrum teams Interdependency Requirements implemented across multiple Scrum teams Extensive Configuration Matrix Extensive HW or SW configurations Waterfall Intersection One or more components are managed using Waterfall Geography Project members are Geographically dispersed
- 16. 4/26/2014 14 Development Usability Test Wireframes Architecture Automation FW/Tools Release Planning Pre-sprints Pre-sprints HW resources HW resources Pre-sprints : Agile with Adaptations 27 Hardening 1 2 3 N-1 N 1 2 3 N-1 N 1 2 3 N-1 N … Sprints Define Plan Develop Launch OLP … Software System Test Extended Sprint Test Stability Agile @ Extended Sprint Test Extended Hardware Configurations • Optimizes HW utilization and cost in support of Scrum teams • Owns Test Execution against extended configurations • Scrum team: – Establishes Acceptance criteria – Identifies all Test scenarios – Owns Test Execution against reference configuration(s) – Develops in-sprint automation tests 28 Configurations Positive X X X X Negative X X X Boundary X X X Stress X X Scalability X X X Concurrency X X X Globalization X X Candidate for Extended Sprint Test
- 17. 4/26/2014 15 Pre-SST Software System Test “Alpha Customer” 29 Hardening • Workflow Design • Test Analysis • Test Design • Test Development • Test Automation Development • Test Execution • Regression Test • Test Execution • Regression Test SST IS IS NOT Requirements-based Functional testing Customer-usage based First time integration test of workflows Interoperability coverage of HCL in specific customer environments Exhaustive interoperability matrix coverage of the HCL Three phased approach Performance baseline testing SAFe1 & 30 Agile @ 1- Courtesy of Dean Leffingwell and Al Goerner of Valtech Scrum teams System team
- 18. 4/26/2014 16 PG Enterprise Validation Agile & Waterfall Converged Model Dell Confidential31 SST Hardware Product Test EST Final EST NEST 1 Sprint1 Sprint2 Sprint N-1 SprintN Sprint1 Sprint2 Sprint N-1 SprintN Sprint1 Sprint2 Sprint N-1 SprintN Hardening Stability Sprint ExtendedSprints SW Regression Test Solution stack with software focus SST EST Exit CCB Review Code Freeze A-Rev Final Extended Regression F V PT Stability SST A-Rev Candidate SSI Stability Sprint RC-0 Regression Regression HW-Specific User Stories Complete HW Enablement Features Complete Vendor Deliverables Required 100% PT Attempted All Stories Accepted Feature Complete X-on-A Stories Accepted PG Enterprise Validation Agile @ Dell Geography 32 • Co-location of a Scrum team is always preferred • Limit geographical distribution of Scrum team to no more than two time-zones • Scrum teams within a project can be distributed
- 19. 4/26/2014 17 33 Automating in an Agile World Opportunities for Automation • It’s not just about automating test cases • Can also provide efficiencies to: –Test Preparation, Setup and Configuration –Non-Functional Testing (“ility”) › Longevity, Scale and Performance Characterization –Compatibility Testing (Solution and Device) 34
- 20. 4/26/2014 18 Test Preparation • Bare-metal Deployment • Setup & Configuration – OS Provisioning and Configuration • Test case staging • Environment Cleanup/Baseline • Virtualization-based test environments 35 Test Content Development • UI automation on Customer Usage workflows 36 CAUTION: Application Architecture can be an enabler or inhibitor • Unit Test Automation • In-Sprint, Automated Acceptance Tests • Web-services (or CLI) Test Automation for Functional Testing • Automated CI (UT, Build, BVT) 2 - Agile Testing: A Practical Guide for Testers and Agile teams 2
- 21. 4/26/2014 19 Application Architecture Matters • Does it facilitate testing at the API/Services level? • Does the Business/Error Logic reside below the services level? • Does the UI architecture support test automation other than record-playback? 37 Large Scale Organizations: • Standardize UI architectures across the product portfolio • HTML5, Flex, Silverlight… • Standardize Service architectures across the product portfolio • SOAP, REST, CLI, API Non-Functional Testing • Performance Characterization • Longevity • Stress • Scale • Concurrency 38 Often times analysis of these areas are simply too cost-prohibitive to be done manually 3- http://lisacrispin.com/wordpress/2011/11/08/using-the-agile-testing-quadrants/ 3
- 22. 4/26/2014 20 Confidential39 Functional Validation Performance- Characterization and non-Functional Validation Longevity 11 M2 1.0 SST Baseline M1 Scale Concurrency Simulation L a r g e - S c a l e L a r g e - S c a l e 12 13 17 18 19 Sprint 15 Performance Characterization Timeline 14 16 Sprint 21 Key Dependency: • 3rd party HW available • 3rd party user stories completed The Lessons of ‘Large’ 40 Courtesy: Gulliver's Travels (2010)
- 23. 4/26/2014 21 The Lessons of ‘Large’ 1. Co-located Teams 2. Project-wide ‘Done’ Criteria 3. Culture of Automation 4. System Test Team 5. Test Architect 6. ALM Tooling Questions? 42
- 24. 4/26/2014 22 Resources • Agile Manifesto - http://agilemanifesto.org/ • Articles/Blogs: – http://support.smartbear.com/articles/testcomplete/automated-testing-agile-environment/ - SmartBear – http://lisacrispin.com/wordpress/2011/11/08/using-the-agile-testing-quadrants/ - Lisa Crispin – http://www.stickyminds.com/sitewide.asp?Function=edetail&ObjectType=COL&ObjectId=177 93&tth=DYN&tt=siteemail&iDyn=2 – Rajini Padmanaban – Beyond Agile Testing: http://www.utest.com/int-v1/a/beyond-agile-testing - uTest – Testing and Checking Refined: http://www.satisfice.com/blog/archives/856 - James Bach and Michael Bolton • Books: – Scaling Software Agility: Best Practices for Large Enterprises – Dean Leffingwell – How Google Tests Software – James Whittaker, Jason Arbon, Jeff Carollo – Agile Testing: A Practical Guide for Testers and Agile teams – Lisa Crispin, Janet Gregory – http://www.dorothygraham.co.uk/automationExperiences/index.html - Dorothy Graham • Presentations: – Approaches to Software Testing: An Introduction – Scott Barber – Scaled Agile Framework Big Picture – Dean Leffingwell 43 Backup 44