A Funny Thing Happened on the Way to User Acceptance Testing
- 1. T1 Test Management 5/8/2014 9:45:00 AM A Funny Thing Happened on the Way to User Acceptance Testing Presented by: Randy Rice Rice Consulting Services, Inc. Brought to you by: 340 Corporate Way, Suite 300, Orange Park, FL 32073 888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com
- 2. Randy Rice Rice Consulting Services, Inc. A leading author, speaker, and consultant with more than thirty years of experience in the field of software testing and software quality, Randy Rice has worked with organizations worldwide to improve the quality of their information systems and optimize their testing processes. He is coauthor (with William E. Perry) of Surviving the Top Ten Challenges of Software Testing and Testing Dirty Systems. Randy is an officer of the American Software Testing Qualifications Board (ASTQB). Founder, principal consultant, and trainer at Rice Consulting Services, Randy can be contacted at riceconsulting.com where he publishes articles, newsletters, and other content about software testing and software quality. Visit Randy’s blog.
- 3. 4/26/2014 1 A FUNNY THING HAPPENED ON THE WAY TO THE ACCEPTANCE TEST RANDALL W. RICE, CTAL RICE CONSULTING SERVICES, INC. WWW.RICECONSULTING.COM 2 THIS PRESENTATION • The account of four different acceptance tests, in three organizations. • The names have been withheld and the data generalized to protect privacy. • One project was in-house developed and the other three were vendor- developed systems. • So, a more traditional UAT approach was taken.
- 4. 4/26/2014 2 3 A COMMON PERCEPTION OF UAT • UAT is often seen as that last golden moment or phase of testing, where • Users give feedback/acceptance • Minor problems are identified and fixed • The project is implemented on time • High fives, all around 4 IN REALITY… • UAT is one of the most risky and explosive levels of testing. • UAT is greatly needed, but happens at the worst time to find major defects – at the end of the project. • Users may be unfriendly to the new system • They like the current one just fine, thank you. • Much of your UAT planning may be ignored. • People tend to underestimate how many cycles of regression testing are needed.
- 5. 4/26/2014 3 5 THERE ARE MANY QUESTIONS ABOUT UAT • Who plans it? • Who performs it? • Should it only be manual in nature? • What is the basis for test design and evaluation? • When should it be performed? • Where should it be performed? • Who leads it? • How much weight should be given to it? 6 PROJECT #1 • Medical laboratory testing business that closely resembles a manufacturing environment. • New Technology for the company and for the industry. • The previous project had failed • The company almost went out of business because of it! • Very high growth in terms of both business and employees. • Company at risk of failure. • This project was truly mission-critical.
- 6. 4/26/2014 4 7 PROJECT #1 – CONT’D. • Few functional requirements. • 8 pages for over 400 modules • Test team had little knowledge of: • subject matter, • test design, • or testing at all. • Very little unit or integration testing being done by developers. • Some system testing was done. • UAT was the focus. 8 DEFECT DISCOVERY AND BACKLOG System Test UAT 1st Deploy 2nd Deploy 3rd Deploy 4 weeks 3 weeks
- 7. 4/26/2014 5 9 PROJECT #1 RESULTS • Very high defect levels in testing. • Many were resolved before implementation. • Severe performance problems. • Old system could process 8,000 units/day • New system could process 400 units/day • Many problems due to the new technology being used • “bleeding edge” issues • “Deadline or else” attitude • The business was under extreme pressure to deploy due to increased processing volumes. • System was de-installed/re-installed 3 times before performance was acceptable to deploy. 10 WHAT WE LEARNED • Requirements are important. • Even if you have to create some form of them after the software has been written. • Early testing is important. • That would have caught early performance bottlenecks. • Teamwork is critical. • Things got so bad we had to have a “do over.” • The deadline is a bad criteria for implementation. • Always have a “Plan B”.
- 8. 4/26/2014 6 11 UAT LESSONS • Build good relationships with subject matter experts. • They often determine acceptance • Listen to the end-users. • Understand what’s important • Don’t rely on UAT for defect detection. • Interesting factoid • A similar project with the exact same technology failed due to performance errors two years later for a city water utility. $1.8 million lawsuit lost by the vendor. 12 PROJECT #2 • Same company as before, but two years later • Integration of a vendor-developed and customized accounting system • Lots of defects in the vendor system • Implemented two months late with practically zero defects.
- 9. 4/26/2014 7 13 WHAT MADE THE DIFFERENCE? • Same people – testers, IT manager, developer • Different project manager who was a big supporter of testing • More experience with the technology • Better understanding of testing and test design • A repeatable process • Less pressure to implement • Having a contingency plan • Having the courage to delay deployment in favor of quality. • The financials had to be right. 14 PROJECT #3 • New government-sponsored entity. • Everything was new – building, people, systems • System was a vendor-developed workers compensation system. • Some customization • Little documentation • Designed all tests based on business scenarios. • We had no idea of the UI design.
- 10. 4/26/2014 8 15 KEY FACTORS • No end-users in place at first to help with any UAT planning. • In fact, we had to train the end-users in the system and the business. • Lots of test planning was involved • 50% or more effort in planning and optimizing tests. • This paid off big in test execution and training 16 RESULTS • Tested 700 modules with 250 business scenario tests. • We had designed over 300 tests • The management and test team felt confident after 250 tests we had covered enough of the system. • Found many defects in a system that had been in use in other companies for years. • Reused a lot of the testware as training aids. • Successful launch of the organization and system.
- 11. 4/26/2014 9 17 HARD LESSONS LEARNED • “You don’t know what you don’t know” AND “You sometimes don’t know what you think you know.” • Newly hired SME with over 30 years workers comp experience provided information that was different (correct) than what we had been told during test design. • We had to assign two people for two weeks to create new tests. • These were complex financial functions – we couldn’t make it up on the fly. 18 HARD LESSONS LEARNED (2) • Real users are needed for UAT. • Sometimes the heavy lifting of test design may be done by other testers, but users need heavy involvement.
- 12. 4/26/2014 10 19 PROJECT #4 • State government, Legal application • Vendor-developed and customized • Highly complex system purchased to replace two co- existing systems. • Half of the counties in the state used one system, the other half used another. • Usability factors were low on the new system • Data conversion correctness was critical 20 THE GOOD SIDE • Well-defined project processes • Highly engaged management and stakeholders • Good project planning and tracking • Incremental implementation strategy • The entire system was implemented, only one county at a time. • Heavy system testing • Good team attitude
- 13. 4/26/2014 11 21 THE CHALLENGES • The system’s learning curve was very high. • The key stakeholders set a high bar for acceptance. • The actual users were few in number and were only able to perform a few of the planned tests. • Very high defect levels. 22 LEADING UP TO VENDOR SELECTION • Over 2 years of meeting with users and stakeholders to determine business needs. • Included: • JAD sessions • Creation of “as-is” and “to-be” use cases • Set of acceptance criteria (approximately 350 acceptance criteria items)
- 14. 4/26/2014 12 23 THE STRATEGY • Create test scenarios that described the trail to follow in testing a task, but not to the level of keystrokes. • Based on use cases. • The problem turned out to be that even the BAs and trainers had difficulty in performing the scenarios. • System complexity was high. • Training had not been conducted. • Usability was low 24 DEFECT DISCOVERY AND BACKLOG System Test UAT 1st Deploy 10 weeks 4 weeks 750 250
- 15. 4/26/2014 13 25 WHAT WAS VALIDATED • The precise “point and click” scripts provided by the vendor were long and difficult to perform. • Each one took days. • Plus, there were errors in the scripts and differences between what the script indicated and what the system did. 26 THE BIG SURPRISES • We planned the system test to be a practice run for UAT. • It turned out to be the most productive phase of testing in terms of finding defects. • We planned for a 10 week UAT effort with 10 users • It turned out to be a 2 week effort with 4 users. • First sense of trouble: initial users were exhausted after 3 days of a pre-test effort.
- 16. 4/26/2014 14 27 THE BIG SURPRISES (2) • We used none of the planned tests (around 350 scenarios) in UAT. • Instead, it was a guided “happy path” walkthrough, noting problems along the way. • Defects were found, but the majority of defects had been found in system test. 28 LESSONS LEARNED • The early system test was invaluable in finding defects. • Learning the system is critical for users in new systems before they are able to test it. • The test documentation is not enough to provide context of how the system works. • It took a lot of flexibility on the part of everyone (client, vendor, testers, users, stakeholders) to make it to the first implementation. • Sometimes actual users just aren’t able to perform a rigorous test.
- 17. 4/26/2014 15 29 WHAT CAN WE LEARN FROM ALL THESE PROJECTS? • UAT is a much-needed test, but happens at the worst possible time – just before implementation. • You can take some of the late defect impact away with system testing and reviews. • You can lessen the risk of deployment by implementing to a smaller and lower risk user base first. • Actual end-users are good for performing UAT, but much depends on what you are testing and the capabilities of the users. • The reality is the users are going to have to use the system in real-life anyway. • However, not all users are good testers! 30 WHAT CAN WE LEARN? (2) • Be careful how much time and effort you invest in planning for UAT before the capabilities are truly known. • That is, senior management may want actual users to test for 8 weeks, but if the people aren’t available or can’t handle the load, then it probably isn’t going to happen. • Don’t place all the weight of testing on UAT. • In project #4 our system testing found a majority of the defects.
- 18. 4/26/2014 16 31 WHAT CAN WE LEARN? (3) • UAT test planning isn’t bad, just expect changes. • People, software, business, timelines – they all change. • Try to optimize and prioritize. • Example: If you have 500 points of acceptance criteria, can they be validated with 200 tests? • Which of the acceptance criteria are critical, needed and “nice to have”? 32
- 19. 4/26/2014 17 33 BIO - RANDALL W. RICE • Over 35 years experience in building and testing information systems in a variety of industries and technical environments • ASTQB Certified Tester – Foundation level, Advanced level (Full) • Director, American Software Testing Qualification Board (ASTQB) • Chairperson, 1995 - 2000 QAI’’’’s annual software testing conference • Co-author with William E. Perry, Surviving the Top Ten Challenges of Software Testing and Testing Dirty Systems • Principal Consultant and Trainer, Rice Consulting Services, Inc. 34 CONTACT INFORMATION Randall W. Rice, CTAL Rice Consulting Services, Inc. P.O. Box 892003 Oklahoma City, OK 73170 Ph: 405-691-8075 Fax: 405-691-1441 Web site: www.riceconsulting.com e-mail: rrice@riceconsulting.com