Useagility Webinar - Automated User Testing
- 1. 4149 Pennsylvania, Suite 202 Kansas City, MO 64111 Automated User Testing Presented by Chrys Sullivan Founder and CEO, Useagility
- 2. When We’re Finished, You Should: 2 Understand how unmoderated user testing works and how you could use it in your organization. 01 Be able to identify when unmoderated testing is and is not the best testing methodology. 02 Know our tips for avoiding the most common pitfalls in automated testing. 03 Understand primary differences between four top automated testing tools. 04
- 4. Our team of experienced professionals draw from a wide range of skills, processes and tools to across the spectrum of user experience research and design services. Each client project uses different design and research tools and process depending on the project needs Our Services User Research Experience Strategy Design Front-End Development Tap into customers and end users to inform strategy and design Envision the future of your digital business strategy Dive into complex UI challenges; simplify, streamline and get results Bring design to life across all devices
- 5. Evolution of Usability Testing Technology 5 Lab-based testing - Moderated user testing, in lab setting Remote, facilitated testing - Moderated testing conducted remotely via web-conference and screen sharing technology Automated, un-facilitated testing - Unmoderated studies conducted online via web-based applications
- 6. Benefits of Unmoderated 6 Easier recruitment process Recruiting Scheduling No worries about no-shows – or replacements Automated incentive payments Simultaneously test large number of participants Less expensive per-study costs than moderated testing Get results quicker than moderated Built-in metrics (time on task, rating scale, etc.)
- 7. Limitations & Pitfalls of Unmoderated 7 User panels and “professional testers” Stakeholders are not involved as observers during the study Can’t revise tasks / questions once the study has been launched Can’t adjust study approach as sessions progress Lacks ability to probe unexpected insights No real-time support for users who get significantly off-track Greater likelihood of prototype-related issues causing task error
- 8. 8
- 9. 9
- 10. 10
- 11. 11
- 12. 12
- 13. Selecting Right Methodology 13 Moderated or Un-moderated? Choosing the right methodology has the greatest impact on your ability to meet expectations with the outcomes of the study.
- 14. First Step: Define Goals for Testing 14 Hard to get business stakeholders to define their goals. Ask them questions: • What is the primary purpose of the study? • What do you need to know coming out of this study? • What are you hoping to do with the study findings?
- 15. Second Step: Define Additional Details 15 After goals are defined, understand other key components: • What is the scope? (Big, small, how focused?) • What are we testing with? (Prototype, wireframes, live site) • Who are the target users? • Which kinds of user scenarios and tasks will help achieve study goals?
- 16. Step 3: Moderated or Unmoderated? 16 Are follow-up questions necessary? Are you in concept/strategy phase? Are you dealing with a limited-functionality prototype? Will it take longer than 20 – 30 minutes to complete tasks? Yes = Moderated
- 17. Step 3: Moderated or Unmoderated? 17 Is the scope focused? Can testing tasks be accomplished in 30 min or less? Are there clear, specific tasks that can be tested? Is testing of task success and usability a primary objective? Is there minimal reliance on attitudinal feedback that requires follow up probing? Yes = Unmoderated
- 18. Which Tool is Right for My Needs? 18
- 19. Which Tool is Right for My Needs? 19 Features Audio/Video Capture Metrics and Data Mobile Native App Provider has User Panel Use My Users Not easily Automated Incentive Flexible Incentives PayPal only PayPal only Consulting Services Pricing $350 per project - up to 1,000 participants per project $50 per study, minimum of 150 purchased upfront ($7,500) $350 set - $49 per project $12,000 + minimum per year
- 20. 20 Pros • Most economic for conversion testing on large sample size • Reliable, time-tested platform • Webcam to see user’s face • Lower entry-level pricing • Ability to share videos easily • Flexible incentives • Full-service consulting support Cons • No video • Reliant on self-reported task success and analytics • Inability to easily recruit from intercepts or own user group • Set incentive amounts • Panel size may be smaller than UT.com • Expensive • May be overkill for some companies who just need task-based testing at affordable price point BottomLine • Best option for high- volume conversion flow testing • No video; only metrics • Least expensive • Easy study set-up • Large panel • Good for “do-it- yourself” testing • Newer, user friendly interface • Lower pricing point to get started • Good for “do-it- yourself” testing • Easy to share clips and full video files • The “Cadillac” • Highest entry price • Not best fit for “do-it- yourself” • Push the consulting services
- 21. Which Tool is Right for My Needs? 21 • Best option for high- volume conversion flow testing • No video; only metrics • Least expensive Pricing $350 per project - up to 1,000 participants per project What’s captured? Metrics and data, no video Features Clickstream analysis, heat mapping, task success, abandonment analysis Mobile? Mobile browser enabled, not able to test native apps Recruitment options Panel, intercept from your site, email distribution Automated incentives? Yes, if their panel is used Incentive types Incentives can be set for any amount; Gift cards, Lotteries, bank transfers, PayPal transfers, and more. Pros Most economic for conversion testing on large sample size Cons No video, reliant on self-reported task success and analytics
- 22. Which Tool is Right for My Needs? 22 • Easy study set-up • Large panel • Good for “do-it- yourself” testing Pricing $50 per study, minimum of 150 purchased up front ($7,500) What’s captured? Metrics and video/audio Features Audio/video capture, exit survey, net-promoter scores, metrics, time- on-task, consulting services to run studies Mobile? Desktop, tablet and phone in browser App must be downloaded for native app testing (ios and Android) Recruitment options Panel - Can’t use your own participants without them having to be added to the UserTesting panel Automated incentives? Yes, if their panel is used Incentive types A set incentive of $10 for every 20 minutes; PayPal Pros Reliable, time-tested platform Cons • Inability to recruit from intercepts or own user group without users being added to their panel and going through “training” • Set incentive amounts
- 23. Which Tool is Right for My Needs? 23 • Newer, more user friendly interface compared to UT.com • Lower pricing point to get started • Good for “do-it- yourself” testing • Video share feature makes easy to share clips and full video files Pricing $350 set up - $49 per user session What’s captured? Metrics and video/audio Features Audio/video capture, webcam to see participant’s face, exit survey, metrics, time-on-task, consulting services to run studies Mobile? Desktop, tablet and phone in browser App must be downloaded for native app testing (ios and Android) Recruitment options Panel or use own users from intercept, email distribution, etc. Automated incentives? Yes, if their panel is used Incentive types $1 - $100 via PayPal Pros Webcam to see user’s face, lower entry-level pricing, ability to share videos with stakeholders and clients easily through links. Cons • Panel size may be smaller than UT.com
- 24. Which Tool is Right for My Needs? 24 • The “Cadillac” • Highest entry price • Not best fit for “do- it-yourself” • Push the consulting services Pricing $12,000 + minimum per year What’s captured? Metrics and video/audio Features Audio/video capture, UX benchmarking, card sorts, tree testing, metrics, heat mapping, click test, consulting services to run studies Mobile? Desktop, tablet and phone in browser App must be downloaded for native app testing (iOS and Android) Recruitment options Panel, or use own users from intercept, email distribution, etc. Automated incentives? Yes, if their panel is used Incentive types Any amount, many different payment method options (Paypal, gift certificates, bitoins Pros Flexible incentives, full-service consulting support Cons Expensive, may be overkill for some companies who just need task- based testing at affordable price point
- 25. Conduct Pilot Session Tips for Avoiding Pitfalls of Unmoderated Testing 01 02 03 04 06 1. Conduct one session as pilot 2. Watch results & adjust tasks and flow 25 Verify Participant Profiles 1. Use screener questions 2. Ask users to give you verbal answers to questions that confirm their profile Ask follow ups 1. Ask probing questions after tasks 2. Use task metrics to prompt follow up questions 03 Question self- reported success 1. Self— reported task success is unreliable. Users don’t want to “mess up” 2. Measure actual success vs. reported success 04 05 Keep tasks simple 1. Keep tasks simple; users skim task instructions
- 26. Our Favorite Ways to Use Unmoderated Testing 26 As quick way to test usability in detailed design phase Use Axure, InVision prototypes Can also use static wireframes, videos and other design files as stimulus During agile design & development sprints to test wireframes and prototypes with users before development begins To test effectiveness of one design approach vs. another To obtain user preference for one design over another To validate suspected usability issues or support Expert Review findings
- 28. Now You Know 28 How unmoderated user testing works and how you could use it in your organization. 01 Identify when unmoderated testing is and is not the best testing methodology. 02 Quick tips for avoiding the most common pitfalls in automated testing. 03 The primary differences between four top automated testing tools. 04
- 29. Thank You 29 Thank you for joining us today We would love to work with you on your next project Connect with us www.Useagility.com info@useagility.com Watch for our newsletter to learn about upcoming webinar topics