Best Practices for Benchmarking the Website User Experience featuring MeasuringU's Jeff Sauro
11 likes3,154 views
The document outlines best practices for benchmarking website user experience, emphasizing the importance of measuring both before and after implementing changes. Key recommendations include collecting data at various levels, using standardized metrics, considering user experience, and ensuring sufficient sample sizes for reliable results. The ultimate goal is to drive continuous improvement through regular assessment and adjustment of user experience metrics.
![2015 MEASURING U
SUPR-Q items & Factors
19
USABILITY CREDIBILITY (TRUST & COMFORT)
LOYALTY APPEARANCE
• It is easy to navigate with this website
• This website is easy to use
• I will likely visit the website in the future
• How likely are you to recommend this website to
a friend or colleague? [NPS]
• The information on this website is credible.
• The information on this website is trustworthy.
• The website has a clean and simple presentation
• I found the website to be attractive
Standardized User Experience Percentile Rank-Questionnaire
Sauro, J. (2015) "SUPR-Q: A Comprehensive Measure of the Quality of the Website User Experience" in Journal of Usability
Studies Issue 2, Vol. 10, Feb. 2015, pp. 68-86](https://image.slidesharecdn.com/benchmarkingwebinarv2-151022013359-lva1-app6891/85/Best-Practices-for-Benchmarking-the-Website-User-Experience-featuring-MeasuringU-s-Jeff-Sauro-19-320.jpg)
Best Practices for Benchmarking the Website User Experience featuring MeasuringU's Jeff Sauro
- 1. Best Practices For Benchmarking the Website User Experience Jeff Sauro
- 2. 2015 MEASURING U RAPIDLY TEST USABILITY & MEASURE UX UserZoom is the All-in-One UX Research Platform for the Data-Driven Enterprise
- 3. 2015 MEASURING U • Chat box is available if you have any questions • Look out for a quick poll question • There will be time for Q&A at the end • We will be recording the webinar for future viewing • All attendees will receive a copy of the slides/recording • Continue the discussion using #UZWebinar Quick Housekeeping
- 4. 2015 MEASURING U About MeasuringU MeasuringU is a UX research firm based in Denver, Colorado USA focusing on the analysis of human behavior and quantifying the user experience. Since 2004, we've been helping clients answer questions such as: • How usable is an application? • What is the probability a user will purchase a product? • What are the interaction problems with the mobile application? • How large of a sample size do we need? • Which design has a statistically higher conversion rate? • What are the problems in our website navigation?
- 5. 2015 MEASURING U Who we work with 5
- 6. 2015 MEASURING U Popular Services: Quant + Qual 6 Measurement Strategy & Statistical Analysis Unmoderated Remote StudiesInformation Architecture/ Findability Testing Eye-Tracking Studies Moderated Lab Based Usability Studies Mobile Usability Testing
- 7. 2015 MEASURING U Why Benchmark? 7 Measure Before Measure After Make Changes Measure, Analyze & Improve Did improvements make a difference?
- 8. 2015 MEASURING U Why Competitive Benchmarking? 8 Makes interpreting the subsequent data easier: the focus becomes less on absolute scores and more on relative comparisons. Less of a focus on problems with task scenarios and more on comparison Answers the question: “Compared to What” VS.
- 9. 2015 MEASURING U UX Benchmarking Process 9 Planning Conducting Analyzing 1. Identify users 2. Finding users 3. Define tasks 4. Comparative vs. Standalone 5. Study metrics 6. Task metrics 7. Sample size 1. Pilot and Program 2. Field the study 1. Summary 2. Study Overview 3. Task Metrics 4. Test Metrics 5. “Why” 6. Click-Paths 2 to 8 Weeks
- 10. 2015 MEASURING U 10 Best Practices for Competitive Benchmarking 1. Collect Data at the Task and Study Level 2. Measure Website & Brand Attitudes 3. Use Standardized Metrics Where Possible 4. Consider Prior Experience 5. Have Task Success Criteria 6. Decide Between Within- vs Between- Subjects 7. Have a Sufficient Sample Size 8. Measure Preference 9. Compare to Other Reported Measures 10.Rinse & Repeat 10
- 11. 2015 MEASURING U 1. Collect Data at the Task and Study Level 11 Task Level • Completion Rates • Task Time • Perceived Ease Study Level • Overall measure of quality • Loyalty • Usability System Usability Scale (SUS) • Brand Affinity • Likelihood to purchase • Preference Collect a mix of attitudes and actions before and after the tasks
- 12. 2015 MEASURING U Task Completion Rates 12 • CODE TASK SUCCESS AS 1 • CODE TASK FAILURE AS 0 • Define a success criteria and do not give partial credit in the computations
- 13. 2015 MEASURING U Task Time 13 • AVERAGE TASK COMPLETION TIME • MEAN TIME TO FAILURE • AVERAGE TIME ON TASK
- 14. 2015 MEASURING U Task Ease 14 SINGLE EASE QUESTION (SEQ) Overall, this task was? 1 2 3 4 5 6 7 Very Difficult Very Easy
- 15. 2015 MEASURING U 2. Measure Website & Brand Attitudes • Unaided Branding • When thinking about laptop computers, which brands come to mind? • What car rental companies can you think of? • Aided Branding • Which of the following hotel booking websites are you familiar with? • Hotels.com • Blozert.com • Expedia.com • Brand Favorability & Overall Attitude • How would you rate your attitude toward the following brands? 15 HierarchyofEffects
- 16. 2015 MEASURING U Interaction of prior attitude on scores Different experiences for the most and least favorable users Most favorable users actually get less favorable -4% Least favorable users actually get more favorable +18% Most Favorable Least Favorable Before After
- 17. 2015 MEASURING U Measure Attitude Lift 17 Change in attitude “Lift” can be more informative than absolute attitude 4.7 5.35.3 4.5 1 2 3 4 5 6 7 Budget Enterprise VeryFavorableVeryUnfavorable 12% 15%
- 18. 2015 MEASURING U 3. Use Standardized Metrics Where Possible 18 SUS SEQ
- 19. 2015 MEASURING U SUPR-Q items & Factors 19 USABILITY CREDIBILITY (TRUST & COMFORT) LOYALTY APPEARANCE • It is easy to navigate with this website • This website is easy to use • I will likely visit the website in the future • How likely are you to recommend this website to a friend or colleague? [NPS] • The information on this website is credible. • The information on this website is trustworthy. • The website has a clean and simple presentation • I found the website to be attractive Standardized User Experience Percentile Rank-Questionnaire Sauro, J. (2015) "SUPR-Q: A Comprehensive Measure of the Quality of the Website User Experience" in Journal of Usability Studies Issue 2, Vol. 10, Feb. 2015, pp. 68-86
- 20. 2015 MEASURING U Selection of websites in SUPR-Q database 20 Retail Amazon eBay Zappos Apple Target JC Pennys Wal-Mart Container Store Crate & Barrel Crumpler Oriental Trading Sears LL-Bean Michaels Pier 1 Imports Macys Netflix BestBuy OfficeMax OfficeDepot Airlines United Airlines Frontier Airlines American Airlines Southwest Jet Blue Delta Frontier 3rd Party Travel Travelocity Expedia Orbitz Kayak Government Illinois Colorado California CDC USA.gov New York State Floral Service FTD 1-800 Flowers ProFlowers 3rd Party Automotive Edmunds Kelly Blue Book Cars Autotrader Financial Services PayPal Fidelity TD Ameritrade Vanguard Bellco US Bank Chase eTrade Scrottrade Wells Fargo News & Information Wall Street Journal CNN FoxNews New York Times IMDB Yelp Craigslist Motley Fool Yahoo Social Networking Facebook Twitter LinkedIn Flickr Relationship/Dating Match.com eHarmony PlentyofFish.com Hotels Hyatt Hilton Cell Phone Carriers AT&T Wireless Sprint T-Mobile • 200+ Total Websites • 10,000 responses • 30-200 per website
- 21. 2015 MEASURING U SUPR-Q “UX Website Quality” 2013 21 88% 91% 99% 89% 90% 98% 98% 87% 79% 60% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Industry Average = 88%
- 22. 2015 MEASURING U Systems Usability Scale (SUS) 22 51 2 3 4 Strongly Disagree Strongly Agree I think that I would like to use this system frequently I found the system unnecessarily complex I thought the system was easy to use I think that I would need the support of a technical person to be able to use this system I found the various functions in this system were well integrated I thought there was too much inconsistency in this system I would imagine that most people would learn to use this system very quickly I found the system very cumbersome to use I felt very confident using the system I needed to learn a lot of things before I could get going with this system
- 23. 2015 MEASURING U Raw SUS Scores & Percentile Ranks 23 Average SUS Score is a 68 (500 products) For Websites, 2 SUPR-Q Usability Items Predict Most (64%+) of SUSs 10 Items
- 24. 2015 MEASURING U 4. Consider Prior Experience 24 Repeat users rated websites as 11% more usable than first-time users 61 67 50 55 60 65 70 75 80 85 90 First Time User Repeat User n=764 n=422 SUSScore 11%
- 25. 2015 MEASURING U Brand Attitude & Lift by Experience 5.66 6.395.76 6.38 1 2 3 4 5 6 7 Low Experience High Experience Favorability(LowtoHigh) Before After Once – 5 times 6 times – 10+ times 11%
- 26. 2015 MEASURING U Net Promoter Score (NPS) By Site Experience 26 -11% 19% 51% 62% -2% 21% 62% 70% -60% -40% -20% 0% 20% 40% 60% 80% 100% 0 times 1 to 6 times 6 to 10 times 10+ times 2011 2013 Analysis by Experience, Weight or Statistically Control (Type I ANOVA)
- 27. 2015 MEASURING U 5. Have Task Success Criteria 27 Spend a few minutes on the website to get to know what it has to offer. Select a product that interests you and add it to your cart. Find a toaster that has been reviewed by at least 30 people so you know what to expect when you purchase it. You also want it to have at least a 4-star rating. You would like to spend no more than $100.00 on your toaster. VS Open: Open tasks are fine, but closed-ended tasks with a solution often better differentiate between competitors. Closed:
- 28. 2015 MEASURING U 6. Within- vs Between- Subjects 28 Factor to Consider Between Within Sample Size & Power - + Carryover Effects + - Impact on Attitudes + - Comparative Judgment - + Study Duration + -
- 29. 2015 MEASURING U 7. Have a Sufficient Sample Size 29 Difference to Detect 90% Confidence & 80% Power Between-Subjects Sample Size Within-Subjects Sample Size 20% 150 50 10% 614 115 5% 2468 246 4% 3860 312 3% 6866 421 2% 15452 640 1% 61822 1297 See Chapter 6 in Quantifying the User Experience
- 30. 2015 MEASURING U Stand Alone Benchmark Sample Size 30 Sample Size Margin of Error (+/-) 10 27% 20 20% 30 17% 40 15% 50 13% 90 10% 115 9% 150 8% 200 7% 260 6% 380 5% 600 4% 1000 3%
- 31. 2015 MEASURING U 8.Measure Preference 31 87% 13% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% WebsitePreference For Statistical Significance See Measuringu.com/blog/preference-data.php
- 32. 2015 MEASURING U 9.Compare to Other Reported Measures 32 Available At Measuringu.com/products.php
- 33. 2015 MEASURING U 10. Rinse & Repeat 33 TaskCompletionRate If the ultimate goal of benchmarking is to drive improvement, then measure overtime 73% 71% 64% 67% 76%85% 82% 72% 60% 68%79% 60% 69% 57% 86% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
- 34. 2015 MEASURING U SUPR-Q: 2011 vs 2013 34 Did the Quality of the Website Improve After the Redesign? 77% 73% 86% 72% 41% 29%81% 76% 79% 75% 75% 35% SUPR-Q Usability Credibility Loyalty Appearance NPS (Raw %) 2011 2013
- 35. 2015 MEASURING U Best Practices for Competitive Benchmarking 1. Collect Data at the Task and Study Level 2. Measure Website & Brand Attitudes 3. Use Standardized Metrics Where Possible 4. Consider Prior Experience 5. Have Task Success Criteria 6. Decide Between Within- vs Between- Subjects 7. Have a Sufficient Sample Size 8. Measure Preference 9. Compare to Other Reported Measures 10. Rinse & Repeat 35 @MeasuringU www.MeasuringU.com #UZWebinar
- 36. 2015 MEASURING U36 Questions Continue the discussion using #UZWebinar
- 37. 2015 MEASURING U Need more info? Get answers about UserZoom. We are here to help. Talk with one of our knowledgeable specialists. Contact us UserZoom 1054 South De Anza Blvd, Suite 102 San Jose, CA 95129 Toll free number: (866) 599 1550 www.linkedin.com/company/userzoom@UserZoomFollow us