Usability evaluations (part 2)
Download as PPT, PDF1 like730 views
The document discusses usability testing and inquiry methods. It provides details on conducting a usability test of the OpenSMSDroid Android app, including testing configurations, representative tasks, and data collection and analysis. It also summarizes findings from a usability study of various websites reported in the book Prioritizing Web Usability.
Usability evaluations (part 2)
- 1. IM2044 – Week 8: Lecture Dr. Andres Baravalle 1
- 2. Interaction design • The next slides are (very loosely) based on the companion slides for the textbook • By the end of this week, you should have studied all chapters of the textbook up to chapter 11 • Today we will covering chapter 14. 2
- 3. Outline • Usability testing (part 2) – Usability testing scenarion: OpenSMSDroid • Usability inquiry 3
- 4. Usability testing • When: common for comparison of products or prototypes • Tasks & questions focus on how well users perform tasks with the product – Focus is on time to complete task & number & type of errors • Data collected by video & interaction logging • Experiments are central in usability testing – Usability inquiry tends to use questionnaires & interviews 4
- 5. Testing conditions • Usability lab or other controlled space • Emphasis on: – Selecting representative users – Developing representative tasks • Small sample (5-10 users) typically selected • Tasks usually last no longer than 30 minutes • The test conditions should be the same for every participant 5
- 6. Some type of data • Time to complete a task • Time to complete a task after a specified time away from the product • Number and type of errors per task • Number of errors per unit of time • Number of navigations to online help or manuals • Number of users making a particular error • Number of users completing task successfully 6
- 7. How many participants is enough for user testing? • The number is a practical issue • Depends on: – Schedule for testing – Availability of participants – Cost of running tests • Typically 5-10 participants – Some experts argue that testing should continue with additional users until no new insights are gained 7
- 8. Experiments & usability testing • An experiment is “a method of testing with the goal of explaining - the nature of reality” (Wikipedia, 2011) • Experiments test hypotheses to discover new knowledge by investigating the relationship between two or more things – i.e., variables. – Experiments may used in usability testing 8
- 9. Usability testing & research • • • • • • Usability testing Improve products Few participants (typically) Results inform design Conditions controlled as much as possible Procedure planned Results reported to developers • • • • • • 9 Experiments for research Discover knowledge Many participants Results validated statistically Strongly controlled conditions Experimental design Scientific report to scientific community
- 10. Experiments • Predict the relationship between two or more variables. • Independent variable is manipulated by the researcher – Dependent variable depends on the independent variable – Typical experimental designs have one or two independent variable • Validated statistically & replicable 10
- 11. Experimental designs • Different participants - single group of participants is allocated randomly to each of the experimental conditions – Different participants perform in different conditions • Same participants - all participants appear in both conditions • Matched participants - participants are matched in pairs, e.g., based on expertise, gender, etc. 11
- 12. Different, same, matched participant design Design Advantages Disadvantages Different No order effects Many subjects & individual differences a problem Same Few individuals, no individual differences Counter-balancing needed because of ordering effects Matched Same as different participants but individual differences reduced Cannot be sure of perfect matching on all differences 12
- 13. Examples • The next slides describe 2 experiments: the one behind the book Prioritizing Web Usability and a fictional one on OpenSMSDroid • Both use Thinking Aloud and video/screen recording for data collection 13
- 14. Prioritizing Web Usability • Prioritizing Web Usability (Nielsen and Loranger, 2006) used the Thinking Aloud method to collect insight on user behaviour: – 69 users, all with at least one year experience in using the web – Broad range of job backgrounds and web experience – but no one working in IT or marketing – 25 web sites tested with specific tasks – Windows desktops with 1024x768 resolution running Internet Explorer – Recordings of monitor and upper body for each session – Broadband speed between 1 and 3 Mbps 14
- 15. Prioritizing Web Usability (2) • The tasks that the users were asked to perform included: – Go to ups.com and find how much does it cost to send a postcard to China – You want to visit the Getty Museum this weekend. Go to getty.edu and find opening times/prices – Go to nestle.com and find a snack to eat during workouts – Go to bankone.com and find best savings account with a $1000 balance 15
- 16. Prioritizing Web Usability (3) • The result of the research is presented as a book: – Categorising the finding in categories (including searching, navigation, typography and writing style) – Using plenty of examples and screenshots to demonstrate the usability issues that were identified 16
- 17. Prioritizing Web Usability: findings • People succeed 66% of the time when working on “single site” activities and 60% of the time when having to browse through the internet for information 17
- 18. Prioritizing Web Usability: findings (2) • Experienced users spend about 25 seconds in a homepage and 45 in an interior page (35 and 60 for inexperienced users) • Only 23% of users scroll on their first visit of a homepage – The number decreases – The average scroll for first visit is 0.8 of a screen 18
- 19. Prioritizing Web Usability: findings (3) • 88% of users go to search engines to find information • Font face and size: different font faces for print and screen – Different font size depending on target audience • More in the book… 19
- 20. OpenSmsDroid evaluation • You have been tasked to evaluate the usability for a new (fictional) Android application to write short text messages, OpenSMSDroid • You have decided to set up an experiment – The next experiment is (loosely) adapted from “Experimental Evaluation of Techniques for Usability Testing of Mobile Systems in a Laboratory Setting” (Beck, Christiansen, Kjeldskov, Kolbe and Stage, 2003) 20
- 21. OpenSmsDroid evaluation • Your test users will be perform a set of tasks in specific configurations using the thinking aloud method for data collection – A constraint of 5 minutes has been set for each of the tasks – The usability researcher will record the session and take notes 21
- 22. OpenSmsDroid evaluation: testing configurations • Configurations for the test (tentative list): – – – – Sitting on a chair at a table Walking on a treadmill at constant speed Walking on a treadmill at varying speed Walking on an 8-shaped course that is changing as obstructions are being moved, within 2 meters of a person that walks at constant speed – Walking on an 8-shaped course that is changing as obstructions are being moved, within 2 meters of a person that walks at varying speed – Walking in Westfield Stratford at 16:00 on Saturday 22
- 23. OpenSmsDroid evaluation: testing configurations (2) • For practical reasons and after reviewing the literature, these settings have been selected for this evaluation: – Sitting on a chair at a table – Walking on a treadmill at constant speed – Walking in Westfield Stratford at 16:00 on Saturday 23
- 24. OpenSmsDroid evaluation: tasks • Writing a new SMS containing the phrase “The quick brown fox jumps over the lazy dog” repeated 2 times to an existing contact (without using predictive text features) • Writing a new SMS containing the phrase “The quick brown fox jumps over the lazy dog” repeated 2 times to an existing contact (using predictive text features) • Taking a picture and sending it to an existing contact • Taking a short 1 minute video and sending it to an existing contact 24
- 25. OpenSmsDroid evaluation: tasks (2) • In each test, you can collect: – Quantitative data: time needed to perform the task, and if the task has been completed – Qualitative data: asking the user to think aloud while interacting with the device and recording the interaction 25
- 26. OpenSmsDroid evaluation: data analysis • The evaluation will analyse the data collected and report on any findings, informing on any difference in performance and suggesting possible changes to the interface – An experiment can also generate further hypothesis which will be used in further experiments 26
- 27. Usability inquiry • Usability inquiry methods focus (at different degrees) on analysing an artefact either from “the native point of view" or looking for “the native point of view" – Used to obtain information about users' likes, dislikes, needs, and understanding of the system 27
- 28. Usability inquiry (2) • They may use one or more of these tecniques: – Talking to users – Observing users using a system in a real working situation – Letting the users answer questions (verbally or in written form) 28
- 29. Data collection & analysis • Data collection (most methods described in the previous weeks): – Observation & interviews (e.g. contextual inquiry) – Notes, pictures, recordings, diaries – Video – Logging • Analyses – Categorizing the findings – Using existing categories can be provided by 29
- 30. Diary method • The diary method requires users to keep a diary of their interactions • Diaries can be free form or structured – The diary method is best used when the researcher does not have the time, the resources or the possibility to use user monitoring methods or when the level of detail provided by user monitoring methods is not needed 30
- 31. Contextual inquiry • Contextual inquiry is a structured field interviewing method which typically evaluates: – – – – User opinions User experience Motivation Context • It is a study based on dialogue and interaction between interviewee and user, and it is one of the best methods to use when researchers need to understand the users' work context. 31
- 32. Data presentation • The aim is to show how the products are being appropriated and integrated into their surroundings. • Typical presentation forms include: vignettes, excerpts, critical incidents, patterns, and narratives. 32
- 33. Interviews and focus groups • Interviews and focus groups are research methods based on interaction between researchers and users – The researcher facilitates the discussion about the issues rose by the questions – In focus groups (multiple users present), the interaction among the users may raise additional issues, or identify common problems that many persons experience 33
- 34. Surveys • Surveys are a quantitative research method, where a set list of questions are asked and the users' responses recorded – When the questions are administered by a researcher, the survey is called a structured interview – When the questions are administered by the respondent, the survey is referred to as a questionnaire 34
- 35. References • Beck, E., Christiansen, M., Kjeldskov, J., Kolbe, N. and Stage, J. (2003). ‘Experimental Evaluation of Techniques for Usability Testing of Mobile Systems in a Laboratory Setting’, OzCHI 2003. • Nielsen, J. and Loranger, H. (2006). Prioritizing Web Usability. 35