IVE 2024 Short Course - Lecture14 - Evaluation
0 likes91 views
The document outlines the importance of prototyping and evaluation in design, emphasizing the need to gather user feedback early to refine ideas and solutions. It details various aspects of evaluation, including what to measure, user involvement, and methods for capturing feedback. The document also provides guidelines for planning and executing evaluations effectively to inform future iterations of prototypes.
IVE 2024 Short Course - Lecture14 - Evaluation
- 1. Gun Lee IVE Winter School 2024 Evaluation
- 2. Why are you building a prototype?
- 3. Why are you building a prototype? To evaluate your idea, early!
- 4. Why did Tom Chi prototype? https://www.youtube.com/watch?v=d5_h1VuwD6g
- 5. Evaluation § What? § Why? § Who? § When? § Where? § How?
- 6. What is Evaluation? § Evaluation is gathering data about the quality of a product/design/idea § User performance § Usability § User eXperience § …
- 7. Outcome of Evaluation § What worked ... § Things users liked or found notable § Useful, efficient, easy, fast ... § What could be improved ... § Failed, difficult, cumbersome, slow ... § Constructive criticism § Questions and Ideas ... § To continue investigation in the future
- 8. Outcome of Evaluation = Feedback Identify needs, problems & requirements (Re)Design Evaluate / Test Implement / Prototype Final A version of solution
- 9. Why? § To validate our prototype/solution/design/idea. § Against the goal, requirement, and problem statement § To refine our prototype and design. § Informs the next iterations of prototypes. § Brings clarity when making decisions. § To learn more about user and the problem. § Often yields unexpected insights. § Test not only the solution, but also if the goal (the problem you are trying to solve) is framed correctly. § To move forward to the next iteration
- 11. Why?
- 13. When? § “You can fix it now on the drafting board with an eraser or you can fix it later on the construction site with a sledge hammer.” - unknown
- 14. When? design implementation evaluation redesign & reimplementation design implementation
- 15. Who? § Designer (yourself!) § Colleagues § Experts § Stake holders § (End) Users § The people for whom you are designing for! § Users are not like you (developer/designer)! § The user is ALWAYS right! - Though they might misunderstand themselves … Informal, Formative testing Formal, Summative & Validation testing
- 16. How many? § 5 users will find approximately 80% of problems - Nielsen
- 17. Where? § Laboratory § In-the-Wild § Remote § Mechanical Turk § Prolific VR Questionnaire Toolkit https://www.youtube.com/watch?v=ctzbobrgcdE https://github.com/MartinFk/VRQuestionnaireToolkit
- 18. How? § Let the user try it first, instead of showing how to. § Not a salesmen but an anthropologist § Not to pursue users about our solution, but to learn more from the user and understand. § Do not defend your prototype; instead, watch how the users use and misuse it. § Be open to unexpected outcomes and misunderstandings. § Your prototype is not precious, but the feedback and new insights it draws out are.
- 19. How? § But my prototype is not fully working… 😞 § No worries, you can be a “Wizard of Oz”! 😉
- 20. How?
- 22. How? § Screen cast is not working… 😞 § Encourage the user to “Think Aloud” 😉 § Or record the screen then review it with the user (Retrospective Think Aloud)
- 23. How? § Feedback capture grid § Be systematic about feedback § Be more intentional about capturing thoughts in the four different areas. Things one likes or finds notable. Constructive criticism, wishes. Questions that the experience raised. Why? Ideas that the experience or presentation spurred.
- 24. How? § Got multiple design options?
- 25. How? § What to measure? § Presence § Co-Presence § Social Presence § User Experience § Usability § Cognitive Load § Emotion § Empathy § ...
- 26. § How to measure? § Presence § Co-Presence § Social Presence § User Experience § Usability § Cognitive Load § Emotion § Empathy § ... How? ? “Ask, and it shall be given to you.” - Jesus
- 27. § Questionnaires § Presence § Co-Presence § Social Presence § User Experience § Usability § Cognitive Load § Emotion § Empathy § ... How?
- 28. How? § Presence Questionnaires § Slater, Usoh, Steed (SUS) (1994) § Witmer & Singer (1998) § Igroup Presence Questionnaire (IPQ) (Schubert et al. 2001) § Co-/Social Presence Questionnaires § Bailenson et al. (2001) § Networked Minds (Harms & Biocca 2004) https://www8.informatik.umu.se/~jwworth/PresenceMeasurement.pdf
- 29. How? § User Experience Questionnaire § UEQ https://www.ueq-online.org § Usability § System Usability Scale (SUS) (Brooke 1996) https://measuringu.com/sus/ § Single Ease Questionnaire (Sauro & Dumas 2009) https://measuringu.com/seq10/ https://measuringu.com/seq-prediction/
- 30. How? § Cognitive Load § NASA TLX (1986) https://humansystems.arc.nasa.gov/groups/TLX/ § Paas (1992) https://www.frontiersin.org/journals/education/articles/10.3389/feduc.2021.702616/full
- 31. How? § Emotion § Self Assessment Manikin https://www.sciencedirect.com/science/article/pii/ 0005791694900639 § Emoji Grid https://www.frontiersin.org/journals/psychology/ar ticles/10.3389/fpsyg.2018.02396/full
- 32. How? § Emotion § Emoji Grid in VR https://link.springer.com/chapter/10.1007/978-3-030- 31908-3_24 § Real-time Continuous Emotion Annotation https://dl.acm.org/doi/abs/10.1145/3411764.3445487 https://www.youtube.com/watch?v=xZyyNTLjvlc
- 33. How? § Empathy? § Inclusion of Other in the Self scale (IOS) https://sparqtools.org/mobility-measure/inclusion-of- other-in-the-self-ios-scale/ § Interpersonal Reactivity Index https://www.eckerd.edu/psychology/iri/ § Empathy Quotient https://psychology-tools.com/test/empathy-quotient
- 34. How? DECIDE framework to guide evaluation: - Determine the goals the evaluation addresses. - Explore the specific questions to be answered. - Choose the evaluation methods and measures - Identify the practical issues. • User recruitment, equipment, space, schedule, budget - Decide how to deal with the ethical issues. - Evaluate, interpret and present the data. Preece, J., Rogers, Y., Sharp, H. (2002), Interaction Design: Beyond Human-Computer Interaction
- 35. How? Test your Test, too!
- 36. Iteration! Identify needs, problems & requirements (Re)Design Evaluate / Test Implement / Prototype Final A version of solution
- 37. Summary § Evaluate your idea as early as possible § First, set your goal for evaluation! § Next, plan ... § How to present your prototype § What questions to answer § How to capture the feedback § Collect, summarize, and reflect on the findings § Apply what you’ve learnt to the next iteration