Evaluating User Experience
- 1. Evaluating User Experience Alistair Sutcliffe & Jenny Hart Manchester Business School University of Manchester Manchester M15 6PB, UK a.g.sutcliffe@manchester.ac.uk NUX July 2011 with thanks to Ons AlShamueli & Rabia Khan
- 2. Presentation Outline 1. Background to UX research 2. A framework for UI ‘attractiveness’ and user engagement 3. Some experiments on users’ perceptions of design quality 4. Defining User Experience 5. Evaluating User Experience
- 3. So What is User Experience (UX) ? “A person’s perceptions and responses that result from the use or anticipated use of a product, system or service”. ISO 2010 “The effect and affect produced by aesthetic experience, the meaning we attach to the product, and the feelings and emotions produced” No real agreement on definitions Essentially beyond the functional- aesthetics, attractiveness – design that excites and holds our attention- user engagement
- 4. UX Research- the quants • Affordances, aesthetics and emotion key factors in design (Norman 2004) • What is beautiful is usable (Tractinsky 2000, 2004); expressive & classic aesthetics, pleasure, • Beauty, goodness, pragmatics & hedonics- Attractdiff (Hassenzahl 2004, 2006, 2010) • Aesthetic perception and interaction (Lingaard 2006, 2009, Hartmann, Sutcliffe & de Angeli 2008) priming effects
- 5. UX Research- The Quals (or contextualists) • Jordon 1998, Pleasure in Products • Mc Carthy & Wright 2005 Technology As Experience, 2010 User Experience • Dourish 2004 Where the Action Is Embodied Interaction • Cockton 2008, Worth maps in UI design • Hallnäs and Redström,2005, Presence and user experience • Designers of User Experience- Gaver, Sengers, Forlizzi and many others
- 6. UX Research- Viewpoints 1. There are fundamental cognitive constructs by which we perceive User Experience- and hence can evaluate it (Hassenzahl, Tractinsky) 2. User Experience can only be understood in context, each experience is unique. It can only be understood by case study analysis and heuristics (Mc Carthy & Wright + contextualists) 3. User Experience is best understood by design- cultural probes, etc (Gaver and other designers) 4. UX is a cognitive process which can be modelled and measured (Sutcliffe, de Angeli)
- 7. The Manchester Attractiveness Framework Usability Content / Services Attractiveness Customisability Aesthetics Reputation / Identity Attractiveness “Pleasing or appealing to the senses, arousing interest” OED
- 8. UX Experiments • User perceptions of Aesthetics and UI Design qualities - Comparing web sites +/- attractive design features (interactive metaphors, dynamic media) - Sutcliffe & de Angeli INTERACT 05, DIS 06 • Framing effects and Customisation on UX - subject background, task scenario effects, customisation - Hartmann, Sutcliffe & de Angeli CHI 07,08 • Avatars, Immersion and UX - attractive chatterbots & persuasion - comparing UX in immersive v. standard environments (Khan, Sutcliffe & de Angeli, Sutcliffe & AlRayes 2010)
- 9. Menu Design style Both sites- equivalent information, different UI styles
- 10. Summary of Differences Dimensions Engaging style Traditional style Usability - + Aesthetic classic - + Aesthetic + - expressive Information - + quality Engagement + - Memory Content-based Interface-based Overall Preference 14 (50%) 14 (50%)
- 11. User Judgement: Effect of Customisation Mobile phone News feed application 2 versions Good/poor aesthetics Customised or General content
- 12. Results aesthetics manipulation content-fit manipulation Measures Aesth++ Standard customised generic Aesthetic classic + - n/a n/a Aesthetic + - n/a n/a expressive Engagement + - n/a n/a Usability = = = (+) = (-) Look and Feel + - + - Customisability = = + - Content = = + - Overall + - + - Preference
- 13. Summary- Components of Attractiveness • Attractiveness is a complex mix of factors, but content and services probably more important • Judgement and choice very dependent on user background and task • Usability is important but defects will be tolerated if overall experience is positive (halo effect) • Aesthetics is important but only in context <user background> • Content and customisation important components in overall preference, but if equal then other criteria come into play
- 14. Experiments on Avatars/ Virtual Characters • Avatars (human like characters) influence preferences - make information more credible - persuade people more effectively than text/speech alone • Attractive avatars are more effective than less attractive - younger, females - similarity to target audience • Leverages “Computer as Social Actor” effect – we treat representations of people on line like real people (Reeves & Nass 1996) Khan, Sutcliffe & De Angeli 2010
- 15. Attractiveness in Agents Small differences in appearance make a huge differences in attractiveness and persuasion
- 16. Avatars in virtual worlds increase engagement (the Second Life experience) more interaction improves UX ratings
- 17. Components of User Experience • UX in the wide - Experience throughout the Product life cycle - Initial contact (aesthetics) - Use (functionality, content) - Customisation - Support • UX in the small (interaction, engagement) -Presence- user as interactor -Immersion in the interactive world -Flow and engagement in the interactive world -Social presence
- 18. Engagement and Interaction Interaction with avatars is more engaging if … - They adapt to the user - Use surprise and are occasionally unpredictable
- 19. User Experience & Engagement increases Arousal Content +ve exp Functionality Emotion induces +ve promotes Interaction Media Presence Pace Flow Human Image Immersion & Voice Complexity Change 3D Worlds Avatars
- 20. Engagement and Attractiveness- revised framework: Usability Content services High level impression Reputation Design Aesthetics Identity Quality Customis- Engagement ability Metaphor Cost Benefit Presence Interaction Flow Interactive Need / Continuum experience Specificity All task / context / use dependent
- 21. UX over time Compared 3 web sites – IKEA avatar virtual shop assistant – NIKE animation and customisation – ALDI baseline Tasks search + interact with features Same tasks and sites 3 visits separated by 1 week Hypotheses - character will improve engagement - interaction (customisation) will improve engagement - effects stronger after more experience
- 24. ALDI Ad pop ups
- 25. Preferences & ratings • No significant difference Nike- IKEA • Preference and rating order the same 1,2 weeks later • But rating of criteria changed aesthetics, usability earlier, content, brand later • Avatar (IKEA) not engaging but animation was • Customisation (Nike) not engaging but animation was • ALDI pop ups disliked
- 26. Evaluating User Experience (Sutcliffe & Hart 2011) • Observation (within session) - user activity - body posture, gestures, facial expressions - attention • Questionnaire (post session) - existing scales- expressive aesthetics + new scales Presence (from VR), Flow-engagement, Media, and immersion - memory recall- salient features, episodes • Physiological Measures (in session) - heart rate - GSR - pupilometry & eye tracking
- 27. Observation • Activity - system logs or sampling: pace, critical incidents & breakdowns, task v error repair actions • Attention - gaze on screen/in world v elsewhere - could supplement with eyetracking but expensive • Non Verbal Communication - post session analysis from video records - rate posture for arousal/excitement - facial expression for emotion
- 28. Questionnaires I General Engagement • Rate your general mood after using the application (positive happy … negative depressed) • Rate the strength of your feelings/ emotions: Pleasure, Joy, Surprise, Sadness, Anxiety, Worry, Fear, Frustration, Disgust. Interaction/Flow • Rate the pace of interaction (too slow, about right, too fast) • How challenging was operating the interface? (too easy, about right, too difficult) Media CASA • Did you notice any images of people? (not at all … very much) • How attractive were the images used in the application?
- 29. Questionnaires II Presence • How natural did your interactions with the application feel? • How aware were you of events occurring in the real world around you? Immersion • How compelling was your sense of moving inside the interactive world? • How natural did the interactive world appear to be? Social Presence • How aware were you of the person you were communicating with? • How well did the application communicate the identity of other people?
- 30. Summary & Conclusions • User Experience is multi-faceted and will change over time • It can be measured/systematically evaluated • UX is context (domain) dependent • Our judgement of UX suffers from biases and framing effects • UX components - in session engagement: avatars, virtual worlds and user as actor- motivate and attract - across session: personalisation, utility, challenge and adaptation
- 31. Thanks for your attention For more information • Sutcliffe, A.G. (2003). Multimedia and virtual reality: Designing multisensory user interfaces. Mahwah NJ: Lawrence Erlbaum Associates. • Sutcliffe, A. G. (2009). Designing for user engagement: aesthetic and attractive user interfaces. In Carroll, J.M. (Ed), Synthesis lectures on human centered informatics. San Rafael CA: Morgan Claypool. • Sutcliffe A.G. & de Angeli A., (2005), Assessing interaction styles in web user interfaces. In Proceedings of Human Computer Interaction INTERACT 05, Eds Costabile M.F and Paterno F., Rome Sept, 2005, Springer Verlag. pp 405-417. • De Angeli, Sutcliffe A.G. & Hartmann J. (2006) Interaction, usability and aesthetics: What influences users’ preferences? In Proceedings of DIS 2006, Designing Interactive Systems, ACM Press. • Hartmann J., Sutcliffe A.G. & de Angeli A. (2007), Investigating attractiveness in web user interfaces. in CHI07, Human Factors in Computing Systems, San Jose, CA, ACM Press. • Hartmann, J., Sutcliffe, A. G., & De Angeli, A. (2008). Framing the user experience: Information biases on website quality judgement. In Proceedings of CHI-08. New York: ACM Press. • Hartmann, J., Sutcliffe, A. G., & De Angeli, A. (2008). Towards a theory of user judgment of aesthetics and user interface quality. ACM Transactions on Computer-Human Interaction, 15(4), 15-30. ags@manchester.ac.uk