Brain-based HCI - What brain data can tell us about HCI - St Andrews, 2019
0 likes17 views
The document discusses brain-based human-computer interaction (HCI), focusing on utilizing brain data to understand user experience and mental workload (MWL) during tasks. It highlights the use of functional near-infrared spectroscopy (fNIRS) as a reliable tool for measuring cognitive responses in naturalistic settings, contrasting it with traditional subjective and objective workload assessments. The findings aim to enhance task performance and users' awareness of their cognitive load in various work environments.
![Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Hypotheses Simplified
• H1: High Demand »» High MWL
• H2: Performance w/ Feedback > Performance w/ ISA?
• H3: Perception of [H2]
• H4: MWL Feedback »» Taking Action
40](https://image.slidesharecdn.com/2019-10-24-standrews-240218090951-e1672ae0/85/Brain-based-HCI-What-brain-data-can-tell-us-about-HCI-St-Andrews-2019-39-320.jpg)
![Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
H3: Perception of [H2]
50](https://image.slidesharecdn.com/2019-10-24-standrews-240218090951-e1672ae0/85/Brain-based-HCI-What-brain-data-can-tell-us-about-HCI-St-Andrews-2019-49-320.jpg)
![Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
H3: Perception of [H2]
51](https://image.slidesharecdn.com/2019-10-24-standrews-240218090951-e1672ae0/85/Brain-based-HCI-What-brain-data-can-tell-us-about-HCI-St-Andrews-2019-50-320.jpg)
Brain-based HCI - What brain data can tell us about HCI - St Andrews, 2019
- 1. Brain-based HCI What brain data can tell us about HCI Max L. Wilson @gingdottwit
- 2. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw Talk Topic Breakdown Brain-based HCI 3
- 3. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw Brain-based HCI • As opposed to Brain-Computer Interaction (BCI) • Even to passive BCI to some extent 4 Zander & Kothe 2010
- 4. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw Brain-based HCI • e.g. studying the effect of interruptions using brain-data • Or indeed the design of them • Ultimately, we (my team) are interested in studying HCI using brain data, rather than creating systems that use brain data • Except for some exceptions 5 Pike (PhD Thesis), 2017
- 5. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw To-Say List • Research Background • Workload Alerts Example • Studying work with fNIRS 6
- 7. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw 8
- 8. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw 9
- 9. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw User Interfaces 10 Great Complexity and Confusion With Great Power Comes
- 10. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw 11 Total Mental Capacity Simple UI EasyTask
- 11. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw 12 Total Mental Capacity Complicated UI EasyTask
- 12. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw 13 Total Mental Capacity Complicated UI DifficultTask
- 13. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw 14 Total Mental Capacity Better UI DifficultTask
- 14. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw Mental Workload 15 Sharples, Sarah and Megaw Ted (2015). Definition and mesurement of human workload. In Wilson John R and Sharples Sarah, editors, Evaluation of human work. CRC Press.
- 15. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw Most important criteria for us • We can run a ‘normal’ user study. • As much ecological validity in - the environment they do the study - natural user behaviour in the study - as normal/natural a task as possible • And tell if theres a cognitive difference between UIs or Tasks 16
- 16. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw What we need… Reliable Brain Measurements of HCI 17
- 17. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw What we need… Reliable Brain Measurements of HCI We can trust the data from uncontrolled conditions (not limiting interaction or body position) 18
- 18. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw What we need… Reliable Brain Measurements of HCI Human-Computer Interaction (doing work tasks, using software, etc) 19
- 19. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw Which Brain Scanner? 20 M. Pike (2017) Exploring the use of Brain-Sensing Technologies for Natural Interactions. PhD Thesis. University of Nottingham.
- 20. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw Our fNIRS - fully portable • This is a fully portable fNIRS - bluetooth connection - Most commercially user friendly 21 Artinis: Octomon
- 21. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw fNIRS • Began in 1977 Frans Jobsis • 700-900nm sees through tissue/bone • But is absorbed by hemoglobin - oxygenated and deoxygenated • Measure either side of cross-over - (isobestic point) • Modified Beer-Lambert Law used to calculate relative concentration of each 22 By Adrian Curtin - Own work, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=20510064
- 22. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw Change in oxygenated hemoglobin • Px 23
- 23. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw 24
- 24. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw Comparing responses 25
- 25. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw Showing that fNIRS works for HCI • CHI2014: fNIRS + Think Aloud • CHI2015: fNIRS vs HCI Artefacts • CHI2015: 3 UIs create different Mental Workload • Horia Maior & Matthew Pike • Working with Prof. Sarah Sharples 26
- 26. Workload Alerts ToCHI Article Presented at CHI2018
- 27. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw 28
- 28. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw Horia on Computerphile 29 https://www.youtube.com/watch?v=TQ0sL1ZGnQ4
- 29. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw When do you need to know MWL? 30
- 30. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw How to measure MWL? • Performance based • Subjective techniques • NASA-TLX • ISA - Instantaneous self-assessment of workload technique • Physiological approach • Indirect: Skin response, heart based measurements, etc. • Direct: brain measurements: e.g. fNIRS 31
- 31. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw NASA Task Load Index (TLX) • Using a 21-point subjective questionnaire • Captures 6 subscales: • Mental Demand • Physical Demand • Temporal Demand • Performance • Effort • Frustration • Developed for demanding jobs • Physically • Mentally 32
- 32. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw Instantaneous Self Assessment (ISA) • ‘Continuous’ subjective technique (could be an app while performing a study task) • On Regular intervals during a study (e.g. every 30 sec) • Popular in analysis critical industry tasks • BUT - it interrupts those tasks 33
- 33. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw Workload Alerts: Aim • What if we provide feedback of workload to people during tasks? • Assess workload objectively using fNIRS • Help people manage their workload during tasks • Facilitate a form of Reflection-in-Action during tasks (similar effect as ISA?) 34
- 34. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw Related - Dynamic Adjustment, fNIRS 35
- 35. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw Workload Alerts System 36
- 36. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw Workload Alerts System 37 VS
- 37. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw Task: Airport Madness 4 38
- 38. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw Hypotheses (Complex) 39
- 39. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw Hypotheses Simplified • H1: High Demand »» High MWL • H2: Performance w/ Feedback > Performance w/ ISA? • H3: Perception of [H2] • H4: MWL Feedback »» Taking Action 40
- 40. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw Participants & Conditions • 32 subjects • Counterbalanced • Average age: 25.3 • 4 Conditions * 7 min 41
- 41. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw Dependent Variables • Demand • Number of airplanes on the screen every 30 seconds • Task Performance • Performance outcomes (number of planes landed, take-offs) • Perceived performance • five point rating scale (1 - poor, 5 - excellent) • Subjective ratings • ISA - 30 sec interval • Objective measures • fNIRS 42
- 42. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw Config & Monitoring • Tracking MWL • Config using study task • Thus also practice round for participants • 3 variations of demand • Rest • Low-Normal Load (3-5 aeroplanes) • Normal-High Load (>7 aeroplanes) • State tracking • Using a rolling average of 30 seconds • Monitoring the most sensitive channel 43
- 43. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw Choice of Feedback • Noticeable • Two states of interest: • Low workload • High Workload • Hue Bulbs (Red and White Colours were used for feedback) • Two Pronged approach: • Phase1: Red for High Workload and White for Low workload • Phase2: Reversed Phase1 44
- 44. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw Choice of Feedback 45
- 45. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw H1: High Demand »» High MWL 46
- 46. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw H1: High Demand »» High MWL 47
- 47. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw H2: Perf. w/ Feedback > Perf. w/ ISA? 48
- 48. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw H2: Perf. w/ Feedback > Perf. w/ ISA? 49
- 49. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw H3: Perception of [H2] 50
- 50. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw H3: Perception of [H2] 51
- 51. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw H4: MWL Feedback »» Taking Action 52
- 52. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw Takeaways • ISA increases demand »» impact on performance • Conversely, no evidence of negative impact of MWL Feedback • But - we didn’t manage to see change in behaviour. • RED IS SCARY - not a good feedback mechanism 53
- 53. Studying Work And tasks, and software, etc
- 54. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw Studying Work • What we’ve done: is put specific tasks in a controlled experiment • In practice ‘work’ is messy, and we are far from understanding it with physiological data • BUT - we are not far from characterising types of cognitive change with physiological data 55
- 55. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw Studying Work • RQ1) how are different cognitive changes affecting physiological measurements of mental workload (in work?)? • RQ2) how much does mental workload change in the daily work of information workers? (what does two hours of work look like in fNIRS data) • RQ3) what are the cognitive effects of high mental workload periods on work over time? • RQ4) how do people reflect on this form of cognitive personal data? 56
- 56. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw fNIRS affected by Stress? 57 Norah Alsuraykh Co-supervised by Paul Tennant and Sarah Sharples
- 57. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw How Stress & MRL are Connected • Montreal Imaging Stress Task (MIST) • Maths equally hard - but adds stress • By time pressure and negative feedback • Measures: • Performance, Stress (SSSQ), MWL (NASA TLX) • fNIRS, EDA, Heart Rate, Skin Temp 58 Alsuraykh et al 2019, Pervasive Health, https://doi.org/10.1145/3329189.3329235
- 58. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw How Stress & MRL are Connected 59 Alsuraykh et al 2019, Pervasive Health, https://doi.org/10.1145/3329189.3329235
- 59. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw How Stress & MRL are Connected 60 Alsuraykh et al 2019, Pervasive Health, https://doi.org/10.1145/3329189.3329235
- 60. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw What does work look like? • Creative Arts • Safety Critical (in rail)? • Information Workers • Manufacturing Industry 61
- 61. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw fNIRS during Creative Practice 62 Caroline Locke and Debra Swann
- 62. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw fNIRS during Creative Practice 63 Caroline Locke and Debra Swann
- 63. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw Train Signal Operators 64
- 64. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw • Self Reported Workload (IWS) • Task Demand • EDA • fNIRS • Heart Rate / HRV 65
- 65. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw • Self Reported Workload (IWS) • Task Demand • EDA • fNIRS • Heart Rate / HRV 66
- 66. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw • Self Reported Workload (IWS) • Task Demand • EDA • fNIRS • Heart Rate / HRV 67
- 67. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw Students studying 68 Discussion of MWL Phase 1 Discussion of High and Low MWL Descriptions Phase 2 Interpreting drawings from interviews Phase 3 Producing metaphors based on descriptions of someones daily activities Phase 4 Phase 5 Prototype Discussion Suitable representation? Analyse and reflect on individuals day?
- 68. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw Students studying 69
- 69. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw Brain Data as Personal Data • Has an MSc in Brain Imaging • Industry Sponsor: Brain+ (Copenhagen) • Proposed Plan: study diary-like brain data • Perhaps: in relation to brain training data (Brain+) 70 Serena Midha
- 70. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw Studying Information Workers 71 Easy Medium Hard (Dual) Reading Reading easy text Reading a Journal article Reading an article + Finding a word Writing Writing an Email Describing research area Describing research area + saying blah
- 71. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw Studying Information Workers 72 Reading Writing Reading (fNIRS) NASA TLX
- 72. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw Studying Information Workers • Interruptions interesting! • Depending on if it stopped the task, or happened in parallel! • People accommodated interruptions in low MWL 73
- 73. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw • Next stop - 3-part study Studying Information Workers 74 Gloria Mark style 5 day Diary- based Lifestyle Study Qualitative data from rich Contextual Inquiry Interviews 2hr High density fNIRS data collection
- 74. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw DigiTOP Grant • Digital Toolkit for optimisation of operators and technology in manufacturing partnerships (DigiTOP) - £1.9M / 3 years - PI: Prof Sarah Sharples (Nottingham) • Global shift towards digital manufacturing techniques • Physical Workload » Mental Workload • Traditional Human Factors » Cognitive Ergonomics 75 http://digitop.ac.uk
- 75. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw Facial Thermography vs fNIRS 76 Adrian Marinescu See Computerphile on YouTube And many other sensors, including Posture Analysis
- 76. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw Current Study • Aim: identify overload and underload • Aim: identify periods of mind-wandering • 40 participants in Sustained Attention task • Low demand High Demand • 77
- 77. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw Example fNIRS data 78
- 78. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw Blink rate during mind-wandering 79
- 79. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw fNIRS by fatigue 80 PFC MTG
- 80. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw Fatigue biggest differentiator 81 (for visual attention task)
- 81. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw Studying Work • RQ1) how are different cognitive changes affecting physiological measurements of mental workload (in work?)? • RQ2) how much does mental workload change in the daily work of information workers? (what does two hours of work look like in fNIRS data) • RQ3) what are the cognitive effects of high mental workload periods on work over time? • RQ4) how do people reflect on this form of cognitive personal data? 82
- 82. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw Questions? 83