Micro Task Interface Evaluation Method
- 1. micro tasks s w i p e t o s t a r t
- 2. our goal: a standardized method for benchmarking control room interfaces • Tablet tool for data collection • Can be used stand-alone or • Linked to simulator • Standardized data collection procedure • Standardized method for question generation • Standardized set of questions • Database
- 3. micro tasks are for Evaluation HRA Training
- 4. Efficiently and objectively benchmark innovative displays against conventional interfaces
- 5. IFE process overview display on tablet Example IFE will conduct a micro task evaluation of this set-up in December 2015 at a U.S. training simulator
- 6. Compare
- 7. IFE design concept for overview displays on tablet, developed for the 2015 U.S. simulator study (screen 1 of 3)
- 9. mass balance conventional innovative Do the innovative displays lead to faster, more reliable identifications and decisions?
- 10. mass balance conventional innovative Is performance (time and reliability) with innovative displays at least as good as with conventional?
- 11. How to test performance benefits of new interface solutions?
- 12. Scenario-based methods. Observational or self-report. Qualitative insights. Relatively few data points. Decontextualised. Performance based. Quantitative data. Large amounts of data. flavors of t&e methods
- 13. micro tasks are…
- 14. Large number of questions Related to systems, components, procedures, etc. Varying levels of difficulty Including higher-level decision making Different display conditions E.g. innovative vs. conventional displays We measure response time / accuracy Compare data between conditions If needed, review eye tracking recordings to understand anomalies detection/decision tasks under time pressure
- 15. Micro task tablet app linked to simulator Tablet app receives signals from simulator Tablet app can send signals / commands to simulator Dynamic scenarios To cover monitoring / vigilance tasks We can now record operator actions Can record operating of components, e.g. “start RCPs” New system for aggregating data New system for managing task lists Makes it easier to set up and manage a study new in 2015
- 16. example
- 18. 4 operators working in the simulator individually (no communication)
- 19. We control which displays are available (eg innovative, conventional)
- 20. Instructions: ”Please answer the questions correctly, but also as quickly as possible. It is very important that you work as fast as you can.”
- 21. Run 1 s w i p e t o s t a r t
- 22. How many condensate pumps are running on turbine 31? 1 2 3 s w i p e t o c o n t i n u e
- 23. “How many condensate pumps are running on turbine 31?” Average identification time Conventional 8 sec Innovative 6 sec
- 24. Is the subcooling margin sufficient? Yes s w i p e t o c o n t i n u e No
- 25. Which steam generators are faulty? SG-1 s w i p e t o c o n t i n u e SG-2 SG-3
- 26. Should safety injection be stopped? Yes s w i p e t o c o n t i n u e No
- 27. What is the narrow-range level in steam generator 1? s w i p e t o c o n t i n u e % 1 2 3 4 5 6 7 8 9 0 37
- 28. up to 200 questions per hour with 4 operators in the simulator working individually, that means we can run up to 800 questions per hour
- 29. Tablet can trigger events in the simulator (e.g. start a tube leak) and receive signal from the simulator (e.g. RCP-2 was started)
- 31. results
- 32. data source where does this data come from?
- 33. 6840 data points 3420 response time measures 3420 accuracy measures
- 34. 20 operators
- 35. 5 hours simulator time in total for the whole data collection
- 36. innovative displays are superior
- 37. HSI mean performance times Current effect: F(2, 8,003 )=21,889, p=,00057 Vertical bars denote 0,95 confidence intervals LSD (Innovative) OWD (Conventional) LSD and OWD 15 16 17 18 19 20 21 22 23 24 25 26 Performancetime(sec) Operators were faster with innovative displays than with conventional displays innovative conventional HSI mean performance times Current effect: F(2, 8,003 )=21,889, p=,00057 Vertical bars denote 0,95 confidence intervals 21 22 23 24 25 26 etime(sec) Highly statistically significant difference
- 38. Slightly more accurate with innovative displays than conventional displays Percentage of correct answers innovative conventional innovative
- 40. conclusion
- 41. As reliable as conventional displays Faster to read innovative displays in the 2014 study are
- 42. summary
- 43. Generate quantitative results about performance benefits (time and reliability) of digital/innovative CR interfaces Highly efficient, objective method Compare to existing reference data Qualitative insights via eye tracking Data directly supports HRA Highly customisable and precise Generate exactly the data needed for the HRA
- 44. big picture micro task tablet app = a mobile companion for Human Factors / Human Reliability specialists
- 45. Micro task tablet app Standardized question sets Database training hra interface design gamification evaluation of training validation benchmarking within / across organisations
- 46. what next?
- 47. Benchmarking studies 2015 / 2016
- 48. benchmarking studies (1) 2015 study at U.S. training simulator analog interfaces vs IFE displays (2) 2016 study of a partially digital control room analog interfaces vs. 1990s-vintage overview display vs IFE display (3) 2016 study of a fully digital control room Measure operator performance and reliability in a fully digital control room and benchmark against data from study 1 and 2
- 49. Micro task database Micro tasks for training Includes gamification approach Micro tasks for actions outside the control room (field operator, mechanical, etc) roadmap
- 50. Micro tasks for team decisions Add secondary task capability Add workload measurement capability roadmap
Editor's Notes
- #37: Obviously this is not a general statement, but specific to the findings in this study
- #38: There were 3 conditions: - Conventional displays only. Only the 30” operators screens were turned on (data in the middle) Innovative symbols only. Only the large screen was turned on (data on the left) Both. All screens were turned on, operators could choose which screens they use (data on the right)