Real-time Multimodal Feedback with the CPR Tutor
Download as PPTX, PDF0 likes5,040 views
The document discusses the implementation of a real-time multimodal tutor designed for cardiopulmonary resuscitation (CPR) training using various data sources, including a Microsoft Kinect and a CPR manikin. It outlines the architecture and methodology for providing immediate feedback based on performance indicators, as well as the challenges faced in creating effective multimodal feedback loops. The findings indicate that the CPR tutor can improve performance in the short term, with suggestions for future research to expand its efficacy and data collection.
![Di Mitri et al. - Real Time Multimodal Feedback 3
Multimodal Tutors
This project implements the idea of the
Multimodal Tutor: an Intelligent Tutoring System
for psychomotor learning tasks.
The Multimodal Tutor leverages multimodal data
to enrich the learner’s digital representation.
In this study we present the CPR Tutor a
Multimodal Tutor for Cardiopulmonary
Resuscitation training.
Di Mitri, D. (2020) The Multimodal Tutor: Adaptive Feedback from Multimodal
Experiences. PhD Dissertation. Open Universiteit, The Netherlands.
[Multimodal data extend ITSs]](https://image.slidesharecdn.com/39-aied-2020-16-9-200708084550/85/Real-time-Multimodal-Feedback-with-the-CPR-Tutor-3-320.jpg)
![Di Mitri et al. - Real Time Multimodal Feedback 14
Data representation
INPUT SPACE:
3D tensor of shape:
[4803 CCs x 17 time-bins x 52 attributes]
HYPOTHESIS SPACE:
Five binary target classes:
(1) classRate, (2) classDepth, (3) classRelease,
(4) armsLocked, (5) bodyWeight
Raw data format:](https://image.slidesharecdn.com/39-aied-2020-16-9-200708084550/85/Real-time-Multimodal-Feedback-with-the-CPR-Tutor-14-320.jpg)
![Di Mitri et al. - Real Time Multimodal Feedback 15
Real-time Exploitation steps
1. Activity Detection:
rule-based approach to detect the CC
using ShoulderLeftY fluctuations
2. Mistake classification:
one LSTM network trained to classify 5
different classes
3. Feedback Logic:
Feedback prioritisation based on
the presence and the priority of
mistakes
1. Detection
2.
Classification
3. Feedback
[the three steps of real-time exploitation]](https://image.slidesharecdn.com/39-aied-2020-16-9-200708084550/85/Real-time-Multimodal-Feedback-with-the-CPR-Tutor-15-320.jpg)
![Di Mitri et al. - Real Time Multimodal Feedback 16
Neural Network configuration
The Neural Network was configured with two
stacked LSTM layers followed by two dense layers.
• a first LSTM with input shape 17x52 with 128
hidden units;
• a second LSTM with 64 hidden units;
• a fully-connected layer with 32 units with a
sigmoid activation function;
• a fully connected layer with 5 hidden units
(number of target classes)
• a sigmoid activation.
[generic LSTM network]](https://image.slidesharecdn.com/39-aied-2020-16-9-200708084550/85/Real-time-Multimodal-Feedback-with-the-CPR-Tutor-16-320.jpg)
![Expert group (n=10)
Titel van de presentatie 18
Each expert executed 4 CPR
sessions of 1 minute each
(only CC – no rescue breath):
1. regular session
2. arms not locked mistake
3. regular session
4. body weight mistake
In two sessions the experts where
asked to purposely make mistakes.
[instructions given to experts]](https://image.slidesharecdn.com/39-aied-2020-16-9-200708084550/85/Real-time-Multimodal-Feedback-with-the-CPR-Tutor-18-320.jpg)
![Di Mitri et al. - Real Time Multimodal Feedback 19
Feedback group (n=10)
10 users tested the
system - performing
each 2 sessions in
alternated order
- 1 minute without
feedback
- 1 minute with
feedback
[feedback explanation given to users before starting]
The users were not novices!](https://image.slidesharecdn.com/39-aied-2020-16-9-200708084550/85/Real-time-Multimodal-Feedback-with-the-CPR-Tutor-19-320.jpg)
![Di Mitri et al. - Real Time Multimodal Feedback 21
Feedback Group results (1)
Error rate:
i: i-th CC in the series
j: Target Class
n: no. of CCs in 10s
ratio of predicted Zeros and Ones
Performance:
[Plot of one session with feedback]](https://image.slidesharecdn.com/39-aied-2020-16-9-200708084550/85/Real-time-Multimodal-Feedback-with-the-CPR-Tutor-21-320.jpg)
Real-time Multimodal Feedback with the CPR Tutor
- 1. Real-time Multimodal Feedback with the CPR Tutor Full paper at the International Conference in Artificial Intelligence in Education (AIED’2020) 8th July 2020 daniele.dimitri@ou.nl - @dimstudio Daniele Di Mitri, Jan Schneider, Kevin Trebing Sasa Sopka, Marcus Specht, and Hendrik Drachsler
- 2. Di Mitri et al. - Real Time Multimodal Feedback 2 Imagine this learning situation
- 3. Di Mitri et al. - Real Time Multimodal Feedback 3 Multimodal Tutors This project implements the idea of the Multimodal Tutor: an Intelligent Tutoring System for psychomotor learning tasks. The Multimodal Tutor leverages multimodal data to enrich the learner’s digital representation. In this study we present the CPR Tutor a Multimodal Tutor for Cardiopulmonary Resuscitation training. Di Mitri, D. (2020) The Multimodal Tutor: Adaptive Feedback from Multimodal Experiences. PhD Dissertation. Open Universiteit, The Netherlands. [Multimodal data extend ITSs]
- 4. Di Mitri et al. - Real Time Multimodal Feedback 4 Multimodal Feedbak Loops - Multimodal data in learning is getting more and more attention - Still most of the related studies stand at the level of “data geology” - E.g. they investigate correlations or predictability - they do not provide direct and immediate feedback to the learner - Problem: Multimodal feedback loops are technically complex. We identify five main challenges in the collection, storing, processing, annotation and exploitation. Multimodal Feedback Loops Keep me in the loop!
- 5. Di Mitri et al. - Real Time Multimodal Feedback 5 Why CPR training? • CPR can be taught singularly to one learner • CPR is a highly standardized procedure • CPR has clear and well-defined criteria to measure the quality • CPR is a highly relevant skill
- 6. Previous work*: Detecting CPR mistakes 6 Hardware setup: • Microsoft Kinect v2 • Myo Armband • Laerdal ResusciAnne QCPR manikin + SimPad Dateset collected: • ~5500 Chest Compressions from 14 experts • Each CC tagged with 5 classes Trained 5 Neural Networks to classify CPR mistakes *Di Mitri, D., Schneider, J., Specht, M., & Drachsler, H. (2019). Detecting mistakes in CPR training with multimodal data and neural networks. Sensors (Switzerland), 19(14) Di Mitri et al. - Real Time Multimodal Feedback
- 7. Di Mitri et al. - Real Time Multimodal Feedback 7 CPR Performance indicators Indicator Ideal value Compression rate 100 to 120 compr./min Compression depth 5 to 6 cm Compression release 0 - 1 cm Arms position Elbows locked Body position Using body weight assessed by the ResusciAnne maniking not measured by the ResusciAnne manikin
- 8. Di Mitri et al. - Real Time Multimodal Feedback 8 Feedback with the CPR Tutor Lock your arms! Use your body weight! Release the compression! *Metronome sound 110bpm* Check compression depth!
- 9. System Architecture Two main software components: - CPRTutor app (C#) for data collection & feedback - SharpFlow script (Python) for data analysis and machine learning Two 3rd party components: - Visual Inspection Tool for the data annotation - Multimodal Learning Hub data synchronisation logic and data storing format Two flows of data: 1) Offline data collection 2) Real-time exploitation All the components are available Open Source on GitHub!
- 10. Di Mitri et al. - Real Time Multimodal Feedback 10 Flow 1) Offline TrainingSensors MLT session Kinect.json Myo.json Video.mp4 CPRTutor C# app SharpFlow Python3 Model training ML Models ClassRate ClassRelease ClassDepth ArmsLocked BodyWeight
- 11. Di Mitri et al. - Real Time Multimodal Feedback 11 Flow 2) Real-time ExploitationSensors CPRTutor C# app SharpFlow Python3 TCP client TCP server Chunk (1 CC – 0.5 sec) Classification ML Models ClassRate ClassRelease ClassDepth ArmsLocked BodyWeight Feedback
- 12. Di Mitri et al. - Real Time Multimodal Feedback 12 Data storing example of serialisation of Myo data in JSON example annotation.json MLT data format MLT = Meaningful Learning Task
- 13. Di Mitri et al. - Real Time Multimodal Feedback 13 Data annotation With the Visual Inspection Tool 1. Load each recorded session 2. Load the SimPad external annotation file providing CC segmentation and values for classRate, classDepth, classRelese 3. Annotate manually armsLocked, bodyweight by looking at the video sequences Di Mitri D., Schneider J., Specht M., Drachsler H. (2019) Read Between the Lines: An Annotation Tool for Multimodal Data for Learning. LAK19 Proceedings
- 14. Di Mitri et al. - Real Time Multimodal Feedback 14 Data representation INPUT SPACE: 3D tensor of shape: [4803 CCs x 17 time-bins x 52 attributes] HYPOTHESIS SPACE: Five binary target classes: (1) classRate, (2) classDepth, (3) classRelease, (4) armsLocked, (5) bodyWeight Raw data format:
- 15. Di Mitri et al. - Real Time Multimodal Feedback 15 Real-time Exploitation steps 1. Activity Detection: rule-based approach to detect the CC using ShoulderLeftY fluctuations 2. Mistake classification: one LSTM network trained to classify 5 different classes 3. Feedback Logic: Feedback prioritisation based on the presence and the priority of mistakes 1. Detection 2. Classification 3. Feedback [the three steps of real-time exploitation]
- 16. Di Mitri et al. - Real Time Multimodal Feedback 16 Neural Network configuration The Neural Network was configured with two stacked LSTM layers followed by two dense layers. • a first LSTM with input shape 17x52 with 128 hidden units; • a second LSTM with 64 hidden units; • a fully-connected layer with 32 units with a sigmoid activation function; • a fully connected layer with 5 hidden units (number of target classes) • a sigmoid activation. [generic LSTM network]
- 17. Study procedure 1. Expert group data collection (N=10) •collecting the data corpus 2. Data annotation and model training •training the models 3. Feedback group (N=10) •testing the feedback functionality of the CPR Tutor Titel van de presentatie 17 The experiment was run at Medical Simulation centre AIXTRA Uniklinik Aachen, Germany
- 18. Expert group (n=10) Titel van de presentatie 18 Each expert executed 4 CPR sessions of 1 minute each (only CC – no rescue breath): 1. regular session 2. arms not locked mistake 3. regular session 4. body weight mistake In two sessions the experts where asked to purposely make mistakes. [instructions given to experts]
- 19. Di Mitri et al. - Real Time Multimodal Feedback 19 Feedback group (n=10) 10 users tested the system - performing each 2 sessions in alternated order - 1 minute without feedback - 1 minute with feedback [feedback explanation given to users before starting] The users were not novices!
- 20. Di Mitri et al. - Real Time Multimodal Feedback 20 Expert Group results Note: Due to imbalanced class distribution the dataset was reduced to 3434 samples (-28.5%).
- 21. Di Mitri et al. - Real Time Multimodal Feedback 21 Feedback Group results (1) Error rate: i: i-th CC in the series j: Target Class n: no. of CCs in 10s ratio of predicted Zeros and Ones Performance: [Plot of one session with feedback]
- 22. Feedback Group results (2) Titel van de presentatie 22
- 23. Di Mitri et al. - Real Time Multimodal Feedback 23 Short-term positive effect of feedback feedback Average Error Rates’ first derivative 10 seconds before and 10 seconds after feedback.
- 24. Di Mitri et al. - Real Time Multimodal Feedback 24 Conclusions This study proves the concept of a Real-time Multimodal Feedback Loop. This study provided: 1) an architecture design 2) a methodological approach to design multimodal feedback 3) a field study on real-time feedback for CPR training. The experimental results suggest that the CPR Tutor short-term improvement of the CPR performance
- 25. Di Mitri et al. - Real Time Multimodal Feedback 25 Future developments • For measuring long-term improvements we need to collect more data from more participants • Testing complete novices can produce more interesting results for the CPR Tutor • The rule-based CC detection can be replaced with automatic detection using a streaming approach • Neural networks are optimal for detecting regularities e.g. CC-rate, better approaches are possible • Kinect camera can be replaced by a webcams body pose estimation libraries (OpenPose, PoseNet…)