Virtual IMU Data Augmentation by Spring-Joint Model for Motion Exercises Recognition without Using Real Data
0 likes181 views
This document proposes a method to augment virtual IMU data using a spring-joint model for motion exercise recognition without real data. The method aims to address the limitation of current virtual IMU data which has a limited motion length. It introduces a spring-joint virtual sensor module that can simulate different acceleration distributions and augment the virtual acceleration data spatially and temporally. An experiment tested the method on three aerobic exercises and showed the proposed data augmentation approach improved motion recognition accuracy from 45.5% to 85.3% compared to non-augmented virtual data.
![2
Background: real IMU dataset for motion recognition
Participants
Wearing
sensor
Produce the
data
Real wearable IMU sensor data collection: time-costly and expensive.
Opportunity dataset [1]
[1] Daniel Roggen, Alberto Calatroni, Mirco Rossi, Thomas Holleczek, Gerhard Tröster, Paul Lukowicz, Gerald Pirkl, David Bannach, Alois Ferscha, Jakob Doppler, Clemens Holzmann, Marc Kurz, Gerald Holl, Ricardo Chavarriaga, Hesam Sagha,
Hamidreza Bayati, and José del R. Millàn. "Collecting complex activity data sets in highly rich networked sensor environments" In Seventh International Conference on Networked Sensing Systems (INSS’10), Kassel, Germany, 2010.
Dataset
Dataset
Machine
learning
model
Application
Change
Real world
A motion recognition system: machine learning (ML).](https://image.slidesharecdn.com/v3-iswc-virutalaug-230530070722-7190d012/85/Virtual-IMU-Data-Augmentation-by-Spring-Joint-Model-for-Motion-Exercises-Recognition-without-Using-Real-Data-2-320.jpg)
![6
Related Work: virtual sensor data for human motion recognition
Virtual IMU data: IMUTube [1]
Virtual Doppler data: Vid2Doppler [2]
Virtual distance data [3]
[1] Kwon, Hyeokhyen, et al. "IMUTube: Automatic extraction of virtual on-body accelerometry from video for human activity recognition." Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 4.3 (2020): 1-29.
[2] Ahuja, Karan, et al. "Vid2Doppler: synthesizing Doppler radar data from videos for training privacy-preserving activity recognition." Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 2021.
[3] Xia, Chengshuo, Ayane Saito, and Yuta Sugiura. "Using the virtual data-driven measurement to support the prototyping of hand gesture recognition interface with distance sensor." Sensors and Actuators A: Physical 338 (2022): 113463.](https://image.slidesharecdn.com/v3-iswc-virutalaug-230530070722-7190d012/85/Virtual-IMU-Data-Augmentation-by-Spring-Joint-Model-for-Motion-Exercises-Recognition-without-Using-Real-Data-6-320.jpg)
![7
Related Work: data augmentation for time-series
[1] Li, Xi'ang, Jinqi Luo, and Rabih Younes. "ActivityGAN: Generative adversarial networks for data augmentation in sensor-based human activity recognition." Adjunct Proceedings of the 2020 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of
the 2020 ACM International Symposium on Wearable Computers. 2020.
[2] Um, Terry T., et al. "Data augmentation of wearable sensor data for parkinson’s disease monitoring using convolutional neural networks." Proceedings of the 19th ACM international conference on multimodal interaction. 2017.
GAN-based [1]: Time-series manipulation [2]:
Traditional augmentation: operate on real data
Our method: augment the virtual sensor data](https://image.slidesharecdn.com/v3-iswc-virutalaug-230530070722-7190d012/85/Virtual-IMU-Data-Augmentation-by-Spring-Joint-Model-for-Motion-Exercises-Recognition-without-Using-Real-Data-7-320.jpg)
![9
Method: how to obtain the virtual acceleration data
Posistion
Velocity
Acceleration
[1] Young, Alexander D., Martin J. Ling, and Damal K. Arvind. "IMUSim: A simulation environment for inertial sensing algorithm design and evaluation." Proceedings of the 10th ACM/IEEE International Conference on Information
Processing in Sensor Networks. IEEE, 2011.](https://image.slidesharecdn.com/v3-iswc-virutalaug-230530070722-7190d012/85/Virtual-IMU-Data-Augmentation-by-Spring-Joint-Model-for-Motion-Exercises-Recognition-without-Using-Real-Data-9-320.jpg)
![Experiment: tested exercise motions
Test motions: 3 types of aerobic exercises from YouTube online sources, 20 seconds
3D motion conversion: DeepMotion [1]
19
[1] https://www.deepmotion.com/
Reverse Lunge Warm Up High Knee Tap](https://image.slidesharecdn.com/v3-iswc-virutalaug-230530070722-7190d012/85/Virtual-IMU-Data-Augmentation-by-Spring-Joint-Model-for-Motion-Exercises-Recognition-without-Using-Real-Data-19-320.jpg)
![21
Experiment: information
• Real sensor: Xsens Dot [1]
• Participants: 7 people
Conduct the exercises by 90 seconds
• Classifiers: SVM, Random Forest, Decision Tree
• Handcrafted features:
• Sensor Location: Right upper leg
Time & Frequency features
2-dimension features
PCA
[1] https://www.xsens.com/xsens-dot
• Virtual dataset (initial):
• 20s * 3 motions
• Virtual dataset (with augmentation):
• 20s * 8 modules * 3 nodes * 3 speeds * 3 motions
• Real dataset: for testing
• 90s * 3 motions * 7 participants](https://image.slidesharecdn.com/v3-iswc-virutalaug-230530070722-7190d012/85/Virtual-IMU-Data-Augmentation-by-Spring-Joint-Model-for-Motion-Exercises-Recognition-without-Using-Real-Data-20-320.jpg)
Virtual IMU Data Augmentation by Spring-Joint Model for Motion Exercises Recognition without Using Real Data
- 1. Virtual IMU Data Augmentation by Spring-Joint Model for Motion Exercises Recognition without Using Real Data 2022 ACM ISWC (note) Chengshuo Xia, Yuta Sugiura Keio University, Japan
- 2. 2 Background: real IMU dataset for motion recognition Participants Wearing sensor Produce the data Real wearable IMU sensor data collection: time-costly and expensive. Opportunity dataset [1] [1] Daniel Roggen, Alberto Calatroni, Mirco Rossi, Thomas Holleczek, Gerhard Tröster, Paul Lukowicz, Gerald Pirkl, David Bannach, Alois Ferscha, Jakob Doppler, Clemens Holzmann, Marc Kurz, Gerald Holl, Ricardo Chavarriaga, Hesam Sagha, Hamidreza Bayati, and José del R. Millàn. "Collecting complex activity data sets in highly rich networked sensor environments" In Seventh International Conference on Networked Sensing Systems (INSS’10), Kassel, Germany, 2010. Dataset Dataset Machine learning model Application Change Real world A motion recognition system: machine learning (ML).
- 3. 3 Background: virtual IMU sensor data in machine learning Virtual sensor data generation help to reduce the dataset cost. Dataset Machine learning model Application Change Virtual sensor data Simulation envrionment Virtual IMU sensor data extraction from 3D human motion:
- 4. 4 Background: current problems in virtual data • Limited 3D motion length. • Obtaining a longer 3D motion length still is time-consuming. • Real data distribution is needed.
- 5. 5 Proposed Method: spring-joint module-based data augmentation • Spring-joint virtual IMU sensor module. • Used for training the classifier. • Recognize the real IMU data. More virtual sensor data Spring-Joint virtual sensor module 3D motion Training the ML model
- 6. 6 Related Work: virtual sensor data for human motion recognition Virtual IMU data: IMUTube [1] Virtual Doppler data: Vid2Doppler [2] Virtual distance data [3] [1] Kwon, Hyeokhyen, et al. "IMUTube: Automatic extraction of virtual on-body accelerometry from video for human activity recognition." Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 4.3 (2020): 1-29. [2] Ahuja, Karan, et al. "Vid2Doppler: synthesizing Doppler radar data from videos for training privacy-preserving activity recognition." Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 2021. [3] Xia, Chengshuo, Ayane Saito, and Yuta Sugiura. "Using the virtual data-driven measurement to support the prototyping of hand gesture recognition interface with distance sensor." Sensors and Actuators A: Physical 338 (2022): 113463.
- 7. 7 Related Work: data augmentation for time-series [1] Li, Xi'ang, Jinqi Luo, and Rabih Younes. "ActivityGAN: Generative adversarial networks for data augmentation in sensor-based human activity recognition." Adjunct Proceedings of the 2020 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2020 ACM International Symposium on Wearable Computers. 2020. [2] Um, Terry T., et al. "Data augmentation of wearable sensor data for parkinson’s disease monitoring using convolutional neural networks." Proceedings of the 19th ACM international conference on multimodal interaction. 2017. GAN-based [1]: Time-series manipulation [2]: Traditional augmentation: operate on real data Our method: augment the virtual sensor data
- 8. 8 Overview of Proposed Method More virtual sensor data Spring-joint virtual IMU sensor module Machine learning model 3D human motion Recognize the real motion
- 9. 9 Method: how to obtain the virtual acceleration data Posistion Velocity Acceleration [1] Young, Alexander D., Martin J. Ling, and Damal K. Arvind. "IMUSim: A simulation environment for inertial sensing algorithm design and evaluation." Proceedings of the 10th ACM/IEEE International Conference on Information Processing in Sensor Networks. IEEE, 2011.
- 10. 10 Method: spatial and temporal augmentation Spatial Augmentation: Spring-Joint-based Sensor Module Temporal Augmentation: Playback Speed
- 11. 11 Method: spatial and temporal augmentation Spatial Augmentation: Spring-Joint-based Sensor Module Temporal Augmentation: Playback Speed
- 12. 12 Method: spring-joint module design 1 module consists of 3 nodes
- 13. 13 Method: adding multiple modules (The modules follow the movement of Right Upper Leg)
- 14. 14 Method: the nodes connection and parameters Different stiffness of spring-joint: Different structure of spring-joint: Choose four stiffness 0.2, 0.4, 0.6, 0.8 Two types of connection structure
- 15. 15 Method: spatial and temporal augmentation Spatial Augmentation: Spring-Joint-based Sensor Module Temporal Augmentation: Playback Speed
- 16. 16 Method: data augmentation by motion speed shift Speed adjustment: access the time scale to shift the motion speed
- 17. 17 Method: coordinate transformation Virtual acceleration Real acceleration Same sensor coordinate Coordinate transformation Coordinate transformation Fusion & Normalization z y x Uniform coordinate
- 18. 18 Experiment: validation method Virtual dataset (initial): Real dataset Purpose: validate of effeteness of proposed method Virtual dataset (with augmentation): Real dataset Real dataset Classifier Result Real dataset Train Test Classifier Result Train Test Classifier Result Train Test Baseline 1: Baseline 2: Proposed method: Leave-one-subject-out
- 19. Experiment: tested exercise motions Test motions: 3 types of aerobic exercises from YouTube online sources, 20 seconds 3D motion conversion: DeepMotion [1] 19 [1] https://www.deepmotion.com/ Reverse Lunge Warm Up High Knee Tap
- 20. 21 Experiment: information • Real sensor: Xsens Dot [1] • Participants: 7 people Conduct the exercises by 90 seconds • Classifiers: SVM, Random Forest, Decision Tree • Handcrafted features: • Sensor Location: Right upper leg Time & Frequency features 2-dimension features PCA [1] https://www.xsens.com/xsens-dot • Virtual dataset (initial): • 20s * 3 motions • Virtual dataset (with augmentation): • 20s * 8 modules * 3 nodes * 3 speeds * 3 motions • Real dataset: for testing • 90s * 3 motions * 7 participants
- 21. 22 Experiment: Result PCA-1 PCA-2 Virtual ReverseLunge Real ReverseLunge Real HighKnee Virtual HighKnee Real HighKnee Virtual HighKnee Method (Random Forest) Baseline 1 (initial virutal data) Baseline 2 (using real data) Proposed Method (spring- joint module) Accuracy 45.5 % 78.2 % 85.3 %
- 22. 23 Discussion and Limitation Further evaluation of different physical simulation parameters Test on more motions Virtual-to-real data features -> domain-invariant features
- 23. 24 Summary Background Virtual IMU data suffers from limited 3D motion length, which leads to less virtual data samples Related Work Virtual sensor data/Time-series data augmentation Proposed Scheme Using the spring-joint module to simulate different acceleration distribution and augment virtual acceleration data Details Method Spring-joint connection/playback speed adjust/domain adaption Experiment Tested on there aerobic exercises picked from YouTube Result Using proposed augmentation method can get 85% vs. 45% from not augmented Limitation More motions tested/Boundary condition test virtual acceleration 3 virtual acceleration 1 virtual acceleration 2 real acceleration
- 24. Thank you very much!