Sensor Based Ambient Assisted Living
Download as PPTX, PDF3 likes391 views
The document discusses innovative solutions for dementia care, emphasizing the integration of IoT and wearable sensors for continuous monitoring and personalized feedback to enhance patient safety and independence. It outlines various technological advancements, existing projects, and clinical interventions aimed at improving dementia care efficiency and effectiveness. Future directions include further integration of AI and big data to support cognitive and physical interventions for patients.
Sensor Based Ambient Assisted Living
- 1. Sensor-based AAL ΕΛΕΒΗΤ 2019 CERTH-ITI MKLab Group Dr Ioannis Kompatsiaris* Researcher Grade A, MKLab Director Dr Thanos Stavropoulos Postdoc Associate Dr Spiros Nikolopoulos Senior Researcher Mrs Ioulietta Lazarou Clinical Researcher
- 2. The problem: Dementia Care • Inaccessible – Unaffordable - Inefficient Care of Dementia • No specific treatment – lifestyle/non-pharmaceutical • High 1 Nurse per 1 Patient Effort • High cost • Lack of objective information Yearly cost of dementia care in 2030 People living with dementia in 2050 People living with dementia now 46M 135.5M 2T $ 1 in 2 over 80 (US) Growing Numbers for Dementia
- 3. Overall approach Enhance current clinical workflow: Continuous, comprehensive monitoring of PwD, condition and progression Objective multi-sensor measurements (limiting interpretation subjectivity) Connect monitoring results with dementia staging and assist diagnosis Provide PwD with regular personalized feedback, updates and interventions Improving condition Enhancing a sense of safety and increased independence Relieve informal carers
- 4. The Solution • Existing IoT and wearable sensors provide diverse measurements (steps, HR, presence, object usage) • Intelligent analysis turns them into meaningful and useful behaviors and symptoms (cooking, chores, TV, sleep, stress) … in order for clinicians to provide care more effectively and efficiently
- 5. Technological Approach Sensors, Analysis & Integration
- 6. • 2013 - Use of prototypes • Philips • 2015 - Emergence in the market • FitBit • Jawbone • MS Band • Withings • 2019 - Growing capabilities • Empatica Embrace • Epilepsy FDA-app. • Omron HeartGuide • 24/7 BPM • Withings Move / Move ECG • 1 year battery life + ECG Evolution of IoT Devices
- 7. Device/Hardware Layer • A moving target; Need for future-proof modular support • Service-oriented architecture • Components need to support data retrieval • Streaming Real-time transfer e.g. over Smartphone • Use of Manufacturer SDKs to build smartphone apps for storage or upload • MS Band, Empatica • Data-logging storage in the device or on 3rd party cloud • Use of 3rd party (provider) cloud API for retrieval • Fitbit, Withings, Jawbone Our App via SDK Our Cloud Device Device Cloud Device App Device Our Cloud
- 8. Device variety and modalities in the platform Sleep Sensor IR Presence Object Movement Door Sensor Wearables • Beddit • Withings Aura • Philips DTI2 • Jawbone UP24, UP3 • Fitbit Zip, Charge HR • Empatica E4 • MS Band • Wireless Sensor Tags Appliance Usage • Plugwise
- 9. Wearable Cameras • Go Pro Hero For more sophisticated functions such as: • Activity Recognition • Object Recognition • Room Recognition
- 10. Depth Cameras • Microsoft Kinect • ASUS Depth Camera • Activity Recognition • Indoor localization
- 11. Architecture for Integration • Integration of • Device Layer to retrieve data • Sensor & Image processing layer to analyze them • Store unanimously in a Knowledge Base using semantic web technologies • Ontology with context & clinical information • Semantic Interpretation • Provides activities, behavior using ontology + reasoning • Provides symptoms and problems using rules • User interfaces to present Stavropoulos, T. G., Meditskos, G., & Kompatsiaris, I. (2017). DemaWare2: Integrating sensors, multimedia and semantic analysis for the ambient care of dementia. Pervasive and Mobile Computing, 34, 126-145.
- 12. Visual Analytics for Activity Recognition 12 • Detailed activity and location detection and recognition at home and lab environments. Crispim-Junior, C. F., Buso, V., Avgerinakis, K., Meditskos, G., Briassouli, A., Benois-Pineau, J., ... & Bremond, F. (2016). Semantic event fusion of different visual modality concepts for activity recognition. IEEE transactions on pattern analysis and machine intelligence (IEEE TPAMI), 38(8), 1598- 1611.
- 13. Signal Analysis Raw skin conductance signal Baseline histogram for a week Filtering and segmentation in 5 Stress levels Stress level signal Developed with Philips NL, evaluated in Lulea Technical University Kikhia, B., Stavropoulos, T., Andreadis, S., Karvonen, N., Kompatsiaris, I., Sävenstedt, S., ... & Melander, C. (2016). Utilizing a wristband sensor to measure the stress level for people with dementia. Sensors, 16(12), 1989.
- 14. Semantic Knowledge Structures (OWL 2) • Formal vocabularies for capturing context in different levels of granularity • Low-level observations (e.g. objects, locations) • Complex activity models (e.g. tea preparation) • Clinical knowledge (e.g. problems, monitoring parameters) 14
- 15. Context-based Multi-sensor Fusion and Analysis 15 • Objective: Fusion of information coming from heterogeneous sources in order to derive high-level interpretations of the behaviour of the person • Our approach: Knowledge-driven semantic segmentation and classification of context • Combination of SPARQL and OWL 2 meta-modelling ADL REC PRE Prepare Drug Box 92.00% 88.46% Make Phone Call 89.29% 96.15% Watch TV 84.00% 95.45% Water the plant 80.00% 95.24% Read Article 95.83% 85.19% Meditskos, G., Dasiopoulou, S., & Kompatsiaris, I. (2016). MetaQ: A knowledge-driven framework for context-aware activity recognition combining SPARQL and OWL 2 activity patterns. Pervasive and Mobile Computing, 25, 104-124.
- 16. End-result: Signals + Activities
- 17. Applications & Pilots Use Cases, Clinical Approaches, Interventions
- 18. EU FP7 ICT Project • Dementia Ambient Care 2011 – 2015 • Multimodal sensing • 3 use cases • 300+ participants
- 19. @Lab • Controlling equipment • Dementia Assessment in a controlled environment
- 20. @Lab Results • 290 participants Nice & Thessaloniki • 80% accuracy Healthy - MCI - AD
- 21. @Home • System-supported interventions • 4 in Dublin, 6 in Thessaloniki for 4 – 12 months • Improvement or non-deterioration in cognitive state • Compared to non-system-supported interventions or regular care
- 22. Apps for self-management • Confidence, inclusion
- 23. Clinical Observations - Correlation between metrics Increased Physical Activity – Decreased Sleep Latency Increased Physical Activity – Increased Sleep Duration
- 24. Clinical-Neuropsychological Assessment • 6 Users • 4 with Mild Cognitive Impairment (MCI) • 2 with Alzheimer’s Disease (AD) • Neuropsychological and Clinical Evaluation • Baseline (1st half) and Follow-up Assessment (2nd half) • Standardized and Validated Cognitive Test • Tailored non-pharmaceutical interventions (4-12 months) i.e., Cognitive Behavioral psychotherapy, reminiscence, memory exercises etc Tests Assessment Time Users 1-6 M SD MMSE 1st half 26.50 3.56 2nd half 28.33 1.86 NPI 1st half 1.67 0.52 2nd half 0.83 1.33 FRSSD 1st half 3.83 3.13 2nd half 2.67 2.66 FUCAS 1st half 48.67 8.48 2nd half 44.00 2.45 RAVLT - copy spelling 1st half 4.83 1.72 2nd half 5.50 2.43 RAVLT - learning 1st half 7.67 3.01 2nd half 9.00 4.05 RAVLT - recall 1st half -3.50 1.05 2nd half -0.50 5.17 RAVLT total 1st half 38.67 13.53 2nd half 45.83 15.94
- 25. Clinical Observations User 1 User 2 User 3 User 4 User 5 User 6 Total Time Asleep (hours) 1st half 7.24 6.4 8.09 6.39 5.98 6.24 2nd half 8.14 7.36 8.62 6.47 7.1 7.58 P 0.001 0.0001 0.17 0.09 0.02 0.003 Number of Interruptions 1st half 2.32 5.52 5.8 3.74 3.72 3.67 2nd half 2.13 5.85 3.8 2.25 2.4 4.75 P 0.44 0.57 0.001 0.02 0.0001 0.001 Shallow Sleep (hours) 1st half 4.36 2.31 3.78 3.88 3.00 2.67 2nd half 3.88 3.04 3.67 4.37 3.35 3.1 P 0.01 0.02 0.64 0.36 0.0001 0.02 Sleep Latency (min) 1st half 6.83 0.47 8.8 10.11 5.81 8.8 2nd half 3.33 0.46 8.4 5.1 5.5 8.4 P 0.009 0.95 0.88 0.03 0.59 0.02 Deep Sleep (hours) 1st half 1.61 0.9 1.83 1.83 1.35 1.13 2nd half 1.94 1.22 2.13 2.13 1.42 1.54 P 0.02 0.0001 0.04 0.13 0.54 0.0001 Physical Activity (min) 1st half 56.07 43.31 68.62 33.91 13.1 109.7 2nd half 57.16 43.17 74.25 35.13 43.26 112.94 P 0.22 0.91 0.0001 0.29 0.04 0.72 • Improvement in Sleep Parameters and Physical Activity in the 2nd half of observational period
- 26. EU H2020 IoT LSP Project • EU IoT Large Scale Pilots 2017 – 2020 • 9 Deployment Sites All Partners 25.772.829 € MEDTRONIC (coord) & 40 partners • Integrate major open IoT platforms • Active and Healthy Ageing (AHA) applications • 9 Use Cases • Mobility & Transport, emergency, home activity & behavioral monitoring etc. • A Marketplace to discover and install apps
- 27. Activage IoT Ecosystem Suite - AIOTES • Not only dementia but any scenario of Active and Healthy Ageing • Allows you to build and monetize eHealth apps over 9 open European platforms • 2nd open call to be launched soon
- 28. National Project • EU-funded Erevno-Kainotomo-Dimiourgo 2018 – 2020 • 75 Home Participants • Wearable & Apps to support Alzheimer • Minimum equipment to support easy deployment and maintenance • Promoted to national Telecom providers All Partners 587.450 € Role CERTH Thessaloniki IoT Integration, AI Analysis, RnD ARX .NET Thessaloniki Mobile Apps, Business Dev Frontida Zois Patras Clinical Pilots & Evaluation
- 29. IMI2 RADAR AD • EU and EFPIA (Pharma) co-funded and co-developed 2019 – 2021 • Platform for Alzheimer’s “digital biomarkers” brought closer to medical practice • Building over RADAR-CNS platform and trials for multiple diseases • 3-Tiers of Clinical Trials • 1) Small-scale Homes 2) Large-scale Homes 3) CERTH Smart Home - multiple equipment and incubator environment
- 30. CERTH-ITI Smart Home • Rapid-prototyping & demonstration for actual living scenarios • 1st near Zero Energy Building in Greece • 2-story, 4-bedroom, 3-bathrooms, living room, kitchen • one of the most important pillars of the Digitise European Industry effort • Equipment • Smart Home • Energy – Solar Panel • Open to any choice of RADAR AD Tier 3 ROBOTICS & AI HEALTH ENERGY BIG DATA
- 31. User-acceptance and Perspective • A Patient-Advisory Board (PAB) to select devices • Game with cards to extract preferences (1st meeting in Luxembourg) • Anonymously select device representatives and aspects • Top voted aspects are • Appearance and Style, Weight, Water-proof, Emergency button feature and Battery life
- 32. Other Projects - Wearable Cameras for the Blind • eVision • National Pilot Project • Wearable Cameras for the blind • SUITCEYES • EU Pilot Project • Suit & Wearable Cameras for the deafblind All Partners 660.075 € CERTH Thessaloniki TETRAGON LTD Thessaloniki PRISMA Alexandroupoli MASOUTIS Thessaloniki THESSALONIKI MUNICIPALITY Thessaloniki All Partners 2.359.963 € CERTH Thessaloniki UNI BORAS Sweden UNI OFFENBURG Germany UNI LEEDS UK UNI EIDHOVEN NL LES DOIGTS France HARPO SP Poland
- 33. Conclusions • IoT Constantly evolving – especially wearables • New modalities, reliability, acceptance and user-related parameters • Platform flexibility, also to pilot – deployment scale and complexity • Sensors, analytics, visualization, interpretation can assist staging and interventions • Difficult to differentiate between actual contribution and enhanced social activity • Large-scale pilots and security aspects • Big Data – Machine Learning approaches • Emphasis on physical and cognitive interventions • EEG mobile devices • New applications • Outdoor environment, smart cities • Working environments, e.g. Mental Health of Employers
- 34. Thank You Email : ikom@iti.gr Links Projects: demcare.eu activageproject.eu radar-ad.org Lab & all other scientific activity mklab.iti.gr Videos Lab Trials - https://www.youtube.com/watch?v=AEuX58HLIDo Home Monitoring - https://www.youtube.com/watch?v=0JNlaM6BpMA CERTH-ITI Smart Home Video - https://www.youtube.com/watch?v=8pcw1Xhk240
Editor's Notes
- #3: We all know the importance of fighting the increasing upsurge of people with dementia and other chronic diseases. The market opportunity for dementia care products is huge as million new cases of dementia occur each year. What many don’t know/An interesting fact is that currently the only known treatment are psychological interventions, such as brain games, exercise, consuming natural products e.g. tea. These are driven/designed by nurses or psychologists observing people, which makes them costly & error prone.
- #5: In carealia we utilse Interconnected IoT sensors such as retail wearables gather vitals & various raw data signals These are then interpreted and fused into more meaningful symptoms and behavioral patterns related to the disease, such as stress etc. Monitoring anytime through web & mobile applications supports doctors to tailor interventions & care while supporting users & their loved ones feel included.
- #16: SPARQL to query and retrieve observations C1, C2, etc (e.g. CupMoved, KettleMove, KettleOn, KitchenPresense) from the Knowledge Base Then try to form clusters of observations which form activities, according to the ontology context descriptors (e.g. MakeTea) This is done with our own algorithm which performed as follows (see Table)
- #25: Bold - purple statistical significant difference between two neuropsychological assessments
- #26: P values show Statistical Significant Difference between 1st and 2nd half mean values (bold - purple)