Track 1 session 1 - st dev con 2016 - contextual awareness
0 likes1,337 views
This document provides an agenda and presentation slides for a contextual awareness solutions event. The agenda includes an introduction, presentations on applications, algorithms, and solutions integration. The slides cover various applications of context-aware systems, building algorithms to recognize context through low-level and high-level contexts, sensor solutions enabled by MEMS sensors, activity recognition, gesture recognition, and free sensor fusion and activity recognition software. The presentation aims to demonstrate diverse applications of contextual awareness systems and how algorithms and sensors can be used to recognize context and take appropriate actions.
![Understanding High Level Context (1/2)
Motion activity vector (MAV)
= [stationary; walking; jogging; escalator; elevator; bicycling; driving;
none of these]’
Voice activity vector (VAV)
= [silence; face-to-face talking; phone conversation; none of these]’
Spatial environment vector (SEV)
= [street; nature: garden/park; beach; stadium; office;
mall/supermarket/restaurant/cafeteria; home; in moving vehicle;
lecture/conference room; none of these]’
11](https://image.slidesharecdn.com/track1-session1-stdevcon2016-contextualawareness-161007094652/85/Track-1-session-1-st-dev-con-2016-contextual-awareness-11-320.jpg)
Track 1 session 1 - st dev con 2016 - contextual awareness
- 1. October 4, 2016 Santa Clara Convention Center Mission City Ballroom Contextual Awareness Solutions MEMS Sensor Solutions Software Team
- 2. 2:50 Introduction Sankalp Dayal 3:30 Applications Algorithms Solutions Integration Agenda Time Speaker Presentation 2
- 3. Sense (Sensors) Recognize Context (Solutions) Take Action (Your Magic) New Things to augment life Existing Things augmented 3
- 4. Applications • Diverse applications of context-aware systems • Smart Me • Smart Home • Smart Office • Industrial • Healthcare • Fitness • Gaming • Virtual/Augmented Reality • Human Machine Interaction • Security and Monitoring • …… 4
- 5. Some Applications of context-Aware systems • HealthCare: • Monitoring posture and patterns to prevent conditions before they actually happen. • Actively support elderly or disabled people in performing everyday activities • System to detect potentially dangerous situations in a person’s life • Industrial Applications • Activity-aware applications can support workers in their tasks, help to avoid mistakes and increase workplace safety • Multimodal sensing using on-body and environmental sensors to determine context and user’s current activity • Entertainment and Games • Provide feedback to athletes, artists in performing arts, etc, to augment their performance • Entertainment based on context aware applications for mood recognition and then using playlist recommendation 5
- 6. Building Algorithms to Recognize Context • Real value is in recognition of high level contexts. • Shopping in a mall, having lunch with friends, busy with office work • Classification of low-level contexts form the basis to model and infer more complex contexts. • e.g. Shopping in mall requires location, human activity, human object interaction • Low Level Contexts • Human Activity • Human Environment • Human Location • Human Interaction (with human or objects) 6
- 7. Generic Context Awareness Architecture 7
- 8. Understanding Human Activity • Gestures/ Movements/ Motifs • Brief and distinct body movements, • e.g. taking a step, bending the arm • (Low-Level) Activities • Sequence of movements/ a distinct posture, • e.g. walking, sitting, cleaning windows • High-Level Activities/ Scenes/ Routines • Collection of activities, • e.g. office work, lunch, shopping at mall 8 Seconds Minutes Hours
- 9. Accelerometer Barometer Gyroscope Windowing and Feature extraction Windowing and Feature extraction Compute P(Activity/Features) in Motion Activity Posteriorgram Compute P(Activity/Features) in Motion Activity Posteriorgram Compute P(Activity/Features) in Motion Activity Posteriorgram Windowing and Feature extraction Microphone Windowing and Feature extraction Compute P(Activity/Features) in Motion Activity Posteriorgram Activity Inference Human Motion Activity Detection Architecture 9
- 10. Understanding Human Environment 10 Microphone Windowing and Feature extraction Compute Prob. (Environment/Features) in Spatial Environment Posteriorgram
- 11. Understanding High Level Context (1/2) Motion activity vector (MAV) = [stationary; walking; jogging; escalator; elevator; bicycling; driving; none of these]’ Voice activity vector (VAV) = [silence; face-to-face talking; phone conversation; none of these]’ Spatial environment vector (SEV) = [street; nature: garden/park; beach; stadium; office; mall/supermarket/restaurant/cafeteria; home; in moving vehicle; lecture/conference room; none of these]’ 11
- 12. Understanding High Level Context (2/2) Time sequence of Motion Activity Posteriorgram (MAP) Time sequence of Voice Activity Posteriorgram (VAP) Time sequence of Spatial Environment Posteriorgram (SEP) Embedded Application producing Soft Decision of High-Level Context 12
- 13. Building Good Solutions • Accurate • Large user testing, diverse scenarios • Low Power • Smart Sensor Usage, Hardware Accelerators, On Sensor Computation • Low Footprint • Minimalist models, optimized software • Flexible • Multi Platform, Tunable, Personalization 13
- 14. ST Sensor Solutions enabled by MEMS Sensor fusion for device orientation (Any Device) • Gives rotation, orientation, quaternions, linear acceleration, calibration User Activity recognition (Mobile/Wrist) • Travel mode, Pedestrian mode, workout, sports Gesture recognition (Mobile/Wrist) • Glance, pick-up, look-at, shake, tap, swipe, rotate, symbols, user defined Carry position (device placement on body) determination (Mobile) • Shirt pocket, holster, trouser pocket, backpack, handbag, near the head, .. Pose Detection (Wrist) • Detect Standing, Sitting, Lying Down 14
- 15. ST Sensor Solutions enabled by MEMS Stairs Counting/Vertical Context (Any Device) • Elevator, Escalator, Stairs Up/Down, Number of stairs, Floor Change Bicep Curl Detection and Counting (Wrist) • Detects bicep curls and counts reps Situational awareness using Audio + Motion sensors • Environment detection, number of speakers, key word spotting Pedestrian Dead-Reckoning (PDR) • Indoor location, Augmenting GPS based location 15
- 16. Sensor Fusion osxMotionFX • Magnetometer calibration routine • Gyroscope bias compensation • Dynamic distortion (hand jitter), measured by the accelerometer • Power saving osxMotionFX – GUI 16
- 17. Activity Recognition 17 osxMotionAR provides real-time information on user activity: • Stationary • Walking • fast walking The algorithm manages only the data acquired from the accelerometer at a low sampling frequency of 16 Hz, contributing to reduction of power consumption of the hosting platform. The application recognizes the performed activities and stores them in the on-board memory for offline analysis. • Jogging • biking • driving osxMotionAR
- 18. Activity Recognition Test Results 18 Stationary Walking Fast Walking Jogging Biking Driving Detection Probability 16279 1 0 0 98 1431 Stationary 91.41% 3 49030 51 9 483 25 Walking 98.85% 0 116 3143 6 10 3 Fast Walking 95.88% 0 14 11 2781 8 2 Jogging 98.76% 63 132 4 0 5292 633 Biking 86.41% 1113 6 1 0 436 7912 Driving 83.57% Classified As Actual Activity Test Data • 682 data sets (71 unique individuals) 48 hours of activity data • Activities included (stationary, walking, fast walking, jogging, vehicle, bicycle) for different carry positions (body placement) • Pedestrian: Trouser pocket, in-hand, shirt pocket, in back pocket, near-the-head, .. • Vehicle: in cup-holder, in-shirt pocket, in- trouser pocket, .. • Bicycle: in-shirt pocket, in-trouser pocket
- 19. Gesture Recognition osxMotionGR provides real-time information on Gestures: • Glance • Pick-up • Wake-up The algorithm manages only the data acquired from the accelerometer. Designed for always-on operation. Machine learning algorithms are used for the feature The application recognizes the performed activities and stores them in the on-board memory for offline analysis. 19 osxMotionGR
- 20. Carry Position Recognition osxMotionCP provides real-time information on Carry Position: • On-Desk • In-hand (like texting) • Near Head (talking on Phone) The algorithm manages only the data acquired from the accelerometer. Uses Machine Learning based models to achieve higher accuracy The application recognizes the performed activities and stores them in the on-board memory for offline analysis. • Shirt Pocket • Trouser Pocket • Arm Swing 20 osxMotionCP
- 21. Pose Estimation Estimates pose of human body • Standing • Sitting • Lying Down The algorithm manages only the data acquired from the Pressure Sensors at 10 Hz and Accelerometer Data at 16 Hz Precise pressure sensor allows to differentiate between standing and sitting at desk Uses Machine Learning models to achieve higher accuracy Performance: maximum latency < 2 sec 21 Pose Estimation Library
- 22. Scenarios for Data Collection Hands in trouser pocket - standing Head resting on hand - sitting Hand resting on side of couch while watching TV Eating with Fork and Knife Driving Cooking Hands resting on thigh/pillow while watching TV Holding phone in hand Reading or Texting - standing Holding phone in hand Reading or Texting -sitting Holding a cup of coffee/can of soda - standing Holding a cup of coffee/can of soda - sitting Talking on phone – standing Talking on phone - sitting Washing face Talking with gestures - standing Talking with gestures - sitting Typing Mouse Handling Doing dishes Brushing teeth Shaving Hands in jacket Writing Sitting vs Standing Confusion Matrix Activity Duration Sitting Standing Probability of Detection Sitting 7386 97.49 2.51 97.49 Standing 3843 11.04 88.96 88.96 Total Duration 11229 22
- 23. Vertical Context Detection 23 Detects: • Elevator • Escalator • Stairs Up/Down (uses step detection function) • On-floor The algorithm manages only the data acquired from the Pressure Sensors at 10 Hz. Corrects drift to achieve higher accuracy Able to prevent drift < 50 cm. Performance: maximum latency < 2 sec Vertical Context Library
- 24. Bicep Curl Detection • Detects Bicep Curls • Counts number of repetitions The algorithm manages only the data acquired from the Pressure Sensor at 25 Hz and Accelerometer at 25 Hz Corrects drift to achieve higher accuracy Precise pressure sensor allows to differentiate between random motion and actual bicep curl motion Uses Machine Learning models to achieve higher accuracy Acc X Acc Y Acc Z θ angle Vertical direction 24 Bicep Curl Detection Library
- 25. Start Building Your Own Solution In less than 5 minutes 25
- 26. Steps to Build Your Own Solution 26 Choose Preferred Hardware Download Firmware Get Library License Install App Activity, Carry, Gesture, Pedometer, Vertical Context, Pose, … Software Developer’s Platform $10 $86 Free Free Free • Gerber files • BOM • Schematics • Source Code • Application Note • User Guide • Getting Started • API • Library • Free License • Data Logging & Viewing • Complete Project • Source Code
- 27. STM32 Open Development Environment STM32 Nucleo 27 One STM32 MCU flavor with 64 pins Arduino™ extension connectors Easy access for add-ons Morpho extension headers direct access to all MCU I/Os 2 push buttons, 2 color LEDs Flexible board power supply Through USB or external source Available for STM32F401, STM32L152, STM32F030 and STM32F103. Other STM32s coming soon www.st.com/stm32 Integrated ST-Link/V2-1 Mass storage device flash programming
- 28. Features • Complete solution with STM32 Nucleo board includes: • LSM6DS0: MEMS 3D accelerometer + 3D gyroscope • LIS3MDL: MEMS 3D magnetometer • LPS25HB: MEMS pressure sensor, (absolute digital output barometer) • HTS221: capacitive digital relative humidity and temperature • DIL 24-pin socket available for additional MEMS adapters and other sensors (UV index) Product Specification Gerber Files Bill of Material Schematics SW Packages download HW description of the X-Nucleo Getting Started HW User Manual of the X-Nucleo Quick User Guide User Manual for SW packages Motion & Environmental Sensors STM32 Nucleo Expansion Board 28 X-NUCLEO-IKS01A1
- 29. SensorTile Core System Sensing, processing and BLE connectivity Sensors Ultra Low Power Connectivity Low-Power MCU MP34DT04 LPS22HB LSM6DSM LSM303AGR STM32L4 BlueNRG-MS • SD Card Slot for Extra Memory • Wearable Form Factor • Easy Programmability 13.5 mm 13.5mm Miniaturized Tile that can be soldered or plugged on a host board Motion MEMS Environmental sensors MEMS microphone Low-power brain Sensor fusion Bluetooth Smart 29
- 30. Free SW Licensing Program 30 Unleashing the power of embedded software with algorithms and complete demonstrators for the Internet of Things. Bring your ideas to now!
- 31. 31
- 32. 32
- 33. OpenSoftware The Middleware Libraries 33 Open Software – Middleware Libraries • Open.MEMS: sensor processing libraries • Open.RF: protocols and profiles for the wireless connectivity • Open.Audio: libraries and processing algorithms for audio capturing systems based on digital MEMS microphones Includes several free libraries: • Open.MEMS • Open.RF • Open.Audio Open.Framework (Implementation Examples): • BlueMicrosystem • BlueVoiceLink
- 34. Mobile and PC Apps 34
- 35. .Thank You!