Best Practices in Testing Biometric Wearables
1 like242 views
The document outlines the operations and methodologies of the Valencell Biometrics Laboratory, which specializes in testing biometric wearables. It details the lab's workflow, participant requirements, data collection processes, and analysis techniques, emphasizing the extensive testing conducted on various devices each year. Additionally, it covers comfort and security ratings for devices, as well as data analysis metrics used to ensure accuracy in biometric measurements.
Best Practices in Testing Biometric Wearables
- 1. Best Practices in Testing Biometric Wearables © 2017 Valencell, Inc Chris Eschbach, PhD, Director Biometrics Laboratory
- 2. © 2017 Valencell, Inc Webinar outline Biometrics Laboratory overview • Biometrics team • Lab workflow and charge Data collection process • Areas for consideration Data analysis and reporting
- 3. © 2017 Valencell, Inc Webinar outline Biometrics Laboratory overview • Biometrics team • Lab workflow and charge Data collection process • Areas for consideration Data analysis and reporting
- 4. © 2017 Valencell, Inc The Valencell Biometrics Lab is a cutting edge biometric science lab designed to provide comprehensive analysis of the latest in biometric sensor and wearable technology. • Team of exercise scientists lead by PhD Exercise Physiologist Dr. Chris Eschbach • Pool of nearly 100 volunteers testing prototypes and new technology every week • Validation datasets on participants of numerous body types, fitness levels, gender, & skin tones • Testing protocols that match the use cases - resting, lifestyle activities, mild exercise, aggressive exercise, interval training, etc. • Testing facilities for running, biking, swimming, gym activities, and lifestyle activities Valencell Biometrics Lab Overview Biometrics Lab testing each year: • Conducts over 24,000 different device tests • Analyses over 12 million biometric data points • Measures over 1,200 hours of testing & validation sessions
- 5. © 2017 Valencell, Inc Valencell Biometrics Lab Overview
- 6. © 2017 Valencell, Inc Biometrics laboratory operations & staff Operations • Exercise sessions (R & D + validation): 40 per week • Exercise sessions (validation commercial/training programs): 20 per week • Other trials (longitudinal and rest): 30 per week Staff • Director • Operations lead • Data lead • Reports lead • Trials specialists
- 7. © 2017 Valencell, Inc Biometrics Laboratory Workflow • Scheduling • Prioritization • Experimental design • Device/Experiment intake • Reporting
- 8. © 2017 Valencell, Inc Webinar outline Biometrics Laboratory overview • Biometrics team • Lab workflow and charge Data collection process • Areas for consideration Data analysis and reporting
- 9. © 2017 Valencell, Inc Data collection process Internal or external trials and projects • Participants • Data collection • Areas for consideration Data analysis and reporting
- 10. © 2017 Valencell, Inc Participants • Requirements: age, health, and availability • Participant characteristics • Health history • Laboratory introduction • Anatomical measures • Pictures • Goal for all DUTs: 20 participants with 8-min dynamic treadmill test
- 11. © 2017 Valencell, Inc Participants- anatomical measures
- 12. © 2017 Valencell, Inc Participants- Fitzpatrick Skin Type
- 13. © 2017 Valencell, Inc Participants - comfort and security Comfort Rating Comfort Description 1 • Painful and extremely uncomfortable. • Worse than any other earbuds you have used. • Would not wear longer than 15 minutes. 2 • More uncomfortable than preferred. • Below average compared to other earbuds. • Would not wear longer than 1 hour. 3 • Neither comfortable nor uncomfortable. • Average compared to other earbuds. • Would wear up to 2 hours. 4 • Comfortable. • Above average compared to other earbuds. • Would wear up to 3 hours. 5 • Extremely comfortable. • Better than any other earbuds you have used. • Would wear for 4 hours or longer. Security Rating Security Description 1 • Won’t stay in ear. • Worse than any other earbuds you have used. 2 • Will stay in place while still, but slides out of place with movement. • Below average compared to other earbuds. 3 • Feels secure at rest and moderately insecure while running but doesn’t fall out. • Average compared to other earbuds. 4 • Feels secure at rest and stable while running but falls out with a slight tug on the cord. • Above average compared to other earbuds. 5 • Extremely secure at rest and while running. • Better than all other earbuds.
- 14. © 2017 Valencell, Inc Data collection process Internal or external trials and projects • Participants • Data collection • Areas for consideration Data analysis and reporting
- 15. © 2017 Valencell, Inc Data collection process • Metrics collection • Raw data • Heart rate, RRi, steps, blood pressure, etc. • Calories, VO2, Speed, Distance, etc. • Baseline devices • Chest strap HRM, ECG, Video, Manual and automatic BP • Indirect calorimetry, measured courses, and calibrated treadmills
- 16. © 2017 Valencell, Inc Data collection process - laboratory tools • Wide range of participants • Experienced staff • Databases and data experts to overcome data collection and UX challenges • Data collectors (Raw, multi BLE, single BLE): commercial applications and custom • Exercise equipment • Treadmills, bikes, strength, lifestyle, etc • Reference devices (baseline and procedures for them) • Collection devices: phones, cameras, programs, computers
- 17. © 2017 Valencell, Inc Data collection process - laboratory tools Reference devices (baseline and procedures for them)
- 18. © 2017 Valencell, Inc Data collection process • Protocol matches the intended use case(s) • Participant places devices (followed by instruction as needed) • Metrics collection- Heart rate • DUT - Ideally raw data collected for analysis • If not, then BLE HR • Lastly some proprietary HR output • Post processing to provide heart rate at 1 second intervals
- 19. © 2017 Valencell, Inc Data collection process - protocols Data is collected for the following 8-minute protocol:
- 20. © 2017 Valencell, Inc Data collection process - protocols
- 21. © 2017 Valencell, Inc Data collection process - protocols • Various field trials beyond the foundational protocol (8-min dynamic treadmill) • Sunlight testing • Lifestyle (typical daily movements) • Resting heart rate • Dynamic bike indoor (10-minutes) • Running intervals (15-minutes) • Free run/bike (>20 minutes) • Rowing ergometer (13-minutes) • Strength circuit (typical strength exercises) • High intensity interval strength • Swimming
- 22. © 2017 Valencell, Inc Data collection process - CTA protocols • Consumer Technology Association • Co-chair, Health Fitness and Wellness Technology Standards Committee • Physical Activity Monitoring for Fitness Wearables: Step Counting • Definitions and Characteristics for Wearable Sleep Monitors • Methodology of Measurement for Features in Sleep Tracking Consumer Technology Devices and Applications • Physical Activity Monitoring for Heart Rate • Consumer Stress Monitoring Technologies https://cta.tech/Research-Standards/Standards-Listing/Health-Fitness-Technology-Standards.aspx
- 23. © 2017 Valencell, Inc Data collection process – areas of consideration Source: “Optical heart-rate measurement’s top 5 challenges” Dr. Steven LeBoeuf; EDN Magazine; 8-25-15 Optical Noise Skin Tone Blood Perfusion Sensor Location Crossover Problem
- 24. © 2017 Valencell, Inc Webinar outline Biometrics laboratory overview • Biometrics team • Lab workflow and charge Data collection process • Areas for consideration Data analysis and reporting
- 25. © 2017 Valencell, Inc Data analysis and reporting • Both subjective and quantified scoring of data • Provides confidence with all data • Easily understood during design cycles • Erratic data: physiologically unlikely in a health individual. • Highly variable heart rate “jitter” • Reported heart rate outside of expected ranges (for example, reported at 0 or greater than 200)
- 26. © 2017 Valencell, Inc Data analysis and reporting • Excellent • Signal was strong and tracking good the entire session • Good • Signal was strong and tracking was good but occasional short segments of mis-tracking occurred • A moderately scored test would likely not be noticed by a consumer • Lost tracking • Segments of moderate to long duration where biometric was not correct
- 27. © 2017 Valencell, Inc Data analysis and reporting Subjective results: • Reference device • DUT • Excellent: >90% within +/-5% • Good: >80% within +/-5% • Poor: <80% within +/-5% and/or mis-tracking outside of +/- 10% for greater than 40 seconds
- 28. © 2017 Valencell, Inc Data analysis and reporting • Statistical analysis • mean ± SD (when appropriate) • bias as percent error • mean absolute percent error (MAPE) • frequency distribution within ±5% windows • Utilize thresholds: 85% of data will be within ±5% • Bland-Altman and correlation as needed • accuracy related to heart rate intensity (≤ moderate and ≥ vigorous)
- 29. © 2017 Valencell, Inc Data analysis and reporting – areas for consideration • Baseline errors (remove but report errors) • Time for HR lock • Data alignment • Latency
- 30. © 2017 Valencell, Inc Validation Report - representative data setsInitials (1st 2 last 2) Unit ID Date and time Picture Core Gel Size Ear Tip Size Needs Warm-up Good Mod Fail GeSc XXX5.58.3.M#8 7/10/14 10:50 M 1 CiSh DoCa HuRo XXX5.58.3.M#8 7/10/14 14:44 M 1 FaSa JuSa XXX5.58.3.M#8 7/21/14 10:20 M 1 VeSo DaBa XXX5.58.3.M#8 7/21/14 16:20 M 1 MiYo XXX5.58.3.M#8 7/22/14 9:25 M 1 JaAy XXX5.58.3.M#8 7/15/14 10:03 M 1 AnSu XXX5.58.3.M#8 7/22/14 11:34 M 1 ThYo PhMu ScSa RiBr JoVa TrMa XXX5.58.3.M#8 7/14/14 15:33 M 1 JaSt XXX5.58.3.M#8 7/16/14 11:50 M 1 WaFu XXX5.58.3.M#8 7/16/14 12:11 M 1 Erma XXX5.58.3.M#8 7/21/14 13:31 M 1 JaJa XXX5.58.3.M#8 7/21/14 15:37 M 1 JeSc XXX5.58.3.M#8 7/14/14 9:36 S 1 HoSh LoWh JiWh RiGr StDa XXX5.58.3.M#8 7/10/2014 9:13 M 1 TrBo XXX5.58.3.M#8 7/10/2014 9:52 M 1 SeLo XXX5.58.3.M#8 7/10/2014 13:37 M 1 ChEs XXX5.58.3.M#8 7/10/2014 14:09 M 1 FrTh AnSm JoSm XXX5.58.3.M#8 7/21/14 14:25 M 1 HeBa BoSm XXX5.58.3.M#8 7/16/14 14:26 M 1 BeSt XXX5.58.3.M#8 7/16/14 17:36 M 1 Total 0 13 3 4 % 0% 65% 15% 20% Grand Total 20 0% 65% 15% 20% Breakdown of Performance Needs Warm- up Good Mod Fail Average HR (BLECS | BE1.2) 125 | 126 Bias 0.02% SD of Bias ± 1.9% MAPE 1.0% <-15% 0% -15% to -10% 0% -10% to -5% 1% 5% to 5% 98% 5% to 10% 1% 10% to 15% 0% >15% 0% Average HR (BLECS | BE1.2) 125 | 126 Bias 0.4% SD of Bias ± 3.2% MAPE 2.6% <-15% 1% -15% to -10% 1% -10% to -5% 2% -5% to 5% 91% 5% to 10% 2% 10% to 15% 1% >15% 2%
- 31. © 2017 Valencell, Inc Validation Report - representative data sets
- 32. © 2017 Valencell, Inc Validation Report - representative data sets
- 33. © 2017 Valencell, Inc Validation Report - representative data sets
- 34. Q&A Thanks for your time! If you have further questions or would like to discuss testing further, please reach out at info@valencell.com