Starwest 2017 IoT testing
3 likes834 views
This document discusses testing challenges for IoT applications in healthcare. It describes how sensors require invasive and independent testing against different devices and versions to ensure safety and accuracy. Testing sensors directly is difficult, so functions may move to smartphones. Connected apps also need rigorous testing using real and synthetic sensor data to validate the application logic and behavior. An effective test approach is proposed using a "Bleno Box" device simulator and JSON files to automate testing the application response to various data inputs across environments while meeting requirements for coverage, automation, scalability, compliance and reporting.
![IoT Data-Driven Testing (DDT): JSON File Example
// This file exemplifies the syntax used for creating a BLE Peripheral
// It defines the following Services: Health Thermometer, Blood Pressure, Heart Rate, Battery
name : "HMO", // The Peripheral name ("Health Maintenance Organization")
services : [ {// This service is defined according to https://www.bluetooth.com/specifications/…org.bluetooth.service.health_thermometer.xml
name : "Health Thermometer", // The service name may be used by a script for updating it
uuid : 1809, // Service assigned number as defined in the link above
characteristics : [ { // See https://www.bluetooth.com/specifications/…org.bluetooth.characteristic.temperature_measurement.xml
name : "Temperature Measurement", // The service name may be used by a script for updating it
uuid : 2A1C, // The Characteristic assigned number as defined in the link above
value : [
{float : 36.2}, // Temperature in centigrade (Floating point)
0x07 // Temperature type (See https://www.bluetooth.com/specifications/…org.bluetooth.characteristic.temperature_type.xml)
],
properties : [read, notify], // Characteristic is readable and notifiable
interval : 30000 // Notification interval is 30 seconds
},{
name : "Temperature Type", // See https://www.bluetooth.com/specifications/gatt/viewer?attributeXmlFile=org.bluetooth.characteristic.temperature_type.xml
uuid : 2A1D,
value : [0x06], // Mouth
…](https://image.slidesharecdn.com/starwest2017iottesting-171005232708/85/Starwest-2017-IoT-testing-20-320.jpg)
Starwest 2017 IoT testing
- 1. Transforming Agile Digital Quality Testing in the IoT Era Amir Rozenberg, Director of Product
- 2. BUT: I still have ~20 GitHub projects! • Product manager by day, coder at night • Responsible for the core platform at Perfecto • Publications: DZone, CIO Magazine, MobileTestingBlog.com etc. • I toy with • Security & performance testing • Globalization • Elasticity • Music and video streaming quality • … Amir_rozenberg
- 3. 10/5/2017 3© 2015, Perfecto Mobile Ltd. All Rights Reserved. Agenda IoT Driving Digital Transformation Quality Requirements for Healthcare Apps Achieving Success with IoT Apps Q&A
- 4. It’s Here It’s Massive It’s Impactful
- 5. How does IoT fit into your Agile Strategy?
- 6. Different IoT Segments Source: Charalampos Doukas
- 7. Digital IoT Healthcare Improves Patient Care
- 9. IoT Healthcare Example: Application Architecture (Pro/Reactive) Sensors Connected Apps Cloud Operations Healthcare Ecosystem • Are the measurements correct? • Is the action correct? • Is the data sent correct? • Given data, does the app behave as expected? • Feature/function • Compliance • Backend processes • Sharing • Compliance
- 11. Sensors Requirements • Proactive, Invasive • Independent (not relying on backend availability) • Class 3 FDA approval • Certification against every device, OS, version, Bluetooth version • Cannot leave premises • Retail/Medical • Reactive- informational • Semi-Independent
- 12. How to test the sensors? : Firmware approach Test Considerations • LONG development cycles • Extensive compliance requirements and process • Follows embedded SW practices Examples of Test Practices • Testing mostly HW/FW oriented • Connectivity (BLE): On/Off • No API- No way to automate • Test bench onprem • Scale challenges (especially proactive devices) • Inability to ‘shrink time’
- 13. How to test the sensors? : Simulators (fake device, same SW) Examples of Test Practices • Simulated sensor data • Test application logic • Test firmware update process • Trend responsiveness of action Test Challenges • Vendor typically owns the simulator • No/Few APIs, no automation • No connectivity testing • Test bench onprem
- 14. Testing real sensors is hard, functions will move to smartphones
- 16. Healthcare Applications Mission critical: saving lives and improving patient care Treatment research Streamlining healthcare processes
- 17. Data-driven testing • Mainstream data • Extreme data • Data at different frequency • Long term data • Mix of conditions Auth Data intake AUX functions Dev-Test-Staging-Prod Stub Backend Testing the App (Given sensor data)
- 18. Testing the App (Given sensor data) BLENO Box 20-30 m Value: 140 BPM Object (Json file) Auth Data intake AUX functions Dev-Test-Staging-Prod Stub Backend
- 19. The Bleno Box BLENO Box Docker Image Bleno 3rd Party Code (JS) Spring Front-end nodejs Bluetooth Library Linux • Script sends ‘Create’ message • Contains a Json file • File is translated to JS code, based on Bleno classes • JS is executed with nodejs
- 20. IoT Data-Driven Testing (DDT): JSON File Example // This file exemplifies the syntax used for creating a BLE Peripheral // It defines the following Services: Health Thermometer, Blood Pressure, Heart Rate, Battery name : "HMO", // The Peripheral name ("Health Maintenance Organization") services : [ {// This service is defined according to https://www.bluetooth.com/specifications/…org.bluetooth.service.health_thermometer.xml name : "Health Thermometer", // The service name may be used by a script for updating it uuid : 1809, // Service assigned number as defined in the link above characteristics : [ { // See https://www.bluetooth.com/specifications/…org.bluetooth.characteristic.temperature_measurement.xml name : "Temperature Measurement", // The service name may be used by a script for updating it uuid : 2A1C, // The Characteristic assigned number as defined in the link above value : [ {float : 36.2}, // Temperature in centigrade (Floating point) 0x07 // Temperature type (See https://www.bluetooth.com/specifications/…org.bluetooth.characteristic.temperature_type.xml) ], properties : [read, notify], // Characteristic is readable and notifiable interval : 30000 // Notification interval is 30 seconds },{ name : "Temperature Type", // See https://www.bluetooth.com/specifications/gatt/viewer?attributeXmlFile=org.bluetooth.characteristic.temperature_type.xml uuid : 2A1D, value : [0x06], // Mouth …
- 22. Your Success Criteria: Lab Maximize Test Coverage Maximize % of Automation Compliance Scalable Reporting
- 23. Your Success Criteria: Reporting Maximize Test Coverage Maximize % of Automation Compliance Scalable Reporting
- 24. Your Success Criteria: Network Traffic Compliance Maximize Test Coverage Maximize % of Automation Compliance Scalable Reporting HAR File • What is impacting the responsiveness of my app? • What servers am I interacting with? • What data is exposed in my service API calls?
- 25. Your Success Criteria: Compliance Maximize Test Coverage Maximize % of Automation Compliance Scalable Reporting Uncompromised Security • Network Operations team • Support Desk • Infrastructure monitoring 24x7 14 Global data centers • High Availability architecture • Auto recovery • Data recoveryWorld Class
- 26. Let’s Summarize Complexity, compliance Time, $, Resources Divide and Conquer (Test what you can) Define, prioritize platforms, test cases and data (Factors for IoT) Set the lab: Assets, global, scalable & compliant Build service virtualization Automate, Automate, Automate Audit- ready reporting
- 28. @Amir_Rozenberg
Editor's Notes
- #5: Investments being made Still massive opportunity to innovate
- #7: How they connect to the internet
- #10: TBD Add Google Acquisition healthcare
- #13: Shrink time: (ex: sensor sample every 30 minutes, test may take 7 days)
- #18: FDA challenges with certification of apps, versions etc.
- #19: FDA challenges with certification of apps, versions etc.
- #20: FDA challenges with certification of apps, versions etc.
- #23: FDA challenges with certification of apps, versions etc.
- #24: FDA challenges with certification of apps, versions etc.
- #25: FDA challenges with certification of apps, versions etc.
- #26: FDA challenges with certification of apps, versions etc.
- #27: FDA challenges with certification of apps, versions etc.
- #28: FDA challenges with certification of apps, versions etc.