Random_Stability_Testing
0 likes197 views
- Random stability testing involves randomly switching between applications on a device for an extended period, like 48 hours, to identify any failures. This helps resolve the problem of not having a solution for random stability testing. - A model-based solution is used where actions, triggers, results and exceptions are modeled to implement, execute, and report on the random testing. Event recording and playback are also used to help create tests quickly for third party applications. - Bugs were found during testing, including app crashes and screen flashing, which provides feedback to improve the system. The testing can also be expanded to control multiple devices and account for different inputs like sensors.
Random_Stability_Testing
- 1. Random Stability Testing Yu, Yucai 2013/11/05
- 2. 2 Agenda • Why we do this? • What problem we want to resolve? • Model based solution • Demo • Deep dive
- 3. 3 Why we do this? - Request from Toyota, part of IVI system testing “Continue using for an extended period (48 hours)” - No available solution for random stability test - Leverage fMBT and iGAS - Enhancement for flexibility, usability, portability
- 4. 4 What problem we should resolve? - Do what user usually do, to cover kinds of activities - Randomly switch among applications with some probability - Identify Pass/Fail, handle failures - Easy for issue analysis with context, get trend and statistics - Portability for different platforms
- 5. 5 Model based solution Action Trigger Result Analysis Exception Handling Modeling Implementing Executing Reporting
- 6. 6 Feedbacks We found bugs: • TIVI-1965 PA crashes after long time operations • TIVI-1819 Web apps process still alive after launch/close many times • TIVI-1864 weston will crash after random and continuous operation • TIVI-1996 Click app has no response after switching app for many times • TIVI-1988 The screen of all app list flashes sometimes after system boot up Feedback from Jaska(murphy key developer): • This is excellent for policy use case testing and when to open?
- 7. 7 We can do more… Multi device control: • Mobiles(Tizen, android) call each other • Tizen IVI HFP call • Dual OS
- 8. 8 Demo
- 9. 9 Deep dive • Modeling • Implementation • Image recognition with iGAS • Deal with failure
- 10. 10 Modeling Using fMBT to create the model and drive test execution
- 11. 11
- 12. 12 Implement test steps • Method 1: invoke external command(middleware scripts, iGAS etc.) • Method 2: programming to gain great flexibility
- 13. 13 Programming test steps fMBT’s device class library: • Support several platforms: Tizen, android, windows, general Linux. • Support function: Screenshot, tapBitmap(tapItem), verifyBitmap, Shell etc. Our compliment: event record and playback
- 14. 14 Event record and playback Two benefit: 1. Solve some scenario that gfx recognition way not work well. • Complex gesture • Terminal operation • App theme Change • Record children style’s random operation • Deal with sensors 2. Help create tests quickly(third party apps etc.)
- 15. 15 Image recognition with iGAS • fMBT’s graphics engine is effective, but not very “smart”. • Improve with iGAS’s smart image recognition engine: SIFT, pHASH etc. • Improve with iGAS’s camera solution.
- 16. 16 How to deal with failure Collect visual action logs Collect app logs and system status Monitor system resource: CPU, memory, process etc. Collect crashes Reboot and restart testing
- 17. Thanks