Real-world Vision Systems Design: Challenges and Techniques
Download as PPTX, PDF3 likes1,376 views
The document discusses challenges and techniques in designing real-world vision systems, emphasizing the importance of handling specific project requirements, predicting data variations, and selecting appropriate compute platforms. It highlights the need for data collection and prototyping prior to hardware selection, while addressing maintenance issues and the significance of customer feedback. Additionally, it offers insights into the necessity for continuous updates to initial requirements and the inclusion of additional sensors to enhance vision algorithms.
Real-world Vision Systems Design: Challenges and Techniques
- 1. Copyright © 2016 Itseez 1 Real-World Vision Systems Design Challenges and Techniques Yury Gorbachev May 3, 2016
- 2. Copyright © 2016 Itseez 2 • Computer vision experts, 10 years on market • Maintainers of OpenCV library • Own products • ADAS algorithms suite • Facence (Face detection, recognition and analysis) • 3D Scanning (Itseez3D) • Accelerated CV (Optimized Computer Vision functions) • Consulting in CV area • Lot of Proof of Concept work per customer requests Itseez at Glance
- 3. Copyright © 2016 Itseez 3 • Specific handling of requirements for vision projects • Underestimated role of data • Compute platform selection fallacies • Hardware assisted vision algorithms • Planning for customer feedback • Functionality for easier product maintenance Topics to Cover
- 4. Copyright © 2016 Itseez 4 • Requirements for vision products require special handling • Not always possible to predict data variation upfront • Specific cases will appear during design or test • Missed scenarios • Lighting conditions, blur and glare effects, etc. • Impact can be pretty substantial • More scenarios to handle — increase in algorithm complexity • Additional data could be required Requirements for Vision Products
- 5. Copyright © 2016 Itseez 5 • Initial requirement — detect speed limits in Europe • Standards describe signs pretty well (size, color, etc.) • Pretty easy to put into requirements • Some things are not so straightforward to predict though Example — Speed Limit Sign Detection
- 6. Copyright © 2016 Itseez 6 • Computer vision heavily depends on data for training and test • Innovative products require specific data • Usually not available in public datasets • Data collection stage is needed prior to algorithm design • Collect videos/images, preferably with variation and target HW • Annotate collected data. At first stages done manually • This is not always well understood by non-vision related customers • Not always correctly planned within a project Good Dataset is a Major Part of Solution
- 7. Copyright © 2016 Itseez 7 • Statistics from road sign dataset used at Itseez ADAS • Not considering different sign types, weather and lighting conditions • Annotation will take even more Example — Speed Limit Signs Dataset
- 8. Copyright © 2016 Itseez 8 • Plenty of options for compute platform selection • GPU, DSP and SIMD-enabled SoCs • Frequently hardware selection is separated from software design • Could be even pre-selected already before project start • FACT: Not all compute platforms fit all algorithms • Separated selection usually results in suboptimal systems • Higher power consumption & unnecessary complicated algorithms Premature Compute Platform Selection
- 9. Copyright © 2016 Itseez 9 • Vectorization is commonly seen as a solution for most of the problems • Some SoCs sport few vector units with low frequency and few cores • Obvious choice, perfect fit for vision! • Not really… Following algorithm will have almost no benefit from SIMD Example — Suboptimal Hardware Selection Keypoint search External sensor Detailed search (Cascade) ROIs Candidates Candidate analysis
- 10. Copyright © 2016 Itseez 10 • Optics and sensors should be selected for a given task • Color vs. grayscale imagers, FOV vs. distance • Some tasks are significantly simpler when depth information is available • E.g., stairway detection for robotics • Vision algorithm can benefit from other sensor systems • IMU to consider motion and position (e.g., visual odometry) • LIDAR like cheap sensors in ADAS to limit search range • Consider cloud for heavy offline processing if no latency requirements Missed Opportunities for HW Assisted Vision
- 11. Copyright © 2016 Itseez 11 • Algorithm working on RGBD is much easier comparing to RGB • Consider following inputs Example — Staircase Detection RGBD image courtesy of: Reza Farid. Region growing planar segmentation for robot action planning
- 12. Copyright © 2016 Itseez 12 • Prototype algorithms prior to major work and HW selection • Understand bottlenecks and challenges update requirements • Perform HW selection considering algorithm specifics • Plan for sync between HW and SW during development Possible Vision Product Design Model Algorithm PoC design Prototype results Hardware approach selection Hardware Arch. Software drop Algorithm development Hardware drop HW Design Final product Original reqs Updated reqs
- 13. Copyright © 2016 Itseez 13 • Vision in products is still pretty new to consumers • Designers of the product create it with some use cases in mind • Consumers usually use them differently • This gap need attention prior to product release • Target group testing, people outside of CV area • Leave some time for corrective actions • Algorithm enhancements and changes • Documentation updates, tutorial videos, etc. Customer Acceptance Need Some Work
- 14. Copyright © 2016 Itseez 14 • Itseez3D application • Freely available on iPad, needs sensor (Occipital) • Just walk around a person and scan to get 3D model • Consumer use cases needed attention • Tracking from sensor SDK did not support scan interruption • Own enhanced tracking was implemented • UI enhancements and video tutorials were done • Currently about 300 scans per day Example — Full Body Scanning
- 15. Copyright © 2016 Itseez 15 • Most vision products require calibration • Re-calibration might be needed after some time • Surveillance cameras change position due to wind/shaking • Focus/visibility loss in optics due to harsh conditions • Product tampering • E.g., cover optics intentionally for some time periods • Always good to be able to detect those things in software • Blur & sharp edges detection, circular buffer recording Maintenance Issues in Vision Products
- 16. Copyright © 2016 Itseez 16 One Video Tells More Than Few Slides • Two problems here • Optics needs maintenance (algorithm works though) • Low sales figures...
- 17. Copyright © 2016 Itseez 17 • Do not consider requirements as final • Periodically update with findings, exceptions • For complicated algorithms perform PoC and iterate planning • Perform analysis of existing datasets plan for dataset collection • Investigate/prototype algorithm prior to hardware selection if possible • Consider additional sensors as help to vision algorithms • Ask for customer feedback prior product release • Place frame source checks in software to detect maintenance issues Conclusion: Useful Hints
- 18. Copyright © 2016 Itseez 18 Q&A
- 19. Copyright © 2016 Itseez 19 • Itseez Web — http://www.itseez.com • Itseez3D Scanning app — http://www.itseez3d.com • Contact me: yury.gorbachev@itseez.com Resources and Contacts