“Image Signal Processing Optimization for Object Detection,” a Presentation from Nextchip
0 likes65 views
Nextchip focuses on image signal processing optimization aimed at enhancing object detection capabilities by tuning image quality for both human and machine vision. The document outlines the challenges in image tuning, discusses various methods and metrics for improving detection accuracy, and presents experimental results highlighting the impact of different image settings. Future work is suggested to further analyze optimal tuning trends to enhance performance.
“Image Signal Processing Optimization for Object Detection,” a Presentation from Nextchip
- 1. Young-Jun Yoo EVP NEXTCHIP CO., LTD. Image Signal Processing Optimization for Object Detection
- 2. NEXTCHIP Overview • Developing & optimizing vision core; Image signal processing technology for 27 years • Tuning know-how with various MP models with global OEMs and Tiers • Tuning capability for human vision & machine mision • Open architecture with various image sensors, CFAs (color filter arrays) World-class ISP In-house Core Automotive Reliability ASIC Design Technology • Automotive process foundry experience; 14nm/28nm/55nm/60nm/95nm Samsung/Global Foundries/USJC/TSMC • ISO26262; Functional safety • Cyber security • CMMI Lv.-3 • A-Spice process • AEC-Q100 Gr.2 lineup © 2024 NEXTCHIP 2
- 3. 3 © 2024 NEXTCHIP Image Signal Processing Optimization for Object Detection Chapter 1: What is the Difference? Human Vision vs. Machine Vision Chapter 2: The Image Tuning Challenges for Human Vision Chapter 3: The Image Tuning Challenges for Machine Vision
- 4. 4 Image Signal Processing Optimization for Object Detection © 2024 NEXTCHIP Chapter 1: What is the Difference? Human Vision vs. Machine Vision Chapter 2: The Image Tuning Challenges for Human Vision Chapter 3: The Image Tuning Challenges for Machine Vision
- 5. Human Vision vs. Machine Vision 5 © 2024 NEXTCHIP • We asked this question to ChatGPT… It gave this image as an answer! Do you feel the same way?
- 6. Image Tuning Needed for Both Types of Vision 6 ISP Image Sensor Display Lens The key point of image sensor Delivery of Robust “raw (bayer) data” ① Sensitivity (Pixel technology) ② Color (CFA-color filter array) The key point of ISP Processing “image signal data” ① Reproduction signal to vision ② Color/less noise • What is image tuning? Why is it needed? © 2024 NEXTCHIP
- 7. 7 Image Signal Processing Optimization for Object Detection © 2024 NEXTCHIP Chapter 1: What is the Difference? Human Vision vs. Machine Vision Chapter 2: The Image Tuning Challenges for Human Vision Chapter 3: The Image Tuning Challenges for Machine Vision
- 8. Image Tuning Challenges for Human Vision 8 What is the challenge? Make the image as similar as possible to one seen through a human eye What is the key factor to tune for human vision? • Color reproduction • Lower noise level • Brightness/edge/HDR (high dynamic range), etc. Tuning under various environment, e.g., day & night © 2024 NEXTCHIP
- 9. Image Tuning Challenges for Human Vision 9 Quantitative TEST Qualitative TEST It presents you with numerical value. E.g., Δ-E, HDR dB, AE (auto exposure) speed, etc. It determines the user’s motivation, comments, feeling, etc. throughout the test process. E.g., color balance and bright in a sight Color Accuracy Field Test for Repeat Indoor & Outdoor Noise/Edge Adjustment How to do? © 2024 NEXTCHIP
- 10. Image Tuning Challenges for Human Vision 10 02 Weakn ess Day Night Day Result ① Tuning ② Test ③ Tuning ④ Repeat Test Night Result How to do? © 2024 NEXTCHIP
- 11. Image Tuning Challenges for Human Vision 11 ▶ Problem • Generally dark • Too strong color • Too strong edge level © 2024 NEXTCHIP
- 12. Image Tuning Challenges for Human Vision 12 ▶ Tuning#1 • Brightness • HDR & Contrast • GCE (global contrast enhancement) © 2024 NEXTCHIP
- 13. Image Tuning Challenges for Human Vision 13 © 2024 NEXTCHIP ▶ Tuning#2 • Color (hue, saturation) • Color suppression
- 14. Image Tuning Challenges for Human Vision 14 ▶ Problem ▶ Final tuned image © 2024 NEXTCHIP
- 15. Image Tuning Challenges for Human Vision 15 ▶ Problem • Generally dark • Too strong color • Too strong edge level © 2024 NEXTCHIP
- 16. Image Tuning Challenges for Human Vision 16 ▶ Tuning#1 • Brightness • HDR & Contrast • GCE (global contrast enhancement) © 2024 NEXTCHIP
- 17. Image Tuning Challenges for Human Vision 17 ▶ Tuning#2 • Color (hue, saturation) • Color suppress © 2024 NEXTCHIP
- 18. Image Tuning Challenges for Human Vision 18 ▶ Problem ▶ Final tuned image © 2024 NEXTCHIP
- 19. 19 Image Signal Processing Optimization for Object Detection © 2024 NEXTCHIP Chapter 1: What is the Difference? Human Vision vs. Machine Vision Chapter 2: The Image Tuning Challenges for Human Vision Chapter 3: The Image Tuning Challenges for Machine Vision
- 20. Image Tuning Challenges for Machine Vision 20 What is the challenge? • Higher detection rate is needed Methods to increase detection rate such as: • Retraining • Changing training method • Changing field of view and resolution • Image tuning, etc. © 2024 NEXTCHIP
- 21. Measure Factors for Test 21 Detection network • YOLOv5s Datasets • Location : Pangyo, Korea • Scene : Sunny, daytime & nighttime, rearview fisheye 190° • Training image resolution : 640x360 / training images : 12,732 © 2024 NEXTCHIP
- 22. Test Dataset & Tuning 22 Test Dataset • Quantitative experiments: Stationary object + Ground Truth • Qualitative experiments: Driving scene ISP Tuning • Brightness level: Auto exposure (AE) • Edge sharpness level: Edge enhancement (EDGE) • Noise level: Noise reduction (NR) © 2024 NEXTCHIP
- 23. Quantitative Experiments – Metric 23 Metrics of best ISP for object detection High Detection Accuracy Best ISP High Confidence Score Precision Consistency of Detection Results Frame(t-1) Frame(t) Detection Rate Object Score False Positive © 2024 NEXTCHIP
- 24. Quantitative Experiments – ISP Tuning & Test Dataset 24 T1 Viewing Optimization Tuning T2 All ISP OFF except color related T4 AE Down, NR Up T3 AE Up, EDGE Up • 4 different ISP settings for the same scene • About 3200 frames for each tuning point Test dataset ISP tuning • Brightness level: Auto exposure (AE) • Edge sharpness level: Edge enhancement (EDGE) • Noise level: Noise reduction (NR) © 2024 NEXTCHIP
- 25. Quantitative Experiments – Detection Accuracy 25 • An indicator of recognition accuracy for each tuning point High detection accuracy © 2024 NEXTCHIP
- 26. Quantitative Experiments – Confidence Score 26 • A score which represents likelihood that the bounding box contains an object High confidence score © 2024 NEXTCHIP
- 27. Quantitative Experiments – Detection Consistency 27 • An indicator of whether the same object is consistently recognized High consistency score © 2024 NEXTCHIP consistency
- 28. Quantitative Experiments – Precision 28 • An indicator of recognition precision Precision © 2024 NEXTCHIP
- 29. Quantitative Experiments – Result 29 • For all metrics, the higher the better Total evaluation results Best Second Best © 2024 NEXTCHIP
- 30. • EDGE has the greatest impact on detection performance 1. Too many EDGE Worse detection performance 2. More EDGE More false detections • Darker image Reduced false detection rate and accuracy • Need to fine the best ISP setting value between T2 and T4 Quantitative Experiments - Conclusion T2 Result T4 Result Best ISP Point © 2024 NEXTCHIP 30
- 31. T1 : Original Setting T5 : Edge Sharpness Off + Bright Up T6 : Edge Sharpness Off + Bright Up + NR Up T8 : Edge Sharpness Off + Bright Down + NR Up T7 : Edge Sharpness Off + Bright Down Qualitative Experiments – Evaluation Methods Evaluation Methods • Estimate the false detection rate • Counting false positives (FP) for period in which false detection occurs in all tuning points © 2024 NEXTCHIP 31
- 32. T1 : Original Setting T5 : Edge Sharpness Off + Bright Up T6 : Edge Sharpness Off + Bright Up + NR Up T8 : Edge Sharpness Off + Bright Down + NR Up T7 : Edge Sharpness Off + Bright Down • Additional 5 ISP settings for the same driving path Daytime test Qualitative Experiments – Best ISP for Object Detection © 2024 NEXTCHIP 32
- 33. Daytime evaluation result Qualitative Experiments – Best ISP for Object Detection T8 : EDGE sharpness OFF + AE Down + NR Up Best © 2024 NEXTCHIP 33
- 34. T1 : Viewing Optimization Tuning T8: Edge Sharpness Off + AE Down + NR Up Nighttime test • 2 ISP settings are applied for same driving path Qualitative Experiments – Best ISP for Object Detection © 2024 NEXTCHIP 34
- 35. Nighttime evaluation result Sensing Optimization Best Qualitative Experiments – Best ISP for Object Detection © 2024 NEXTCHIP 35
- 36. Quantitative Experiments – Conclusion • Qualitatively, the detection rates are similar at all tuning points • Datasets1 (Day time) 1. When noise level is high, reduces false detection rate 2. In daytime, brightness does not seem to have a significant effect on false detection • Datasets2 (Night time) 1. T8 (Sensing) false detection rate is 0.1 better than T1 (viewing tuning) 2. At nighttime, when brightness level is low, reduced false detection rate © 2024 NEXTCHIP 36
- 37. Future Works The problem with current experiments • Since the performance is evaluated only for specific points, there are some limitations to estimate the tendency value for each tuning factor. Further experiments • We keep working to analyze the trends while changing the AE (brightness), EDGE, and the noise level in optimal ISP tuning. © 2024 NEXTCHIP 37
- 38. Resources • ChatGPT https://chatgpt.com/n • Test by Nextchip Internal Standard of Image Quantitative & Qualitative Test 2024 Embedded Vision Summit ● Booth#109 ● Mr. Young-Jun Yoo ● gisado76@nextchip.com © 2024 NEXTCHIP 38