FPT17: An object detector based on multiscale sliding window search using a fully pipelined binarized CNN on an FPGA
2 likes1,501 views
1) The document describes an object detection system that uses a multiscale sliding window approach with fully pipelined binarized convolutional neural networks (BCNNs) implemented on an FPGA. 2) The system detects and classifies multiple objects in images by applying BCNNs to windows at different scales and locations, and suppresses overlapping detections. 3) Experimental results on a Zynq UltraScale+ MPSoC FPGA demonstrate that the proposed pipelined BCNN architecture can achieve higher accuracy than GPU-based detectors while using less than 5W of power.
![Optimization Techniques
• Binary CNN
• Multiple fully pipelined architecture
16
Batch normalization free
(BNF) [RAW17]
Internal FC layer replacement into a binary average
pooling [FPL17]
[FPL17] H. Nakahara, T. Fujii, S. Sato, ‘’A fully connected layer elimination for a binarized convolutional neural network
on an FPGA,’’ FPL 2017, pp. 1‐4.
[RAW17] H. Yonekawa and H. Nakahara, ‘’On‐chip memory based binarized convolutional deep neural network
applying batch normalization free technique on an FPGA,’’ IPDPS Workshops 2017, pp. 98‐105.](https://image.slidesharecdn.com/fpt17slidingwindowv1-180225213227/85/FPT17-An-object-detector-based-on-multiscale-sliding-window-search-using-a-fully-pipelined-binarized-CNN-on-an-FPGA-16-320.jpg)
FPT17: An object detector based on multiscale sliding window search using a fully pipelined binarized CNN on an FPGA
- 2. Outline • Background • Object detector algorithm • Fully pipelined Binarized CNN • Experimental results • Conclusion 2
- 4. Convolutional Neural Network (CNN) • Convolutional + fully connected + pooling layers • State‐of‐the‐art performance in an image recognition task • Widely applicable 4Source: https://www.mathworks.com/discovery/convolutional‐neural‐network.html
- 5. Image Recognition Tasks • Classification • answer “category” of the object in an image • Object Detection • classification + localization • Semantic Segmentation • Object area in pixel level 5 Easy Hard Children
- 6. Requirements in Embedded System 6 Cloud Embedded Many classes (1000s) Few classes (<10) Large workloads Frame rates (15‐30 FPS) High efficiency (Performance/W) Low cost & low power (1W‐5W) Server form factor Custom form factor J. Freeman (Intel), “FPGA Acceleration in the era of high level design”, 2017
- 7. Outline • Background • Object detector algorithm • Fully pipelined Binarized CNN • Experimental results • Conclusion 7
- 8. Object Detection Problem • Detecting and classifying multiple objects at the same time • Evaluation criteria (from Pascal VOC): 8 Ground truth annotation Detection results: >50% overlap of bounding box with ground truth One BBox for each object Confidence value for each object Person (50%) # . # . # . # 1 11 , ∈ ,. ,…, Average Precision (AP):
- 10. Sliding Window • It is rectangular region of fixed width and height that “slides” across an image 10
- 11. Multi‐Scaling (Pyramid Pooling) • Find objects in images at different scales • Combined with a sliding window, it can find objects in various locations with the same window size 11
- 13. Quantification of Iterations 13 q q q q qq... 1st image 2nd image i-th image ∆ 2 • Trade‐off: Time (Iters), AP, and HW p: Image size (given) q: Window size Δx: Stride → Find good q and Δx
- 14. Outline • Background • Object detector algorithm • Fully pipelined Binarized CNN • Experimental results • Conclusion 14
- 15. Binarized CNN 15 x1 w0 (Bias) fsgn(Y) Y z w1 x2 w2 xn wn ... x1 x2 Y ‐1 ‐1 1 ‐1 +1 ‐1 +1 ‐1 ‐1 +1 +1 1 x1 x2 Y 0 0 1 0 1 0 1 0 0 1 1 1
- 16. Optimization Techniques • Binary CNN • Multiple fully pipelined architecture 16 Batch normalization free (BNF) [RAW17] Internal FC layer replacement into a binary average pooling [FPL17] [FPL17] H. Nakahara, T. Fujii, S. Sato, ‘’A fully connected layer elimination for a binarized convolutional neural network on an FPGA,’’ FPL 2017, pp. 1‐4. [RAW17] H. Yonekawa and H. Nakahara, ‘’On‐chip memory based binarized convolutional deep neural network applying batch normalization free technique on an FPGA,’’ IPDPS Workshops 2017, pp. 98‐105.
- 20. Pipelined Conv2D Circuit x00 x01 x02 x03 x04 x10 x11 x12 x13 x14 x20 x21 x22 x23 x24 x30 x31 x32 x33 x34 x40 x41 x42 x43 x44 x22 x21 x20 x14 x13 x12 x11 x10 x04 x03 x02 x01 x00 + Binarized Weight Mem. Integer Bias Mem. Write Ctrl. Logic Counter Binarized Feature Maps (L=5, K=3) 9 Binarized MACs (EXNORs + Adder Tree) Sign bit Shift Register (2L+K bits) Read M F.Maps at a time
- 21. Used CNN Model 21 Integer Conv2D Binary Conv2D Max Pooling Binary Conv2D Binary Conv2D Binary Conv2D Max Pooling Binary Conv2D Binary Conv2D Binary Conv2D Max Pooling Fully Connect Fully Connect Fully Connect Integer Conv2D Binary Conv2D Max Pooling Binary Conv2D Binary Conv2D Binary Conv2D Max Pooling Binary Conv2D Binary Conv2D Binary Conv2D Average Pooling Fully Connect VGG11 Our VGG • Based on the VGG11 model • 3x3 kernel convolution • Replacement bottleneck (memory intensive) layers into an average pooling one
- 22. Overall Architecture • Weight sharing 22 Pipelined BCNN 1 FIFO Pipelined BCNN 2 FIFO Pipelined BCNN P FIFO AXI4 Bus ... GPIO ARM Processor DDR Mem Camera Input Image ... Weight Mem
- 23. Outline • Background • Object detector algorithm • Fully pipelined Binarized CNN • Experimental results • Conclusion 23
- 24. Implementation Setup • Board: Xilinx Inc. Zynq UltraScale+ MPSoC zcu102 evaluation board • Zynq UltraScale+ MPSoC FPGA (ZU9EG, 68,250 slices, 269,200 FFs, 1,824 BRAMs, 2,520 DSP48Es) • FPGA design tool: Vivado HLS 2017.2 and Vivado 2017.2 • Timing constraint: 200MHz • Deep learning framework: Chainer 1.24.0 • Dataset: KITTI car detection (moderate) scenario 24
- 25. Variation of Fully Pipelined CNNs 25 CNN Parameter Hardware Resource Accuracy Speed Window Size q Stride ∆X #18Kb BRAMs #FFs #LUTs #DSPs mAP FPS 96x96 24 240 194,930 114,870 0 74.36 11.10 48 71.29 45.30 64x64 16 232 189,820 169,500 0 84.80 8.70 32 82.20 34.95 48x48 12 232 171,850 172,100 0 70.50 7.80 24 64.20 31.65 32x32 8 232 169,930 178,220 0 56.32 8.55 16 52.30 34.20
- 26. Comparison with GPU based Detectors 26 0 10 20 30 40 50 60 70 80 90 100 0.01 0.1 1 10 100 mAP(%) Detection Speed (FPS) Ours 29.97 YOLOv2 MV3D(LIDAR) SPD+RPN Deep MANTA RRC FPS Acc (%) RRC 0.27 90.22 Deep MANTA 1.42 90.03 SDP+RPN 2.50 89.90 MV3D (LIDAR) 4.16 79.76 YOLOv2 50.00 28.37 Proposed 34.95 82.20 YOLOv2 (GPU): 250.0 W Proposed(FPGA): 2.5 W
- 27. Conclusion • Applied a pipelined binary CNN to an object detector • Multiple pipeline architecture • Weight sharing • Find good parameters for the KITTI car detection • Better performance and accuracy than GPUs • Future works • Preprocessing to reduce HW and Time • Selective search, BING, etc.. • Post‐processing to adjust bounding boxes • SVR, another CNN, etc.. 27