Running deep learning onto heterogenous hardware
- 1. Running deep learning onto heterogenous hardware Vincent Delaitre CTO @ deepomatic
- 2. Some context • We build a platform allowing to easily train and deploy custom computer vision algorithms • Our customers are mostly large industrial groups • Many different use-cases, one methodology
- 3. « Lean AI » AIs deployment Image analysis Feedback loop Embedded devices (jetson, movidus, etc.) On premise & private cloud Training in the cloud
- 4. My shopping list for deployment Running locally Reliability constraints Bandwidth constraints Privacy constraints Various hardware requirements High vs Low throughput May need to be powered with 24V May need to be as cheap as possible State of the art Various meta-architectures: Caffe / TF / Darknet / ?Pytorch?
- 5. Deployment: frameworks VS hardware Caffe Tensorflow Darknet Pytorch GPU Jetson CPU Myriad FPGA
- 6. Deployment: frameworks VS hardware Caffe Tensorflow Darknet Pytorch GPU Jetson CPU Myriad FPGA
- 7. Solution: use runtimes ! CPU Myriad FPGA Intel OpenVino GPU Jetson Nvidia TensorRT
- 8. Solution: use runtimes ! CPU Myriad FPGA Intel OpenVino GPU Jetson Nvidia TensorRT ONNX
- 9. Solution: use runtimes ! Darknet Tensorflow Caffe Pytorch CPU Myriad FPGA Intel OpenVino GPU Jetson Nvidia TensorRT ONNX
- 10. Deployment logic Caffe models Tensorflow models Current architecture CPU and GPU worker
- 11. Deployment logic Caffe models Tensorflow models Current architecture CPU and GPU worker ONNX models Target architecture TensorRT worker OpenVino worker
- 12. Why does it matter ? Speedratiow/rVanillaTF 0 1 2 3 4 Vanilla TF (FP32) TensorRT (FP32) TensorRT (FP16) TensorRT (INT8)
- 13. Speed-up • From Vanilla to INT8: x3 speedup • Using 4 times less memory, use batch of 4: x3 speedup Total: x9 speedup
- 14. INT8, really ? http://on-demand.gputechconf.com/gtc/2017/presentation/s7310-8-bit-inference-with-tensorrt.pdf • Need a representative and diverse validation set • Increase error on accuracy of 0.4% in average (top-1)
- 15. Why does it matter ? 0 4 8 12 16 20 CPU Myriad 2 Myriad X Jetson TX2 Jetson Xavier GPU (Titan X) FPGA (Arria 10) Speed ratio w/r CPU Price
- 16. Flexibility • Low throughput: CPU • Moderate throughput or embedded: Myriad X or Jetson X • High throughput: GPU
- 17. What’s next ? Workflows ! Image Detector Crop Box 1 Label 1 Label 1 bisClassifier for « Label 1 » Crop Box N Label N Label N bisClassifier for « Label 1 »
- 18. What’s next ? Workflows ! Image Detector Crop Classifier for « Label 1 » Box 1 Label 1 Label 1 bisClassifier for « Label 1 » Classifier for « Label 1 » Fuse Crop Classifier for « Label 1 » Box N Label N Label N bisClassifier for « Label 1 » Classifier for « Label N » Fuse
- 19. What’s next ? Workflows ! Image Detector Crop Box K Label K Label K bis Classifier for « Label 1 » Classifier for « Label 1 » Classifier for « Label K » Fuse Jitter Jitter Jitter
- 20. What’s next ? Workflows ! Image Detector Crop Box K Label K Label K bis Classifier for « Label 1 » Classifier for « Label 1 » Classifier for « Label K » Fuse Jitter Jitter Jitter Tile Box K Label K Box K Label K Box K Label K Regroup
- 21. Facebook Tensor Comprehension C := Ax C := a.AB + bC https://arxiv.org/pdf/1802.04730.pdf
- 22. Facebook Tensor Comprehension https://arxiv.org/pdf/1802.04730.pdf Computation graph Intermediate representation Backend code (CUDA, cuDNN, etc…) • Graph optimisation • Operation scheduling • Operation placement • Just in time compilation + Autotuning • For training and inference
- 24. Merci ! PS: deepomatic recrute ! Et c’est par là: http://careers.deepomatic.com :)