Learning where to look: focus and attention in deep vision
1 like1,151 views
This document summarizes Kevin McGuinness' presentation on deep learning for computer vision. It discusses visual attention models and their ability to predict eye gaze, applications in image cropping, retrieval and classification. It also covers medical image analysis using deep learning for knee osteoarthritis grading and neonatal brain segmentation. Deep crowd analysis is examined for crowd counting. Finally, interactive deep vision for image segmentation using user interactions is presented.
![How far are we from human-level accuracy?
Most errors are between grade 0 and 1 and grade 1 and 2.
Human experts have a hard time with these grades too.
Agreement among humans on OAI
● weighted kappa of 0.70 [0.65-0.76]
Human machine agreement
● weighted kappa of 0.67 [0.65-0.68]
Predictions agree with the “gold standard” about as well as
the “gold standard” agrees with itself.
Confusion matrix](https://image.slidesharecdn.com/dlmmdcud1s1spotlight-170427153808/85/Learning-where-to-look-focus-and-attention-in-deep-vision-37-320.jpg)
Learning where to look: focus and attention in deep vision
- 1. Kevin McGuinness kevin.mcguinness@dcu.ie Research Fellow Insight Centre for Data Analytics Dublin City University DEEP LEARNING WORKSHOP Dublin City University 28-29 April 2017 Learning where to look: focus and attention in deep vision
- 2. Overview Visual attention models and their applications Deep vision for medical image analysis Deep crowd analysis Interactive deep vision: image segmentation
- 3. Visual Attention Models and their Applications
- 4. The importance of visual attention
- 6. The importance of visual attention
- 7. The importance of visual attention
- 9. The importance of visual attention
- 10. Why don’t we see the changes? We don’t really see the whole image We only focus on small specific regions: the salient parts Human beings reliably attend to the same regions of images when shown
- 11. What we perceive
- 12. Where we look
- 13. What we actually see
- 14. Can we predict where humans will look? Yes! Computational models of visual saliency Why might this be useful?
- 15. SalNet: deep visual saliency model Predict map of visual attention from image pixels (find the parts of the image that stand out) ● Feedforward 8 layer “fully convolutional” architecture ● Transfer learning in bottom 3 layers from pretrained VGG-M model on ImageNet ● Trained on SALICON dataset (simulated crowdsourced attention dataset using mouse and artificial foveation) ● Top-5 in MIT 300 saliency benchmark http://saliency.mit.edu/results_mit300.html Predicted Ground truth Pan, McGuinness, et al. Shallow and Deep Convolutional Networks for Saliency Prediction, CVPR 2016 http://arxiv.org/abs/1603.00845
- 18. SalGAN Adversarial loss Data loss Junting Pan, Cristian Canton, Kevin McGuinness, Noel E. O’Connor, Jordi Torres, Elisa Sayrol and Xavier Giro-i-Nieto. “SalGAN: Visual Saliency Prediction with Generative Adversarial Networks.” arXiv. 2017. 18
- 19. SalNet and SalGAN benchmarks
- 20. Applications of visual attention Intelligent image cropping Image retrieval Improved image classification
- 23. Image retrieval: query by example Given: ● An example query image that illustrates the user's information need ● A very large dataset of images Task: ● Rank all images in the dataset according to how likely they are to fulfil the user's information need 23
- 24. Retrieval benchmarks Oxford Buildings 2007 Paris Buildings 2008 TRECVID INS 2014
- 25. Bags of convolutional features instance search Objective: rank images according to relevance to query image Local CNN features and BoW ● Pretrained VGG-16 network ● Features from conv-5 ● L2-norm, PCA, L2-norm ● K-means clustering -> BoW ● Cosine similarity ● Query augmentation, spatial reranking Scalable, fast, high-performance on Oxford 5K, Paris 6K and TRECVid INS BoW Descriptor Mohedano et al. Bags of Local Convolutional Features for Scalable Instance Search, ICMR 2016 http://arxiv.org/abs/1604.04653
- 26. Bags of convolutional features instance search BoW Descriptor Mohedano et al. Bags of Local Convolutional Features for Scalable Instance Search, ICMR 2016 http://arxiv.org/abs/1604.04653
- 27. Using saliency to improve retrieval CNN CNN Saliency Semantic features Importance weighting Weighted features Pooling (e.g. BoW) Image descriptors
- 28. Saliency weighted retrieval Oxford Paris INSTRE Global Local Global Local Global Local No weighting 0.614 0.680 0.621 0.720 0.304 0.472 Center prior 0.656 0.702 0.691 0.758 0.407 0.546 Saliency 0.680 0.717 0.716 0.770 0.514 0.617 QE saliency - 0.784 - 0.834 0.719 Mean Average Precision
- 29. 12.4% Using saliency to improve image classification Conv 1 Conv 3 Conv 4 Conv 5 FC 1 FC 1 FC 3 - Output Drop Out Drop Out Batch Norm. Max-Pooling Max-Pooling RGB Saliency Conv 1 Batch Norm. Max-Pooling Figure credit: Eric Arazo
- 30. Why does it improve classification accuracy? Acoustic guitar +25 % Volleyball +23 %
- 31. Deep Vision for Medical Image Analysis
- 32. Task: predict KL grade from X-Ray images
- 33. Pipeline: locate and classify
- 34. Detection performance FCN detection performance Template matching: (J > 0.5) 8.3% SVM on handcrafted features: (J > 0.5): 38.6%
- 35. Multi-objective learning helps! Same network used to regress on KL grade and predict a discrete KL grade Jointly train on both objectives
- 36. Comparison with the state of the art
- 37. How far are we from human-level accuracy? Most errors are between grade 0 and 1 and grade 1 and 2. Human experts have a hard time with these grades too. Agreement among humans on OAI ● weighted kappa of 0.70 [0.65-0.76] Human machine agreement ● weighted kappa of 0.67 [0.65-0.68] Predictions agree with the “gold standard” about as well as the “gold standard” agrees with itself. Confusion matrix
- 38. Neonatal brain image segmentation Volumetric semantic segmentation: label each pixel with class of brain matter. Applications: ● Prerequisite for volumetric analysis ● Early identification of risk factors for impaired brain development Challenge: ● Neonatal brains very different ● Sparse training data! Neobrains challenge has 2 training examples
- 39. Cerebellum Unmyelinated white matter Cortical grey matter Ventricles Cerebrospinal fluid Brainstem The task
- 40. Model ● 8 layer FCN ● 64/96 convolution filters per layer ● Atrous (dilated) convolution to increase receptive field without sacrificing prediction resolution ● 9D per pixel softmax over classes ● Binary cross entropy loss ● L2 regularization ● Aggressive data augmentation: scale, crop, rotate, flip, gamma ● Train on 2 axial volumes (~50 slides per volume) for 500 epochs using Adam optimizer Animation from: http://nicolovaligi.com/deep-learning-models-semantic-segmentation.html
- 41. Sample results Cerebellum Unmyelinated white matter Cortical grey matter Ventricles Cerebrospinal fluid Brainstem
- 42. Tissue Ours LRDE_LTCI UPF_SIMBioSy s Cerebellum 0.92 0.94 0.94 Myelinated white matter 0.51 0.06 0.54 Basal ganglia and thalami 0.91 0.91 0.93 Ventricles 0.89 0.87 0.83 Unmyelinated white matter 0.93 0.93 0.91 Brainstem 0.82 0.85 0.85 Cortical grey matter 0.88 0.87 0.85 Cerebrospinal fluid 0.83 0.83 0.79 UWM+MWM 0.93 0.93 0.90 CSF+Ven 0.84 0.84 0.79 0.85 0.80 0.83 Neobrains challenge New state of the art on Neobrains infant brain segmentation challenge for axial volume segmentation Deep learning with only 2 training examples! No ensembling yet. Best competing approach is a large ensemble. Second best is also a deep net.
- 43. Deep Vision for Crowd Analysis
- 44. Fully convolutional crowd counting FCN ∑ Crowd count estimate 8x conv layers 2x max pooling Crowd density estimate Marsden et al. Fully convolutional crowd counting on highly congested scenes. VISAPP 2017 https://arxiv.org/abs/1612.00220
- 45. True count: 1544 Predicted count: 1566
- 46. Benchmark results: UCF CC 50 dataset 45-45,000 people per image State of the art improved by 11% (MSE) and 13% (MSE)
- 49. Image Encoder FCN Interaction Encoder FCN Segmentation Decoder FCN Channel Concatenation Ground truth (MS COCO) Weak Labels Cross Entropy Loss Auto-generated User Outline Gaussian Process Input Image Crop from MS COCO Dilated Convolutions Pixel Predictions DeepClip: training
- 50. Image Encoder FCN Interaction Encoder FCN Segmentation Decoder FCN Channel Concatenation User interactions Input Image Dilated Convolutions Pixel Predictions DeepClip: prediction Binary Segmentation Vector tracer Graph Cuts Optimizer
- 52. Closing remarks ● Deep learning has completely revolutionized computer vision ● Human visual attention is important! Incorporating visual attention models helps in many tasks ● You don’t need a huge amount of training data to train an effective deep model ● Simulation techniques are effective for data generation ● Multi-task deep learning is an effective way of providing “more signal” during training.
- 53. Questions?