![Descriptors - history
• Normalized cross-correlation (NCC) [~ 60s]
• Gaussian derivative-based descriptors
– Differential invariants [Koenderink and van Doorn’87]
– Steerable filters [Freeman and Adelson’91]
• Moment invariants [Van Gool et al.’96]
• SIFT [Lowe’99]
• Shape context [Belongie et al.’02]
• Gradient PCA [Ke and Sukthankar’04]
• SURF descriptor [Bay et al.’08]
• DAISY descriptor [Tola et al.’08, Windler et al’09]
• …….](https://image.slidesharecdn.com/modern-features-part-2-descriptors-121214032117-phpapp01/85/Modern-features-part-2-descriptors-4-320.jpg)
![SIFT descriptor [Lowe’99]
• Spatial binning and binning of the gradient orientation
• 4x4 spatial grid, 8 orientations of the gradient, dim 128
• Soft-assignment to spatial bins
• Normalization of the descriptor to norm one (robust to illumination)
• Comparison with Euclidean distance
image patch gradient x
→ →
y](https://image.slidesharecdn.com/modern-features-part-2-descriptors-121214032117-phpapp01/85/Modern-features-part-2-descriptors-5-320.jpg)
![Evaluation
• Descriptors should be
– Distinctive (! importance of the measurement region)
– Robust to changes on viewing conditions as well as to errors of
the detector
– Compactness, speed of computation
• Detection rate (recall) 1
– #correct matches / #correspondences
• False positive rate
– #false matches / #all matches
• Variation of the distance threshold
– distance (d1, d2) < threshold
1
[K. Mikolajczyk & C. Schmid, PAMI’05]](https://image.slidesharecdn.com/modern-features-part-2-descriptors-121214032117-phpapp01/85/Modern-features-part-2-descriptors-7-320.jpg)
![Recent extensions to SIFT
• Color SIFT [Sande et al. 2010]
• Normalizing SIFT with square root transformation
[Arandjelovic et Zisserman’12]](https://image.slidesharecdn.com/modern-features-part-2-descriptors-121214032117-phpapp01/85/Modern-features-part-2-descriptors-11-320.jpg)
Modern features-part-2-descriptors
- 1. Descriptors Cordelia Schmid Tinne Tuytelaars Krystian Mikolajczyk
- 2. Overview – descriptors • Introduction • Modern descriptors • Comparison and evaluation
- 3. Descriptors Eliminate rotational Compute appearance Extract affine regions Normalize regions descriptors + illumination SIFT (Lowe ’04)
- 4. Descriptors - history • Normalized cross-correlation (NCC) [~ 60s] • Gaussian derivative-based descriptors – Differential invariants [Koenderink and van Doorn’87] – Steerable filters [Freeman and Adelson’91] • Moment invariants [Van Gool et al.’96] • SIFT [Lowe’99] • Shape context [Belongie et al.’02] • Gradient PCA [Ke and Sukthankar’04] • SURF descriptor [Bay et al.’08] • DAISY descriptor [Tola et al.’08, Windler et al’09] • …….
- 5. SIFT descriptor [Lowe’99] • Spatial binning and binning of the gradient orientation • 4x4 spatial grid, 8 orientations of the gradient, dim 128 • Soft-assignment to spatial bins • Normalization of the descriptor to norm one (robust to illumination) • Comparison with Euclidean distance image patch gradient x → → y
- 6. Local descriptors - rotation invariance • Estimation of the dominant orientation – extract gradient orientation – histogram over gradient orientation – peak in this histogram 0 2π • Rotate patch in dominant direction The image cannot be displayed. Your computer may not have enough memory to open the image, or the image may have been corrupted. Restart your computer, and then open the file again. If the red x still appears, you may have to delete the image and then insert it again. • Plus: invariance • Minus: less discriminant, additional noise
- 7. Evaluation • Descriptors should be – Distinctive (! importance of the measurement region) – Robust to changes on viewing conditions as well as to errors of the detector – Compactness, speed of computation • Detection rate (recall) 1 – #correct matches / #correspondences • False positive rate – #false matches / #all matches • Variation of the distance threshold – distance (d1, d2) < threshold 1 [K. Mikolajczyk & C. Schmid, PAMI’05]
- 8. Viewpoint change (60 degrees) Detector: * * Hessian-Affine
- 9. Scale change (factor 2.8) Detector: * * Hessian-Affine
- 10. Evaluation - conclusion • SIFT based descriptors perform best • Significant difference between SIFT and low dimension descriptors as well as cross-correlation • Robust region descriptors better than point-wise descriptors • Performance of the descriptor is relatively independent of the detector
- 11. Recent extensions to SIFT • Color SIFT [Sande et al. 2010] • Normalizing SIFT with square root transformation [Arandjelovic et Zisserman’12]
- 12. Descriptors – dense extraction • Many conclusions for descriptors applied to sparse detectors also hold for dense extraction – For example normalizing SIFT with the square root improves in image retrieval and classification • Image retrieval: sparse versus dense SIFT + Fisher vector for retrieval on the Holidays dataset Hessian-Affine MAP = 0.54 Dense MAP = 0.62
- 13. Overview – descriptors • Introduction • Modern descriptors • Comparison and evaluation
- 14. Overview • Introduction • Modern descriptors • Comparison and evaluation
- 15. Modern descriptors • Efficient descriptors • Compact binary descriptors • More robust descriptors • Learned descriptors
- 16. DAISY Cita%ons: • Op<mized for dense sampling 150 (2012) • Log-‐polar grid • Gaussian smoothing • Dealing with occlusions Engin Tola, Vincent Lepe<t, and Pascal Fua, DAISY: An Efficient Dense Descriptor Applied to Wide-‐Baseline Stereo, TPAMI 32(5), 2010.
- 17. SURF: Speeded Up Robust Features • Approximate deriva<ves with Haar wavelets • Exploit integral images Cita%ons: 4500 (2012) Herbert Bay, Andreas Ess, Tinne Tuytelaars, Luc Van Gool "SURF: Speeded Up Robust Features", Computer Vision and Image Understanding (CVIU), Vol. 110, No. 3, pp. 346-‐-‐359, 2008
- 18. SURF: Speeded Up Robust Features • Orienta<on assignment
- 19. U-‐SURF: Upright SURF • Rota<on invariance is o`en not needed. Don’t use more invariance than needed for a given applica<on • Orienta<on es<ma<on takes <me. • Orienta<on es<ma<on is o`en a source of errors.
- 20. Beyond the classics • Efficient descriptors • Compact binary descriptors • More robust descriptors • Learned descriptors
- 21. Fast and compact descriptors • (Very) large scale applica<ons – > memory issues, computa<on <me issues • Mobile phone applica<ons
- 22. Fast and compact descriptors • Binary descriptors • Comparison of pairs of intensity values -‐ LBP -‐ BRIEF -‐ ORB -‐ BRISK
- 23. LBP: Local Binary Paeerns • First proposed for texture recogni<on in 1994. Cita%ons: 2500 (2012) T. Ojala, M. Pie<käinen, and D. Harwood (1994), "Performance evalua<on of texture measures with classifica<on based on Kullback discrimina<on of distribu<ons", ICPR 1994, pp.582-‐585. M Heikkilä, M Pie<käinen, C Schmid, Descrip<on of interest regions with LBP, Paeern recogni<on 42 (3), 425-‐436
- 24. BRIEF: Binary Robust Independent Elementary Features • Random selec<on of pairs of intensity values. • Fixed sampling paeern of 128, 256 or 512 pairs. • Hamming distance to compare descriptors (XOR). Cita%ons: 149 (2012) M. Calonder, V. Lepe<t, C. Strecha, P. Fua, BRIEF: Binary Robust Independent Elementary Features, 11th European Conference on Computer Vision, 2010.
- 25. D-‐BRIEF: Discrimina<ve BRIEF • Learn linear projec<ons that map image patches to a more discrimina<ve subspace • Exploit integral images T. Trzcinski and V. Lepe<t, Efficient Discrimina<ve Projec<ons for Compact Binary Descriptors European Conference on Computer Vision (ECCV) 2012
- 26. ORB: Oriented FAST and Rotated BRIEF • Add rota<on invariance to BRIEF Cita%ons: 43 (2012) • Orienta<on assignment based on the intensity centroid Ethan Rublee, Vincent Rabaud, Kurt Konolige, Gary Bradski, ORB: an efficient alterna<ve to SIFT or SURF, ICCV 2011
- 27. ORB: Oriented FAST and Rotated BRIEF • Select a good set of pairwise comparisons: minimize correla<on under various orienta<on changes High variance High variance + uncorrelated
- 28. BRISK: Binary Robust Invariant Scalable Keypoints Cita%ons: • Regular grid 20 (2012) • Orienta<on assignment based on dominant gradient direc<on (using long-‐distance pairs) • 512 bit descriptor based on short-‐distance pairs Stefan Leutenegger, Margarita Chli and Roland Y. Siegwart, BRISK: Binary Robust Invariant Scalable Keypoints, ICCV 2011
- 29. Various others • FREAK: Fast Re<na Keypoint • CARD: Compact and Real<me Descriptor • LDB: Local Difference Binary
- 30. FREAK: Fast Re<na Keypoint • Inspired by the human visual system A. Alahi, R. Or<z, and P. Vandergheynst. FREAK: Fast Re<na Keypoint. In IEEE Conference on Computer Vision and Paeern Recogni<on 2012.
- 31. CARD: Compact and Real<me Descriptor – Look up tables – learning-‐based sparse hashing
- 32. LDB: Local Difference Binary Xin Yang and Kwang-‐Ting Cheng, LDB: An Ultra-‐Fast Feature for Scalable Augmened Reality on Mobile Devices, Interna<onal Symposium on Mixed and Augmented Reality 2012 (ISMAR 2012)
- 33. Beyond the classics • Efficient descriptors • Compact binary descriptors • More robust descriptors • Learned descriptors
- 34. LIOP: Local Intensity Order Paeern for Feature Descrip<on (and predecessors MROGH and MRRID) • Robustness to monotonic intensity changes • Data-‐driven division into cells Zhenhua Wang Bin Fan Fuchao Wu, Local Intensity Order Paeern for Feature Descrip<on, ICCV 2011.
- 35. LIOP: Local Intensity Order Paeern for Feature Descrip<on
- 36. Beyond the classics • Efficient descriptors • Compact binary descriptors • More robust descriptors • Learned descriptors
- 37. Winder & Brown • Learn configura<on and other parameters from training data obtained from 3D Cita%ons: reconstruc<ons 194 (2012) M. Brown, G. Hua and S. Winder, Discriminant Learning of Local Image Descriptors.. IEEE Transac<ons on Paeern Analysis and Machine Intelligence. 2010.
- 38. Winder & Brown • Training data = set of corresponding image patches
- 39. Descriptor Learning Using Convex Op<misa<on • Convex learning of Pre-‐rec<fied keypoint patch – spa<al pooling regions – dimensionality reduc<on Non-‐linear transform • Learning from very weak supervision learning Spa<al pooling Normalisa<on and cropping Dimensionality learning reduc<on Descriptor vector K. Simonyan et al., Descriptor Learning Using Convex Op<misa<on, ECCV 2012
- 40. Learning Spa<al Pooling Regions • Selec%on from a large pool using L1 regularisa%on • So`-‐margin constraints: – squared L2 distance between descriptors of matching feature pairs should be smaller than that of non-‐matching pairs • Convex objec<ve (op<mised with a proximal method) 768-‐D 576-‐D 320-‐D pooling region configura%ons learnt with different levels of sparsity
- 41. Overview • Introduction • Modern descriptors • Comparison and evaluation
- 42. Sepng up an evalua<on • Which problem? Performance in different applica<on/niches may vary significantly. – Category recogni<on, – Matching, – Retrieval • What dataset? – Pascal VOC 2007 – Oxford image pairs – Oxford -‐ Paris buildings • Protocol and criteria? – Public dataset, – Avoiding risk to over-‐fipng/op<mizing to the data
- 43. Detector evalua<ons homography A B B Two points are correctly matched if # correct matches precision = T=40% # all matches A∩ B >T A∪ B # correct matches recall = # ground truth correspondences
- 44. Previous Evalua<ons • 2D Scene – Homography – C. Schmid, R. Mohr, and C. Bauckhage, “Evalua<on of interest point detectors,” IJCV, 2000. – K. Mikolajczyk and C. Schmid, “A performance evalua<on of local descriptors,” CVPR, 2003. – T. Kadir, M. Brady, and A. Zisserman, “An affine invariant method for selec<ng salient regions in images,” in ECCV, 2004. – K. Mikolajczyk, T. Tuytelaars, C. Schmid, A. Zisserman, J. Matas, F. Schaffalitzky,T. Kadir, and L. Van Gool, “A comparison of affine region detectors,” IJCV, 2005. – A. Haja, S. Abraham, and B. Jahne, Localiza<on accuracy of region detectors, CVPR 2008 – T. Dickscheid, FSchindler, Falko, W. Förstner, Coding Images with Local Features, IJCV 2011
- 45. Previous Evalua<ons • 3D Scene -‐ epipolar constraints – F. Fraundorfer and H. Bischof, “Evalua<on of local detectors on non-‐planar, scenes,” in AAPR, 2004. – P. Moreels and P. Perona, “Evalua<on of features detectors and descriptors based on 3D objects,” IJCV, 2007. – S. Winder and M. Brown, “Learning local image descriptors,” CVPR, 2007,2009. – Dahl, A.L., Aanæs, H. and Pedersen, K.S. (2011): Finding the Best Feature Detector-‐Descriptor Combina<on. 3DIMPVT, 2011.
- 46. Recent Evalua<ons • Recent detectors O. Miksik and K. Mikolajczyk, Evalua<on of Local Detectors and Descriptors for Fast Feature Matching, ICPR 2012 ECCV 2012 Modern features: 46/60 … Detectors.
- 47. Recent detector evalua<ons • Completeness (coverage), complementarity between detectors T. Dickscheid, FSchindler, Falko, W. Förstner, Coding Images with Local Features, IJCV 2011 – Completeness, Edgelap (Mikolajczyk), Salient (Kadir), MSER (Matas) – Complementrity, MSER + SFOP
- 48. Descriptor Evalua<ons • Matching precision and recall O. Miksik and K. Mikolajczyk, Evalua<on of Local Detectors and Descriptors for Fast Feature Matching, ICPR 2012
- 49. Recent Descriptor Evalua<ons • Computa%on %mes for the different descriptors for 1000 SURF keypoints O. Miksik and K. Mikolajczyk, Evalua<on of Local Detectors and Descriptors for Fast Feature Matching, ICPR 2012 J. Heinly E. Dunn, J-‐M. Frahm, Compara<ve Evalua<on of Binary Features, ECCV2012
- 50. Previous Evalua<ons • Image/object categories – K. Mikolajczyk, B. Leibe, and B. Schiele, “Local features for object class recogni<on,” in ICCV, 2005 – E. Seemann, B. Leibe, K. Mikolajczyk, and B. Schiele, “An evalua<on of local shape-‐based features for pedestrian detec<on,” in BMVC, 2005. – M. Stark and B. Schiele, “How good are local features for classes of geometric objects,” in ICCV, 2007. – K. E. A. van de Sande, T. Gevers and C. G. M. Snoek, Evalua<on of Color Descriptors for Object and Scene Recogni<on. CVPR, 2008.
- 51. Approach • Bags-‐of-‐features 1. Interest point / region detector 2. Descriptors 3. K-‐means clustering (4000 clusters) 4. Histogram of cluster occurrences (NN assignment) 5. Chi-‐square distance and RBF kernel for KDA or SVM classifier • J. Zhang and M. Marszalek and S. Lazebnik and C. Schmid, Local Features and Kernels for Classifica<on of Texture and Object Categories: A Comprehensive Study, IJCV, 2007 • K. E. A. van de Sande, T. Gevers and C. G. M. Snoek, Evalua<on of Color Descriptors for Object and Scene Recogni<on. CVPR, 2008
- 52. Evalua<on • PASCAL VOC measures – Average precision for every object category – Mean average precision Category AP output=> precision AP recall
- 53. MAP Ranking color/gray, density, dimensionality ... MAP #dimensions density • SIFT s<ll dominates (Histograms of gradient loca<ons and orienta<ons) • Opponent chroma<c space (normalized red-‐green, blue-‐yellow, and intensity Y
- 54. Grayvalue descriptors MAP Ranking #dimensions density • Observa<ons • Color improves • All based on histograms of gradient loca<ons and orienta<ons • Dimensionality not much correlated with the performance • Density Strongly correlated (the more the beeer) • Results biased by density • Implementa<on details maeer
- 55. Break !