[CVPR 2018] Visual Search (Image Retrieval) and Metric Learning
3 likes1,998 views
The document reviews advancements in image retrieval, particularly focusing on key papers and techniques presented at CVPR 2018, including image embedding and similarity search approaches. It discusses the application of these methods in various areas like retail products and fashion, highlighting significant contributions from researchers and institutions. The summary also touches on state-of-the-art practices in metric learning and the effectiveness of different learning strategies for image similarity evaluation.
[CVPR 2018] Visual Search (Image Retrieval) and Metric Learning
- 1. CVPR 2018 Review In Perspective of Image Retrieval Naver Vision / Insik Kim Vision / YuKyung Choi More complete review (internal only): https://oss.navercorp.com/insikk/public/issues/2
- 2. Contents • Background on Image Retrieval • Interesting “CVPR 2018” papers • Image Embedding (Metric Learning) • Similarity Search
- 3. Background
- 4. Image Retrieval • Application: Naver SmartLens (스마트렌즈) • Search Query as Image • Find relevant information • Products • Flowers, Plants • Clothes • Lotto • Food
- 5. What people do Similarity Search System Approximate Nearest Neighbor Search PQ Encoding Query Expansion and Reranking (RANSAC) Image Embedding Representation Learning Metric Learning
- 6. What people do • State-of-the-art Image Retrieval Models use end-to-end image representation learning
- 7. What people do • Representation Learning • Feature Aggregation • Or, Learn everything in End-to-end
- 8. Who are they? • Diane Larlus (Naver Labs Europe) • Hervé Jégou (Facebook AI Research) • Ondrej Chum (VRG Group, Czech Technical University in Prague) • Other groups • Inria • Alibaba • SenseTime
- 9. Task & Dataset Task Dataset Instance Retrieval (Retail Product) INSTRE Instance Retrieval (Building, Landmark Retrieval) ROxford, RParis, Google Landmark (DeLF paper) Instance Retrieval (Fashion Items) DeepFashion Person Re-identification Market-1501, DukeMTMC • We can also get ideas from • Face verification, recognition • Fine grained visual classification (FGVC) • Of course from general computer vision fields • Detection, Classification, Segmentation • Image Matching
- 10. Image Embedding • Image -> n-dim feature vector • 𝐼 → 𝜙 𝐼 • Answer to how similar given two images are? • 𝑆 𝜙 𝐼 𝑝 , 𝜙 𝐼 𝑞 • Cosine similarity • Learned similarity function • Let’s apply deep learning • Deep Metric Learning ImageNet Feature from AlexNet, tSNE visualization
- 11. Metric Learning
- 12. Metric Learning Idea: • Additional constraint • Exploit mini-batch • CRF to model group consistency • Estimate a pair of image similarity from group consistency Effect: • Achieve SOTA (State-of-the-art) on Market-1501, DukeMTMC
- 13. Metric Learning Idea: • Triplet Loss with latent examples • Learning latent example Effect: • Robust to noise • Faster training
- 14. Metric Learning Idea: • Hard positive, negative image mining • Density information (Random Walk) Effect: • Good for triplet loss training
- 15. Metric Learning Idea: • Unsupervised feature learning with discrimination • N training example, N – way softmax classification • Direct comparison with dot- product Effect: • Semi-Supervised learning (use less labeled data)
- 16. Metric Learning Idea: • Unsupervised feature learning with discrimination • N training example, N – way softmax classification • Direct comparison with dot- product Effect: • Semi-Supervised learning (use less labeled data)
- 17. Similarity Search Finding nearest neighbors from 1,000,000,000 images (1B scale)
- 18. What people do
- 19. Inverted File, Multi Index, Product Quantization TPAMI 2010 CVPR 2012
- 20. Similarity Search Idea: • Learning coarse quantizer in supervised manner • Training requires classification label Effect: • Learned binning works better than unsupervised one (e.g. IMI)
- 21. Similarity Search Idea: • Learning coarse quantizer in supervised manner • Training requires classification label Effect: • Learned binning works better than unsupervised one (e.g. IMI)
- 22. Similarity Search https://github.com/facebookresearch/faiss CoRR 2016 Idea: • Local descriptor. 1000 features / image. 8 bytes / feature. => Global descriptor. 1 feature / image. 128 bytes / feature • HNSW (1M scale) • Find nn with less comparison • Compact representation with novel regression Effects: • HNSW 1M -> 1B scale
- 23. Similarity Search https://github.com/facebookresearch/faiss Idea: • Local descriptor. 1000 features / image. 8 bytes / feature. => Global descriptor. 1 feature / image. 128 bytes / feature • HNSW (1M scale) • Find nn with less comparison • Compact representation with novel regression Effects: • HNSW 1M -> 1B scale