Terabyte-scale image similarity search: experience and best practice
6 likes2,020 views
The document discusses terabyte-scale image similarity search leveraging Hadoop to index and search massive image datasets for applications like copyright violation detection. It covers the experimental setup, challenges with heterogeneous cluster performance, and best practices for managing large auxiliary data structures during image indexing and searching. The authors also propose methodologies for efficient Hadoop deployment and raise open questions regarding the analysis of job execution logs.
![Image search with Hadoop
● Index & search huge image collection using
MapReduce-based eCP algorithm
○ See our work at ICMR'13: Indexing and
searching 100M images with MapReduce [18]
○ See Section III for quick overview
● Use the Grid5000 plartform
○ Distributed infrastructure available to French
researchers & their partners
● Use the Hadoop framework](https://image.slidesharecdn.com/bigdata13071013-131017085623-phpapp01/85/Terabyte-scale-image-similarity-search-experience-and-best-practice-7-320.jpg)
![Supporting publications
[18] D. Moise, D. Shestakov, G. Gudmundsson, L. Amsaleg. Indexing and
searching 100M images with Map-Reduce. In Proc. ACM ICMR '13, 2013.
[20] D. Shestakov, D. Moise, G. Gudmundsson, L. Amsaleg. Scalable highdimensional indexing with Hadoop. In Proc. CBMI'13, 2013.
[this-bigdata13]
D. Moise, D. Shestakov, G. Gudmundsson, L. Amsaleg.
Terabyte-scale image similarity search: experience and best practice. In Proc.
IEEE BigData'13, 2013.
[submitted] D. Shestakov, D. Moise, G. Gudmundsson, L. Amsaleg.
Scalable high-dimensional indexing and searching with Hadoop.](https://image.slidesharecdn.com/bigdata13071013-131017085623-phpapp01/85/Terabyte-scale-image-similarity-search-experience-and-best-practice-29-320.jpg)
Terabyte-scale image similarity search: experience and best practice
- 1. Terabyte-scale image similarity search: experience and best practice Diana Moise2, Denis Shestakov1,2, Gylfi Gudmundsson2, Laurent Amsaleg3 1 Department of Media Technology, School of Science, Aalto University, Finland 2 Inria Rennes – Bretagne Atlantique, France 3 IRISA - CNRS, France Denis Shestakov denis.shestakov at aalto.fi linkedin: linkedin.com/in/dshestakov mendeley: mendeley.com/profiles/denis-shestakov
- 2. Terabyte-scale image search in Europe?
- 3. Overview 1. Background: image retrieval, our focus, environment, etc. 2. Applying Hadoop to multimedia retrieval tasks 3. Addressing Hadoop cluster heterogeneity issue 4. Studying workloads with large auxiliary data structure required for processing 5. Experimenting with very large image dataset
- 4. Image search? Content-based image search: ● Find matches with similar content
- 5. Image search applications? ● regular image search ● object recognition ○ face, logo, etc. ● for systems like Google Goggles ● augmented reality applications ● medical imaging ● analysis of astrophysics data
- 6. Our use case ● Copyright violation detection ● Our scenario: ○ Searching for batch of images ■ Querying for thousands of images in one run ■ Focus on throughput, not on response time for individual image ● Note: indexed dataset can be searched on single machine with adequate disk capacity if necessary
- 7. Image search with Hadoop ● Index & search huge image collection using MapReduce-based eCP algorithm ○ See our work at ICMR'13: Indexing and searching 100M images with MapReduce [18] ○ See Section III for quick overview ● Use the Grid5000 plartform ○ Distributed infrastructure available to French researchers & their partners ● Use the Hadoop framework
- 8. Experimental setup: cluster ● Grid5000 platform: ○ Nodes in rennes site of Grid5000 ■ Up to 110 nodes available ■ Nodes capacity/performance varied ● Heterogenous, come from three clusters ● From 8 cores to 24 cores per node ● From 24GB to 48GB RAM per node
- 9. Experimental setup: framework ● Standard Apache Hadoop distribution, ver.1.0.1 ○ (!) No changes in Hadoop internals ■ Pros: easy to migrate, try and compare by others ■ Cons: not top performance ○ Tools provided by Hadoop framework ■ ■ ■ ■ Hadoop SequenceFiles DistributedCache multithreaded mappers MapFiles
- 10. Experimental setup: dataset ● 110 mln images (~30 billion SIFT descriptors) ○ Collected from the Web and provided by one of the partners in Quaero project ■ Largest reported in literature ○ Images resized to 150px on largest side ○ Worked with ■ The whole set (~4TB) ■ The subset, 20mln images (~1TB) ○ Used as distracting dataset
- 11. Experimental setup: querying ● For evaluation of indexing quality: ○ Added to distracting datasets: ■ INRIA Copydays (127 images) ○ Queried for ■ Copydays batch (~3000 images = 127 original images and their associated variants incl. strong distortions, e.g. print-crumple-scan ) ■ 12k batch (~12000 images = 245 random images from dataset and their variants) ■ 25k batch ○ Checked if original images returned as top voted search results
- 12. Image search with Hadoop Distributed index creation ● Clustering images into a large set of clusters (max cluster size = 5000) ● Mapper input: ○ unsorted SIFT descriptors ○ index tree (loaded by every mapper) ● Mapper output: ○ (cluster_id, SIFT) ● Reducer output: ○ SIFTs sorted by cluster_id
- 13. Image search with Hadoop Indexing workload characteristics ● computationally-intensive (map phase) ● data-intensive (at map&reduce phases) ● large auxiliary data structure (i.e., index tree) ○ grows as dataset grows ○ e.g., 1.8GB for 110M images (4TB) ● map input < map output ● network is heavily utilized during shuffling
- 14. Image search with Hadoop
- 15. Image search with Hadoop Searching workflow ● large aux.data structure (e.g., lookup table)
- 16. Index search with Hadoop: results ● Basic settings: ○ 512MB chunk size ○ 3 replicas ○ 8 map slots ○ 2 reduce slots ● 4TB dataset: ○ 4 map slots
- 17. Hadoop on heterogeneous clusters Capacity/performance of nodes in our cluster varied ○ ○ ○ ○ Nodes come from three clusters From 8 cores to 24 cores per node From 24GB to 48GB RAM per node Different CPU speeds ● Hadoop assumes one configuration (#mappers, #reducers, maxim. map/reduce memory, ...) for all nodes ● Not good for Hadoop clusters like ours
- 18. Hadoop on heterogeneous clusters ● Our solution (hack): ○ deploy Hadoop on all nodes with settings addressing the least equipped nodes ○ create sub-cluster configuration files adjusted to better equipped nodes ○ restart tasktrackers with new configuration files on better equipped nodes ● We call it ‘smart deployment’ ● Considerations: ○ Perhaps rack-awareness feature of Hadoop should be complemented with smart deployment functionality
- 19. Hadoop on heterogeneous clusters ● Results ○ indexing 1T on 106 nodes: 75min → 65min
- 20. Large auxiliary data structure ● Some workloads require all mappers to load a largesize data structure ○ E.g., both in image indexing and searching workloads ● Spreading data file across all nodes: ○ Hadoop DistributedCache ● Not efficient if structure is of gigabytes-size ● Partial solution: increase HDFS block sizes → decrease #mappers ● Another solution: multithreaded mappers provided by Hadoop ○ Poorly documented feature!
- 21. Large auxiliary data structure ● Multithreaded mapper spans a configured number of threads, each thread executes a map task ● Mapper threads share the RAM ● Downsides: ○ synchronization when reading input ○ synchronization when writing output
- 22. Large auxiliary data structure ● Let’s test it! ● Indexing 4T with 4 mappers slots, each running 2 threads ○ index tree size: 1.8GB ● Indexing time: 8h27min → 6h8min
- 23. Large auxiliary data structure ● In some application, mappers needs only a part of auxiliary data structure (the one relevant to data block processed) ● Solution: Hadoop MapFile ● See Section 5.C.2 ○ Searching for 3-25k image batches ○ Though it is rather inconclusive ● Stay tuned! ○ A proper study of MapFile is now in progress
- 24. Open questions ● Practical one: ○ What are best practices for analysis of Hadoop job execution logs? ● Analysis of Hadoop job logs happened to be very useful in our project ○ Did with our python/perl scripts ● It is extremely useful for understanding and then tuning Hadoop jobs on large Hadoop clusters ● Any good existing libraries/tools? ○ E.g., Starfish Hadoop Log analyzer (Duke Univ.)
- 25. Open questions E.g., search (12k batch over 1TB) job execution on 100 nodes
- 26. Observations & implications ● HDFS block size limits scalability ○ 1TB dataset => 1186 blocks of 1024MB size ○ Assuming 8-core nodes and reported searching method: no scaling after 149 nodes (i.e. 8x149=1192) ○ Solutions: ■ Smaller HDFS blocks, e.g., scaling up to 280 nodes for 512MB blocks ■ Re-visit search process: e.g., partial-loading of lookup table ● Big data is here but not resources to process ○ E.g, indexing&searching >10TB not possible given resources we had
- 27. Things to share ● Our methods/system can be applied to audio datasets ○ No major changes expected ○ Contact me/Diana if interested ● Code for MapReduce-eCP algorithm available on request ○ Should run smoothly on your Hadoop cluster ○ Interested in comparisons ● Hadoop job history logs behind our experiments available on request ○ Describe indexing/searching our dataset by giving details on map/reduce tasks execution ○ Insights on better analysis/visualization are welcome ○ E.g., job logs supporting our CBMI'13 work: http://goo. gl/e06wE
- 28. Acknowledgements ● Aalto University http://www. aalto.fi ● Quaero project http://www. quaero.org ● Grid5000 infrastructure & its Rennes maintenance team http://www.grid5000.fr
- 29. Supporting publications [18] D. Moise, D. Shestakov, G. Gudmundsson, L. Amsaleg. Indexing and searching 100M images with Map-Reduce. In Proc. ACM ICMR '13, 2013. [20] D. Shestakov, D. Moise, G. Gudmundsson, L. Amsaleg. Scalable highdimensional indexing with Hadoop. In Proc. CBMI'13, 2013. [this-bigdata13] D. Moise, D. Shestakov, G. Gudmundsson, L. Amsaleg. Terabyte-scale image similarity search: experience and best practice. In Proc. IEEE BigData'13, 2013. [submitted] D. Shestakov, D. Moise, G. Gudmundsson, L. Amsaleg. Scalable high-dimensional indexing and searching with Hadoop.
- 30. Thank you!