Towards Web Scale Image Search
1 like742 views
The document discusses limitations of existing text-based and annotation-based image search techniques and proposes using color sketches as an intermediate representation for web-scale content-aware image search. A color sketch captures an image's rough color distribution and spatial information with low computation and storage costs. It represents users' search intentions in a way that is robust to errors and supports large-scale image datasets. Experimental results demonstrate that incorporating color sketches can improve image search over text-only queries.
![Xian-Sheng Hua, MSR Asia
Instant Concept Modeling
• Goal: To learn a simple (but works well on the
candidate image set) visual detector for each concept
butterfly
[Visual instances]
…
Training data for Instance-based
“butterfly” visual detector of
“butterfly”
Graph-based representative image selection](https://image.slidesharecdn.com/2010towardswebscaleimagesearchtudelft-12773721860951-phpapp01/85/Towards-Web-Scale-Image-Search-86-320.jpg)
![Xian-Sheng Hua, MSR Asia
Concept Distribution Estimation
• Goal: estimate the distribution of a concept in an image
[Visual instances]
…
A visual instance
A candidate image
…
…
Estimated distributions of representative butterflies
in the candidate image Estimated distribution](https://image.slidesharecdn.com/2010towardswebscaleimagesearchtudelft-12773721860951-phpapp01/85/Towards-Web-Scale-Image-Search-88-320.jpg)
Towards Web Scale Image Search
- 1. Towards Web-Scale Content-Aware Image Search Xian-Sheng Hua Lead Researcher, Microsoft Research Asia June 23 2010 - TUDelft
- 2. Xian-Sheng Hua, MSR Asia About the Title • Towards Web-Scale Content-Aware Image Search ̵ Towards … ̵ Web-Scale … ̵ Content-Aware … ̵ Image Search …
- 3. Xian-Sheng Hua, MSR Asia Evolution of CBIR Annotation Based Commercial Search ~2006 Engines ~2003 (1) Tagging Based (2) Large-Scale CBIR/CBVR Conventional CBIR ~2001 Direct-Text Based 1990’ Academia Prototypes 1970~ 1980’ Manual Labeling Based 3
- 4. Xian-Sheng Hua, MSR Asia Three Schemes Example-Based Annotation-Based Text-Based
- 5. Xian-Sheng Hua, MSR Asia Limitation of Text-Based Search Surroundings Text Query Social Tags Association
- 6. Xian-Sheng Hua, MSR Asia Limitation of Text-Based Search • High noises ̵ Surrounding text: frequently not related to the visual content ̵ Social tags: also noisy, and a large portion don’t have tags • Mostly only covers simple semantics ̵ Surrounding text: limited ̵ Social tags: People tend to only input simple and personal tags ̵ Query association: powerful but coverage is still limited (non-clicked images will never be well indexed) A complex example: Finding Lady Gaga walking on red carpet and wearing in black
- 8. Xian-Sheng Hua, MSR Asia Limitation of Annotation-Based Search • Low accuracy ̵ The accuracy of automatic annotation is still far from satisfactory ̵ Difficult to be solved • Limited coverage ̵ The coverage of the semantic concepts is still far from the rich content that is contain in an image ̵ Difficult to extend • High computation ̵ Learning costs high computation ̵ Recognition costs high if the number of concepts is large Still has a long way to go
- 9. Xian-Sheng Hua, MSR Asia Limitation of Large-Scale Example-Based Search • You need an example first ̵ How to do it if I don’t have an initial example? • Large-scale (semantic) similarity search is still difficult ̵ Though large-scale (near) duplicate detection is good An example: TinEye
- 10. Xian-Sheng Hua, MSR Asia Possible Way-Outs? • Large-scale high-performance annotation? ̵ Model-based? Data-driven? Hybrid? … • Large-scale manual labeling? ̵ ESP game? Label Me? Machinery Turk? … • New query interface? ̵ Interactive search? Or … To combine text, content and interface?
- 11. Xian-Sheng Hua, MSR Asia Simple Attempts • Exemplary attempts based on this idea ̵ Search by dominant color (Google/Bing) ̵ Show similar images (Bing)/Find similar images (Google) ̵ Show more sizes (Bing)
- 18. Xian-Sheng Hua, MSR Asia Our Solutions: Two Recent Projects • Image Search by Color Sketch • Image Search by Concept Sketch
- 19. Search by Color Sketch WWW 2010 Demo
- 20. Xian-Sheng Hua, MSR Asia Seems It Is Not New? • “Query by sketch” has been proposed long time ago ̵ But on small datasets and difficult to scale up ̵ And seldom work well actually ̵ Not use to use – “object sketch”
- 22. It is based on a SIGGRAPH 95’s paper: C. Jacobs et al. Fast Multiresolution Image Querying. SIGGRAPH 1995 on small scale image set and difficult to scale-up (0.44 second for 10,000 images)
- 23. Xian-Sheng Hua, MSR Asia
- 25. Xian-Sheng Hua, MSR Asia Our Approach • Color sketch … ̵ Is not “object sketch” ̵ But only rough color distribution ̵ Supports large-scale data ̵ And uses text at the same time
- 28. Xian-Sheng Hua, MSR Asia Our Approach • Color sketch … ̵ Is not “object sketch” ̵ But only rough color distribution ̵ Supports large-scale data ̵ And uses text at the same time • Advantages of “Color Sketch” ̵ More presentative than dominant color (and text only) ̵ More robust than “object sketch” ̵ Easy to use compared with “object sketch” ̵ Easy to scale up
- 29. Xian-Sheng Hua, MSR Asia Example: Blue flower with green leaf
- 30. Xian-Sheng Hua, MSR Asia Example: Brooke Hogan walking on red carpet
- 31. Xian-Sheng Hua, MSR Asia Color Sketch Is More Powerful Blue Flower with Green Leaf Brooke Hogan walking on red carpet
- 32. Xian-Sheng Hua, MSR Asia Color Sketch Is More Powerful 32
- 33. Xian-Sheng Hua, MSR Asia The Techniques • Color Sketch: Image Representation ̵ Capturing both color and spatial information ̵ Low computation and storage costs • Intention Map: Intention Representation ̵ Rough: Tolerance to the errors in the colors and positions of the strokes ̵ Sparse: Intention estimation from sparse strokes • Similarity Evaluation ̵ Effective and fast
- 34. Xian-Sheng Hua, MSR Asia Color Sketch: Image Representation Image with grids Color Sketch Computation Cost: less than 50ms/image Storage Cost: 80 byte in average/image
- 35. Xian-Sheng Hua, MSR Asia Intention Map: From User’s Input to Intention Estimation Target: user’s input (1) Do not require users to input the color map exactly (2) Robust to rough and sparse input
- 36. Xian-Sheng Hua, MSR Asia > Target Image 1 Image 2 2010/6/24
- 37. Xian-Sheng Hua, MSR Asia Intention Analysis 2: Relation > Target Image 1 Image 2 2010/6/24
- 38. Xian-Sheng Hua, MSR Asia Intention Analysis 3: Intention Propagation < Propagated Image 1 Image 2 Since the target color sketch is rough, the cells nearby the scribbled cells may have larger chance to have the similar color. Hence, image 2 should be more similar as the blue color appearing in the cells near the scribbled cells. 2010/6/24
- 39. Xian-Sheng Hua, MSR Asia Intention Map: An Example User’s input Intention map for blue In scribbled cells, the intention weight is the largest, nearby cells has relatively smaller intention weight, but the cells scribbled in green and its nearby cells have negative intention weight for blue color.
- 40. Xian-Sheng Hua, MSR Asia Similarity Evaluation • Image color map • Intention map • Similarity
- 41. Xian-Sheng Hua, MSR Asia Experiment: Intention Justification C C+R C+P C+R+P C: Consistency R: Relation P: Propagation 51 text query, each 3 color sketches
- 42. Xian-Sheng Hua, MSR Asia Sensitivity to Grid Dimension
- 43. Xian-Sheng Hua, MSR Asia Comparison
- 44. Xian-Sheng Hua, MSR Asia Demo • It has been productized in Bing
- 62. Gambia flag
- 79. Xian-Sheng Hua, MSR Asia Summary – Why it Works • What are difficult ̵ Semantics is difficult ̵ Intent prediction is difficult • What are easier ̵ Use color to describe images’ content ̵ Use color to describe users’ intent content color sketch intent Color sketch is an intermediate representation to connect images’ content and users’ intent.
- 80. Xian-Sheng Hua, MSR Asia Limitation of Color Sketch • Only works well for those semantics that can be described by color distribution • Only works well when people are able to transfer their desire into color distribution A more complex example: A flying butterfly on the top-left of a flower.
- 81. Search by Concept Sketch SIGIR 2010
- 82. Xian-Sheng Hua, MSR Asia Search By Concept Sketch • A system for the image search intentions concerning: ̵ The presence of the semantic concepts (semantic constraints) ̵ the spatial layout of the concepts (spatial constraints) butterfly flower A concept map query Image search results of our system
- 83. Xian-Sheng Hua, MSR Asia Framework A concept map query A candidate image 1 A: butterfly Create Candidate B: flower Image Set butterfly A: X=0.5, Y=0.5 B: X=0.8, flower Y=0.8 2 3 Instant Concept Concept Distribution Modeling Estimation Concept Models Estimated distributions butterfly butterfly 4 flower Intent flower Interpretation Desired distributions 5 Spatial distribution butterfly comparison flower Relevance score
- 84. Xian-Sheng Hua, MSR Asia Creating Candidate Image Set • Goal: get a group of candidate images from Web ̵ Ensure the presences of the concepts ̵ Reduce computational cost butterfly flower flower butterfly A: butterfly Candidate Images: B: flower Images related to butterfly butterfly & flower flower Training data for concept “butterfly” Training data for concept “flower”
- 85. Xian-Sheng Hua, MSR Asia Instant Concept Modeling • A possible solution – manual tag-to-region Manual tagging is expensive
- 86. Xian-Sheng Hua, MSR Asia Instant Concept Modeling • Goal: To learn a simple (but works well on the candidate image set) visual detector for each concept butterfly [Visual instances] … Training data for Instance-based “butterfly” visual detector of “butterfly” Graph-based representative image selection
- 87. Xian-Sheng Hua, MSR Asia Concept Distribution Estimation • Goal: Automatic concept localization leveraging Web- image resources Web images A: A candidate image Instant Concept B: Distribution A: butterfly B: flower Modeling estimation Estimated distributions Advantages: Scalable to Web-scale concepts butterfly by exploiting rich Web-image resources flower
- 88. Xian-Sheng Hua, MSR Asia Concept Distribution Estimation • Goal: estimate the distribution of a concept in an image [Visual instances] … A visual instance A candidate image … … Estimated distributions of representative butterflies in the candidate image Estimated distribution
- 89. Xian-Sheng Hua, MSR Asia Intention Interpretation • Goal: represent user’s spatial intention as 2D distributions • Principles ̵ A concept should appear near the specified position ̵ A concept should not appear at the positions of the other concepts Desired distributions Desired distributions butterfly butterfly butterfly butterfly flower flower
- 90. Xian-Sheng Hua, MSR Asia Spatial Distribution Comparison • Goal: compute the similarity of the desired distributions and the estimated distributions Desired distributions Estimated distributions butterfly … butterfly flower … flower Distribution comparison Fusion Relevance score
- 91. Xian-Sheng Hua, MSR Asia Demo & Experiments
- 92. Xian-Sheng Hua, MSR Asia Search Results Comparison Task: searching for the images with “snoopy appear at right” snoopy Text-based image search “show similar image” snoopy Image search by concept map
- 93. Xian-Sheng Hua, MSR Asia Search Results Task: searching for the images with “snoopy appear at top” / “snoopy appear at left”
- 94. Xian-Sheng Hua, MSR Asia Search Results Comparison Task: searching for the images with “a jeep shown above a piece of grass” jeep grass
- 95. Xian-Sheng Hua, MSR Asia Search Results Comparison Task: searching for the images with “a car parked in front of a house” house car
- 96. Xian-Sheng Hua, MSR Asia Search Results Comparison Task: searching for the images with “a keyboard and a mouse being side by side” keyboard mouse
- 97. Xian-Sheng Hua, MSR Asia Search Results Comparison Task: searching for the images with “sky/Colosseum/grass appear from top to bottom” Colosseum grass
- 98. Xian-Sheng Hua, MSR Asia Quantitative Evaluation
- 99. Xian-Sheng Hua, MSR Asia Advanced Function: Influence Scope Adjustment • Allow users to explicitly specify the occupied region of a concept house Default house lawn lawn house lawn Search for the long narrow lawn at the bottom of the image
- 100. Xian-Sheng Hua, MSR Asia Advanced Function: Influence Scope Adjustment Searching for small windmill Searching for large windmill
- 101. Xian-Sheng Hua, MSR Asia Advanced Function: Influence Scope Adjustment
- 102. Xian-Sheng Hua, MSR Asia Advanced Function: Visual Modeling Adjustment sky house lawn Clear Search Advanced Function Visual Assistor Previous Next Allow users to indicate or narrow down what a desired concept looks like (visual constraints)
- 103. Xian-Sheng Hua, MSR Asia Advanced Function: Visual Modeling Adjustment Searching for front-view jeeps jeep Searching for side-view jeeps Visual instances
- 104. Xian-Sheng Hua, MSR Asia Advanced Function: Visual Modeling Adjustment Searching for butterfly with yellow flower butterfly flower Searching for butterfly with red flower Visual instances
- 105. Xian-Sheng Hua, MSR Asia Conclusions • The approaches of realizing web-scale content- aware image search ̵ Combing text and content ̵ Providing interface to express search intent ̵ Indexing and searching efficiently • Two exemplary approaches introduced ̵ Search by color sketch ̵ Search by concept sketch
- 106. Xian-Sheng Hua, MSR Asia Thank You