Mind finder: interactive sketch-based image search on millions of images
0 likes784 views
MindFinder is an interactive sketch-based image search system that allows users to search over 2 million images using sketches, tags, and colors. It addresses limitations of prior sketch-based search by representing images as salient curves matched to user sketches. The system indexes sketches, tags, and colors for real-time multimodal search. An evaluation found sketch-based search to be more accurate and convenient than text for specific intents, demonstrating MindFinder's ability to help users better express search needs through simple sketches.
Mind finder: interactive sketch-based image search on millions of images
- 1. 2012.06.07 UX Lab LEE NAMMIN MM '10 Proceedings of the interna4onal conference on Mul4media MindFinder: interac-ve sketch-‐based image search on millions of images Yang Cao, Hai Wang, Changhu Wang, Zhiwei Li, Liquing Zhang, Lei Zhang @MicrosoI Research Keywords: Query formula4on Search process User interface Informa4on search Interac4ve search User centered design
- 2. MOTIVATION | Is text-based query sufficient? “어우. 이거 뭐라고 써야되지?” a text query is usually too ambiguous to properly convey user’s search intentions.
- 3. MOTIVATION | 기존의 검색 방식들 •text-based : 많이 사용되고 있지만 심상 이미지를 텍스트 형태로 만들기란 어려움. •content-based : 이미지 소스가 없으면 사용할 수 없음. To overcome existing problems, a natural solution is to enable users to flexibly express what they want. | 어떻게? •심상 이미지의 형태를 스케치하도록 해서, •기존의 text-based 검색에서처럼 태그도 이용할 수 있게 해서, •몇가지 지배적인 색을 지정할 수 있도록 해서,
- 4. INTRODUCTION | Sketch-based 터치 디바이스 보편화 -> 스케치를 기반으로 한 기능이 동작하기에 충분한 환경이 마련됨. among various query modalities, sketch is probably the most challenging one. 1990년부터 연구되어왔지만, 다음 세 가지 문제요소 때문에 뚜렷한 성과가 없었음. | the 1st barrier There is an unavoidable gap between the binary map sketched by the user and full color natural images in the database. | the 2nd barrier the difficulty in meeting the requirement for both efficiency and accuracy when matching a query sketch with curves in a natural image. | the 3rd barrier Most existing sketch-based search engines have no indexing mechanism.
- 5. INTRODUCTION | revisit these problems, and have achieved very promising results. | the 1st barrier : representation To bridge the gap, we represent a natural image by its salient curves, which has potential to be closer to the sketch queries from users. | the 2nd barrier : matching a raw curve-based algorithm is used to efficiently and precisely calculate the similarity between the salient curve representation of natural images and a user’s sketch query. | the 3rd barrier : indexing design an indexing strategy to speed up the matching process and make the system scalable to millions of images. : the first large-scale indexing framework
- 6. INTRODUCTION | MindFinder It is a real-time sketch-based image search engine. the task suddenly becomes much easier. by simply drawing the appearance of the pendant. gives us the ability to better express our search intentions using a simple sketch.
- 7. INTRODUCTION | Mechanism It enables users to flexibly express search intention by interactively sketching, tagging and coloring. over 2 million web images
- 9. SYSTEM OVERVIEW | Sketch Querying 기본적으로 스케치로 찾는 기능이 메인. 스케치 패널에 심상 이미지의 윤곽선을 그리면 동적으로 결과패널이 업데이트 됨. The sketch-based search in our system is precise and structure-sensitive, which guarantees that the curves in natural images are highly matched with the strokes drawn by users.
- 10. SYSTEM OVERVIEW | Sketch + Tag Querying 스케치 패널에서 텍스트 모드로 변경하고 타이핑하거나, 링바에 나타나있는 일반적으로 많이 사용하는 태그들을 스케치 패널에 드래그 앤 드랍 By only using the keyword-based search, top images were diverse and far from the user’s intention. After adding a sketch query to confine the main structures, all top results met the user’s requirement.
- 11. SYSTEM OVERVIEW | Sketch + Tag + Color Querying 지배적인 특정 색상에 대한 색상 쿼리를 입력하거나, 링 바에 있는 컬러 바에서 색을 선택 by specifying a certain dominant color, the user could easily find pictures of that in any color she/he wants. Besides, the composition of the returned images still meet the user’s search intention.
- 12. TECHNICAL DETAILS | back-end database 2,114,085 Flicker photos with tag information using the top 1000 hot queries. downsampled each image to a suitable size adopted a saliency discovering method to extract major curves each full color images is transformed into a binary map | matching adopted an raw curve-based algorithm to achieve precise matching | indexing further speeded up by an index structure. The tag and color features are indexed by inverted file structures, which totally take less than 1GB memory. The sketch index occupies less than 7GB memory, and thus MindFinder could be easily applied on a normal Intel machine. all index structures were pre-built offline, by our well designed architecture, a typical response time of a complex query is between 1 and 2 seconds.
- 13. TECHNICAL DETAILS | collaborate multimodal search conditions A simple way, 각 조건에 따라 따로 찾아서 나중에 하나의 리스트로 머지하는 방법. -> 규모가 큰 데이터베이스에서는 오버랩 되는 이미지들이 적을 것이기 때문에 머지했을 때 결과가 별로 없을 확률이 높음. tag : to cross the semantic gap To deal with this problem, we use only one query type in the mixed query to retrieve an image set, and other query types are worked as reranking conditions on this set.
- 16. RESULT | comparison between MindFinder & traditional search engine the sketch-‐based search is more accurate and convenient than the tradi-onal search when a user’s search intent is specific and complex.
- 17. CONCLUSIONS | main contributions •MindFinder is the 1st sketch-‐based mul-modal search engine for more than two million web images. •Our system provides a convenient interface for users to freely express their search inten-ons, and enables real-‐-me interac-ons for users to more efficiently locate their desired images. •It is the 1st index-‐based query-‐by-‐sketch solu-on for million level database.
- 19. + | approach 단순해 보이지만, 기본으로 돌아가서 “표현하기 어려우면 표현하는 것을 도와주면 되잖아!” | process 왜 스케치 기반의 디자인을 생각하게 되었는지에 대해 궁금. | mechanism 심상(목표)이 구체적이고 복잡한 경우로 한정지었음. 이미 상용화된거라면, 이 기능을 통해 이미지를 찾아가는 과정을 살폈다면 내 논문이 더 간결해졌을텐데! (심상 형성 -> 타협의 과정) | contributions 간단한 서베이나 인터뷰 등의 UT를 통해 나온 데이터를 바탕으로 이야기했으면 좋았을 것! 우리가 쓴다면, 아마 이 논문의 앞단계의 사용자 조사와 디자인, 그리고 평가까지.. 재밌었겠다는 상상
- 20. EOD THANK YOU