Image Tag Refinement Along the 'What' Dimension using Tag Categorization and Neighbor Voting
0 likes379 views
The document presents a strategy for refining image tags using tag categorization and neighbor voting to improve image retrieval effectiveness. It highlights the prevalence of noisy tags in social media image folksonomies and proposes a method that considers the semantic categories of tags. Experimental results demonstrate the efficacy of the refined tagging approach, particularly in enhancing the recommendation accuracy of relevant tags.
![Neighbor Voting (2/3) input image tag relevance learning [Figure adopted from X. Li et al. ] bridge bicycle perfect MyWinners robyn fishing me bristol court 1 number Sydney bridge Australia architecture bridge tranquil bruges trees NikonE3100 Sydney bridge SuperShot clouds a5PhotosaDay ireland irlanda ingiro northireland irlandadelnord connemara Sweden bridge lake APlusPhot SuperAPlus image folksonomy retrieval of visual neighbors bridge 4 bicycle 0 perfect 0 MyWinners 0](https://image.slidesharecdn.com/lee10image540423-100908043035-phpapp02/85/Image-Tag-Refinement-Along-the-What-Dimension-using-Tag-Categorization-and-Neighbor-Voting-10-320.jpg)
![Evaluation Metrics Image tag refinement Noise Level ( NL ) NL denotes the ratio of noisy tags to all tags in an image folksonomy Image tag recommendation Precision at rank k ( [email_address] ) [email_address] denotes the ratio of recommended tags that are relevant, averaged over all photos](https://image.slidesharecdn.com/lee10image540423-100908043035-phpapp02/85/Image-Tag-Refinement-Along-the-What-Dimension-using-Tag-Categorization-and-Neighbor-Voting-17-320.jpg)
![Objective Performance of Image Tag Recommendation Original folksonomy Refined folksonomy without categorization with categorization [email_address] 0.285 0.505 0.680 [email_address] 0.315 0.330 0.405 the P@1 and P@5 values demonstrate that the proposed tag refinement technique improves the effectiveness of image tag recommendation for non-tagged images](https://image.slidesharecdn.com/lee10image540423-100908043035-phpapp02/85/Image-Tag-Refinement-Along-the-What-Dimension-using-Tag-Categorization-and-Neighbor-Voting-22-320.jpg)
Image Tag Refinement Along the 'What' Dimension using Tag Categorization and Neighbor Voting
- 1. IMAGE TAG REFINEMENT ALONG THE ‘WHAT’ DIMENSION USING TAG CATEGORIZATION AND NEIGHBOR VOTING IEEE International Conference on Multimedia & Expo Singapore – July 19-23, 2010 Sihyoung Lee , Wesley De Neve, Yong Man Ro Image and Video Systems Lab Department of Electrical Engineering Korea Advanced Institute of Science and Technology (KAIST) email: ijiat@kaist.ac.kr
- 2. Outline Introduction Image tag refinement Experiments Conclusions
- 3. Outline Introduction Image tag refinement Experiments Conclusions
- 4. Introduction Social media applications allow users to share and manually annotate photos Flickr hosts more than 4 billion photos as of Oct. 2009 more than 2.5 billion photos are uploaded to Facebook each month as of Jan. 2009 Image folksonomies collections of user-provided images annotated with user-supplied tags user-supplied tags are used by most search engines to retrieve images
- 5. Question Can we trust all image tags in an image folksonomy?
- 6. Noisy Tags What are noisy tags? tags that are irrelevant to the image content meaningless tags (e.g., `asdf’ and `grrr’), tags containing typographical errors, imprecise tags, and incorrect tags Why are noisy tags present? people tend to interpret an image by using their personal experience, knowledge, and imagination people tend to assign the same tags to different images that have been taken at the same place or during the same event (i.e., batch tagging)
- 7. Problem Statement Need for identifying correct tags and noisy tags in order to boost the effectiveness of image retrieval
- 8. Outline Introduction Image tag refinement Experiments Conclusions
- 9. Neighbor Voting (1/3) De facto standard for tag relevance learning X. Li et al. , “ Learning social tag relevance by neighbor voting ”, IEEE ToM, 11(7), pp. 1310-1322, Nov. 2009 Assumption if different people label visually similar images using the same tags, then these tags are likely to reflect objective aspects of the visual image content Algorithm follows a two-step approach retrieval of visual neighbors tag relevance learning by accumulating votes from the tags assigned to the visual neighbors
- 10. Neighbor Voting (2/3) input image tag relevance learning [Figure adopted from X. Li et al. ] bridge bicycle perfect MyWinners robyn fishing me bristol court 1 number Sydney bridge Australia architecture bridge tranquil bruges trees NikonE3100 Sydney bridge SuperShot clouds a5PhotosaDay ireland irlanda ingiro northireland irlandadelnord connemara Sweden bridge lake APlusPhot SuperAPlus image folksonomy retrieval of visual neighbors bridge 4 bicycle 0 perfect 0 MyWinners 0
- 11. Neighbor Voting (3/3) Disadvantage: “a one size fits all” approach the same technique is applied to all tags, regardless of the type of these tags How about (additionally) taking into account location information (e.g., GPS information)? time information (available in Exif metadata)? Idea take into account the type of tags in order to allow for more specialized tag relevance learning techniques
- 12. Previous Tag Refinement Image folksonomy refined tags Retrieval of visual neighbors Voting Neighbor voting algorithm nikon, leaves, jw, east, clouds, coast, ground, rita, red, rain, newengland, pretty, yellow, summer, portrait, pink, macro, grass, flowers, etc …
- 13. Proposed Tag Refinement Tag categorization WordNet refined tags along the what dimension Visual information-based refinement GPS-based refinement where refined tags along the where dimension Time-based refinement when refined tags along the when dimension Refinement using affective content analysis how refined tags along the how dimension who refined tags along the who dimension Face recognition-based refinement what Neighbor voting algorithm Retrieval of visual neighbors Image folksonomy Scope of this paper: tag refinement along the ‘what’ dimension Voting nikon, leaves, jw, east, clouds, coast, ground, rita, red, rain, newengland, pretty, yellow, summer, portrait, pink, macro, grass, flowers, etc …
- 14. Tag Categorization Tags assigned to an image are mapped to the WordNet noun semantic categories The WordNet noun semantic categories are mapped onto five intuitive categories that are relevant for users what , when , who , where , and how WordNet semantic noun categories Our categories animal, artifact, attribute, body, food, object, phenomenon, plant, shape, something, substance what location, space where person, group who time, event when act, cognition, communication, feeling, motive, possession, process, quantity, relation, state how
- 15. Outline Introduction Image tag refinement Experiments Conclusions
- 16. Experimental Setup Image set used: MIRFLICKR-25000 images were annotated with 223,537 tags by 9,862 users concept vocabulary of 68,004 unique tags test set 200 images from MIRFLICKR25000 annotated with at least four tags along the what dimension Image descriptor MPEG-7 Scalable Color (256-D vector) L 1 distance is adopted to measure the visual distance
- 17. Evaluation Metrics Image tag refinement Noise Level ( NL ) NL denotes the ratio of noisy tags to all tags in an image folksonomy Image tag recommendation Precision at rank k ( [email_address] ) [email_address] denotes the ratio of recommended tags that are relevant, averaged over all photos
- 18. Results for Tag Categorization Most frequent WordNet categories for the user-supplied tags in MIRFLICKR-25000 46% 54% 22% 17% 12% 7% 5% 37% 46% 19% 17% 12% 6%
- 19. Objective Performance of Tag Refinement (1/2) Before tag refinement After refinement without categorization with categorization NL 0.774 0.466 0.249 proposed tag refinement removes more noisy tag assignments than tag refinement only making use of neighbor voting
- 20. Objective Performance of Tag Refinement (2/2) before tag refinement after tag refinement without tag categorization after tag refinement with tag categorization
- 21. Subjective Performance of Tag Refinement Image Tags Before tag refinement After tag refinement without categorization with categorization bc, beach , british, cacade, canada, casio, cloud , clouds , columbia, exf1, firstquality, forest, fpg, fun, hike, hiking, island , juandefuca, lagoon , lighthouse, metchosin, mountain , mountains , ocean , picnic, play, range, sand , state, trail, vancouver, victoria, washington, water , wave , waves , witty, wittys clouds , beach , water , ocean , cloud , sand , mountains , island , wave , mountain , canada, washington, forest, trail, waves , hiking clouds , beach , water , ocean , cloud , sand , mountains , island , mountain , trail
- 22. Objective Performance of Image Tag Recommendation Original folksonomy Refined folksonomy without categorization with categorization [email_address] 0.285 0.505 0.680 [email_address] 0.315 0.330 0.405 the P@1 and P@5 values demonstrate that the proposed tag refinement technique improves the effectiveness of image tag recommendation for non-tagged images
- 23. Subjective Performance of Image Tag Recommendation Image Recommended tags Original folksonomy Refined folksonomy without categorization with categorization explore, sky , blue , water, nature, yellow , clouds , nikon, canon, 2007 sky , explore, blue , nature, yellow , water, canon, clouds , white , geotagged sky , blue , nature, yellow , water, clouds , building , architecture , beach, snow explore, nature , green , sky , blue , macro, water, nikon, flower , clouds nature , green , sky , blue , explore, macro, yellow , canon, flower , water nature , green , sky , blue , yellow , flower , water, clouds , bird, grass sky , explore, blue , clouds , nature , water, landscape, nikon, canon, geotagged sky , blue , clouds , nature , explore, water, landscape, hdr, yellow , canon sky , blue , clouds , nature , water, yellow , snow, bird , lake, beach
- 24. Outline Introduction Image tag refinement Experiments Conclusions
- 25. Conclusions Proposed a tag refinement strategy that makes use of tag categorization and neighbor voting allows taking into account other information sources for tag relevance learning, besides visual information dependent on the nature of the tags Presented experimental results focusing on tag refinement along the what dimension Future research design of specialized tag refinement techniques that are able to operate along other dimensions
- 26. Thank you! Any questions?