SuperIDR - ECDL 2009
- 1. Superimposed Image Description and Retrieval for Fish Species Identification 29 September, 2009 ECDL 2009, Corfu, Greece Uma Murthy, Edward Fox, Yinlin Chen, & Eric Hallerman* Computer Science *Fisheries and Wildlife Sciences Ricardo Torres, Evandro Ramos, & Tiago Falcão Institute of Computing
- 2. Context of this work digital libraries content services users scenarios documents annotations search annotation scholars biodiversity text-based fisheries retrieval students faculty images text sub- content-based researchers species documents image retrieval identification 2
- 3. Motivation • Many scholarly tasks involve working with images with significant number of details – Species identification, analyzing paintings, studying architecture styles, analyzing medical images, etc. • Scholars combine paper-based and electronic methods/tools – Tools are not well-integrated, information is fragmented and tedious to access ineffective and inefficient task execution 3
- 4. Paper-based methods for fish identification Dichotomous keys Personal Notes 1a Paired fins absent; jaws absent, mouth in an oral disk (the disk mostly surrounded by a fleshy hood in larvae); 7 external gill openings present in row behind eye .......... Lampreys - Petromyzontidae p. ooo 1b Paired fins present (at least 1 set); jaws present; 1 external gill opening per side ................................... 2 2a Caudal fin heterocercal or abbreviate heterocercal (Figure 5) ............................................................ 3 2b Caudal fin protocercal (Figure 13, Part 2, upper left) or homocercal (Figure 5) ............................................ 6 4
- 5. Popular electronic methods FISHBASE: Image collection with browsing, field-wise searching and use of forums EFISH: Organization and browsing based on taxonomy EKEY: Text search and shape- based image retrieval on predefined shapes 5
- 6. Requirements of a new system • Incorporate more visual and descriptive information – Ability to add user content • Improve information management access to heterogeneous information – Images, textual descriptions, notes, image markings, etc. • Provide well-integrated functionalities to support task execution • Provide capability to share content 6
- 7. The Superimposed Image Description and Retrieval Tool (SuperIDR) Content-based image retrieval (CBIR) description and Digital library retrieval of images services Superimposed using image • searching, information features such as browsing, subdocuments shape, color, etc. indexing, etc. annotations • Managing new structure documents, collections, Text-based metadata, etc. image retrieval description and retrieval of images using text 7
- 9. SuperIDR: species description (1/2) 9
- 10. SuperIDR: species image annotation 10
- 12. SuperIDR: text-based image retrieval query 12
- 13. SuperIDR: text-based image retrieval results 13
- 14. SuperIDR: text- and content-based image retrieval (1/2) 14
- 15. SuperIDR: text- and content-based image retrieval (2/2) 15
- 16. User study in an UG Ichthyology class (Spring 2008) – 1/2 • Goal: to assess the usefulness of SuperIDR in fish species identification • Participants: 28 undergraduates, working in teams of two with a tablet PC 16
- 17. Fill entry questionnaire Get assigned tablet Study procedure PCs with SuperIDR Tutorial Use tool for a week, Compare methods make annotations Session 1: half the teams use SuperIDR and half Task: identify 20 unknown Task: identify 20 unknown used traditional methods specimens in 2 sessions specimens in 2 sessions Session 2: teams Fill exit Fill exit swapped methods questionnaire questionnaire Keep/return SuperIDR Keep/return SuperIDR 17
- 18. Summary of results (1/2) • Method had a significant impact on task outcome (p-value=0.015), with higher likelihood of success in using SuperIDR than traditional methods • In general, students were interested in using SuperIDR for species identification – Positive comments by students – Six teams chose to keep the tool 18
- 19. Comments “it was very helpful” “very helpful for taxonomy but need better photos” “very neat but might take a while to master all the key concepts” 19
- 20. Summary of results (2/2) • No significant evidence to show that annotation or search on parts of images is useful – Possibly due to timing and duration of the study 20
- 21. Conclusion • Developed SuperIDR that combines text- and content- based image description, retrieval, and browsing, including for parts of images • Ichthyology students performed better with SuperIDR than with traditional methods in fish species identification • Current/future work – Conduct further experiments to test retrieval effectiveness and usefulness of combined search (text + CBIR) on parts of images – Make SuperIDR available for download – Connect with Flickr to enable information sharing and to leverage existing Flickr collection 21
- 22. Thank you ? ? http://si.dlib.vt.edu/superidr 22
- 23. Back-up slides 23
- 24. Acknowledgements • Dr. Orth, Ryan McManamay, Lindsey Pierce, and others at the Fisheries and Wildlife Sciences department • Microsoft (tablet PC grant) • Jason Lockhart for loaning tablets • NSF (DUE-0435059) 24
- 25. SuperIDR architecture Annotation Search Browse Annotation Search CBISC Lucene Browse Feature Taxonomy Image Species Images descriptions Annotations Vectors & key data marks 25
- 27. Goal Provide better support to work with images, parts of images, providing description through personal notes, linking related information, providing access to existing and new information easily, and being able to share all this information 27
- 28. Species and image data • 207 species of Virginia freshwater fishes – 25 families and 124 genera • 213 images, mostly from Jenkins and Burkhead’s Freshwater Fishes of Virginia 28
- 29. Data collected in experiment • Entry and exit questionnaire responses • Species identification responses and time to identify • Logs of user interaction with the tool • Data from 9 teams (18 students) was considered in analysis – Others not present for both sessions – Incomplete species id. responses 29
- 30. Species identification response summary Traditional methods SuperIdR Team # Correct Team # Correct ID Session (out of 20) ID Session (out of 20) 2 1 15 2 2 18 4 1 16 4 2 17 6 1 13 6 2 17 11 1 12 11 2 16 3 2 8 3 1 14 5 2 13 5 1 15 9 2 12 9 1 10 10 2 10 10 1 14 13 2 11 13 1 11 Mean 12.2 Mean 14.67 Method had a significant impact on outcome and 30 students did better with SuperIDR
- 31. Next steps • Further evaluation to show that combining text- and content-based image retrieval on parts of images is effective and useful • Explore use in other fields - art history, architecture, and other biology-based fields – Involving images with significant number of details 31