An Image Retrieval System for Digital Pathology
1 like488 views
The document describes a content-based image retrieval system developed by researchers at HES-SO Valais-Wallis university to help pathologists search for similar medical images. The system indexes images from proprietary datasets and PubMed Central using visual features and text. When a user draws a region of interest, the system retrieves similar areas from other whole slide images as well as PubMed Central images. Initial user testing found the system intuitive to use but that retrieval quality and speed could be improved.
An Image Retrieval System for Digital Pathology
- 1. HES-SO Valais-Wallis Page 1 This project has received funding from the Eurostars-2 Joint Program with co-funding from the European Union's Horizon 2020 research and innovation program. Manfredo Atzori1, Roger Schaer1, Sebastian Otálora1, Mats Andersson2, Kristian Eurén2, Martin Hedlund2, Henning Müller1 1University of Applied Sciences Western Switzerland (HES-SO) 2ContextVision AB, Sweden Contextvision
- 2. HES-SO Valais-Wallis Page 2 Who are we • MedGIFT research group - 10-15 persons - Medical data analysis • Information Systems Institute • University of Applied Sciences Western Switzerland (HES-SO)
- 3. HES-SO Valais-Wallis Page 3 What is image retrieval? Why can it help pathologists?
- 4. HES-SO Valais-Wallis Page 4 What is an image retrieval system? • System for browsing, searching and retrieving images • Usually using large image databases
- 5. HES-SO Valais-Wallis Page 5 Retrieval systems techniques 1. Text search: search of images based on associated keywords or text 2. Content-based image retrieval (CBIR): search using visual features such as textures, colors, shapes etc.
- 6. HES-SO Valais-Wallis Page 6 Khremoi retrieval system A multimodal retrieval system for radiologists
- 7. HES-SO Valais-Wallis Page 7 Search in the literature Highlighting visual information
- 8. HES-SO Valais-Wallis Page 8 Why retrieval for pathologists? Clinical practice is increasingly complex: 1. Time constraints • Increasingly larger number of cases • Increasingly larger amount of information requested in clinical reports 2. Complexity • Cancer grading is increasingly complex (e.g. due to quantitative parameters) • Rare cases can be difficult to diagnose Ghaznavi et al., Annual Review of Pathology: Mechanisms of Disease 2013 Weinstein et al., Human Pathology, 2009
- 9. HES-SO Valais-Wallis Page 9 Why an image retrieval system for pathologists? Clinical practice is shifting to digital pathology: 1. Visual inspection of tissue samples under microscope is the standard method to grade most types of cancer 2. Digitalization is an ongoing process • Easy data sharing • Modern data analysis techniques (artificial intelligence, computer vision, pattern recognition, multimodal data analysis, image retrieval) Gurcan et al., IEEE Reviews in Biomedical Engineering, 2009 Rubin et al., Lippincott Williams & Wilkins, 2008
- 10. HES-SO Valais-Wallis Page 10 • Save pathologist’s time by an extended and faster interaction with the available digital knowledge • Reduce the need of 2nd opinions • Make learning easier for students & pathologists Objective Development of an image retrieval system for finding similar cases
- 11. HES-SO Valais-Wallis Page 11 Challenges Very complex task with only partial solutions: • Medical imaging retrieval is used in scientific literature, but not in the market • High image variability due to staining procedures, colours, scaling • Histopathology image features are relevant at different scales • Histopathology images are big • A retrieval system must be fast
- 12. HES-SO Valais-Wallis Page 12 How the retrieval system works
- 13. HES-SO Valais-Wallis Page 13 Retrieval system outline Image Datasets Content based information Text information Indexing Image Retrieval Information Extraction
- 14. HES-SO Valais-Wallis Page 14 Retrieval system outline Image Datasets Content based information Text information Indexing Image Retrieval Information Extraction
- 15. HES-SO Valais-Wallis Page 15 Datasets overview 1. Proprietary datasets (Contextvision data) 2. Scientific literature (PubMedCentral) 3. Rare cases database (in progress)
- 16. HES-SO Valais-Wallis Page 16 Proprietary datasets • Currently: Contextvision prostate database (over 120 manually annotated WSI) • Data annotation system for proprietary datasets. • Capability to extract semantic information from reports. Jimenez-del-Toro, SPIE, 2017
- 17. HES-SO Valais-Wallis Page 17 Scientific literature PubMedCentral Over 240’000 images Multimodal data Data Curation
- 18. HES-SO Valais-Wallis Page 18 Rare cases database • In progress • PubMed Central • Books used for teaching • Proprietary data collections
- 19. HES-SO Valais-Wallis Page 19 Retrieval system outline Image Datasets Content based information Text information Indexing Image Retrieval Information Extraction
- 20. HES-SO Valais-Wallis Page 20 Information extraction: content based information Feature extraction: • Patch extraction • Image feature extraction: CEDD (color and edge directivity descriptors), deep learning Gleason Grading features. • Retrieval of similar patches at 5X, 10X and 20X • Automatic magnification check of the selected ROI
- 21. HES-SO Valais-Wallis Page 21 Retrieval system outline Image Datasets Content based information Text information Indexing Image Retrieval Information Extraction
- 22. HES-SO Valais-Wallis Page 22 Information extraction: text Feature extraction: • Apache Lucene is an open source text search engine • Text indexing currently performed using image captions
- 23. HES-SO Valais-Wallis Page 23 Retrieval system outline Image Datasets Content based information Text information Indexing Image Retrieval Information Extraction
- 24. HES-SO Valais-Wallis Page 24 Retrieval procedure Indexing:
- 25. HES-SO Valais-Wallis Page 25 Retrieval system outline Image Datasets Content based information Text information Indexing Image Retrieval Information Extraction
- 26. HES-SO Valais-Wallis Page 26 Image Retrieval 1. The user draws a region of interest in the image 2. The system provides: • Similar areas in the same WSI • Similar areas in other WSIs • Similar images in PubMed Central
- 27. HES-SO Valais-Wallis Page 28 User tests 1. Spontaneous thoughts: • Advantages/disadvantages of the tool? • Are you aware of similar image search tools? 2. Perform annotation and retrieval task: • Are the results sufficiently accurate? • Can the tool be helpful for research or clinical use? • What do you think of the system speed? • What do you think of the system interfaces?
- 29. HES-SO Valais-Wallis Page 30 Interface of the viewer • Navigation bar (1) • Database/feature selection & annotation edition zone (2) • Image zone (3)
- 30. HES-SO Valais-Wallis Page 31 Visual retrieval 1. Content-based Image Retrieval (CBIR): • Retrieved results sorted by relevance • Top left labels indicate Gleason score • Central label indicates similarity 2. Results can be refined via other positive feedback Contextvision
- 31. HES-SO Valais-Wallis Page 32 Multimodal retrieval Both PubMed Central and proprietary datasets support multimodal search Contextvision
- 32. HES-SO Valais-Wallis Page 33 Results: user tests • The comments of the tested users were positive focusing on the intuitive use of the tool • The results quality and speed could be improved • The system was judged a useful tool to improve diagnosis • The system was judged a useful tool to train interns as interactive educational tool
- 33. HES-SO Valais-Wallis Page 34 Conclusions 1. Image retrieval can help in complex situations • Providing information of similar cases • Save time compared to searching in books 2. The presented system was well accepted in the user tests • Some comments regarding the current quality • Limited number of indexed images • Variability of PubMed Central data 3. New tools are needed to manage the large amount of knowledge available in images
- 34. HES-SO Valais-Wallis Page 35 More information Web: http://medgift.hevs.ch/ Email: manfredo.atzori@hevs.ch