Semi-automatic Tagging of Images on Wikimedia Commons
Download as PPTX, PDF0 likes26 views
This document summarizes a presentation on semi-automatic tagging of images on Wikimedia Commons. It discusses student projects at Bern University of Applied Sciences that developed approaches using deep learning and custom models to tag images and link concepts to Wikidata while keeping a human in the loop. It also describes a current project between Wikimedia Sverige, Bern University of Applied Sciences, and SWITCH to implement a prototypical solution using the ISA Tool, Google Cloud Vision, and a new algorithm for tagging images on Wikimedia Commons semi-automatically based on metadata. The presentation reflects on lessons learned and outlines next steps to address issues, increase visibility, improve algorithms, and include structured data on Commons in SWITCH's Research Data Connectome.
Semi-automatic Tagging of Images on Wikimedia Commons
- 1. Berner Fachhochschule | Haute école spécialisée bernoise | Bern University of Applied Sciences Semi-automatic Tagging of Images on Wikimedia Commons Wikidata Con 2023, 28-29 October 2023, Online & Taiwan on-site conference Beat Estermann ▶ Bern University of Applied Sciences, Bern Academy of the Arts Homepage image of the ISA Tool by Islahaddow, using File:2017 06 Ali- 00213.jpg by user:Alimdaihli, CC BY-SA 4.0 (Wikimedia Commons) The text of this slide deck is made available under a CC BY 4.0 License.
- 2. Berner Fachhochschule | Haute école spécialisée bernoise | Bern University of Applied Sciences ▶ How to leverage the power of artificial intelligence for the tagging of images? ▶ Student Projects at Bern University of Applied Sciences in 2020 and 2021: • Leverage deep learning approaches for the tagging of images. • Link concepts to Wikidata. • Develop custom models to escape the “black box” of proprietary algorithms. • The way forward is not fully automatic tagging, but semi-automatic tagging with a human in the loop. • Combine the CAT Tool and the ISA Tool on Wikimedia Commons for a start. • In addition to image recognition, leverage also existing image metadata. Background / Motivation
- 3. Berner Fachhochschule | Haute école spécialisée bernoise | Bern University of Applied Sciences Purpose: Building a FAIR Knowledge Graph of Open Data for Research & Education Try the Beta: https://opendatanavigator.switch.ch Support: Open Data Navigator & API Project Memberships Switch Connectome Project © SWITCH
- 4. Berner Fachhochschule | Haute école spécialisée bernoise | Bern University of Applied Sciences The SWITCH Foundation is the Swiss national infrastructure service provider for higher education and research. The core aspects of the foundation’s mission are to enable, maintain and promote a secure and networked research and education infrastructure in Switzerland. Who is SWITCH? © SWITCH https://www.switch.ch/ © SWITCH
- 5. Berner Fachhochschule | Haute école spécialisée bernoise | Bern University of Applied Sciences ▶ Project Partners: • SWITCH • Wikimedia Sverige / software developers & power user of the ISA Tool • Bern University of Applied Sciences / Bern University of the Grisons ▶ Project Goal: • Implement a prototypical solution for the semi-automatic tagging of images on Wikimedia Commons, using: • the ISA Tool • Google Cloud Vision • a newly developed algorithm for the extraction of entities based on metadata Project Website: https://commons.wikimedia.org/wiki/Commons:ISA_Tool/Image_to_Concept SWITCH InnoLab “Image to Concept” (2022-2023)
- 6. Berner Fachhochschule | Haute école spécialisée bernoise | Bern University of Applied Sciences The ISA Tool (Live Demo…)
- 7. Berner Fachhochschule | Haute école spécialisée bernoise | Bern University of Applied Sciences The Technical Solution Link to File (Wikimedia Commons, User: Beat Estermann, CC BY-SA)
- 8. Berner Fachhochschule | Haute école spécialisée bernoise | Bern University of Applied Sciences
- 9. Berner Fachhochschule | Haute école spécialisée bernoise | Bern University of Applied Sciences Current State of Implementation ▶ Test Version of the enhanced Tool deployed at: https://isa-dev.toolforge.org/ • Tag suggestions from the algorithms not always working. • Saving tags not always working. • The situation is currently worse than during user testing in winter 2022-2023. A series of potential causes for the issues encountered have been identified; bug fixing has been delayed due to a lack of resources. ▶ Production Version of the Tool available at: https://isa.toolforge.org/ (without the new features) • This version was down between May and August 2023 due to prolonged maintenance issues. • Up and running again since mid-August 2023.
- 10. Berner Fachhochschule | Haute école spécialisée bernoise | Bern University of Applied Sciences ▶ Resolve the remaining performance and reliability issues !! ▶ Increase the visibility and take-up of the tool among potential contributors. ▶ Assess and monitor the relevance of the ISA Tool in comparison to other tools (for adding Structured Data on Commons). ▶ Engage in a dialogue with various stakeholders on what constitutes “good” tagging of images. ▶ Further improve/complement the algorithms used for semi-automatic tagging. ▶ Develop alternatives to the current requirement of uploading all media files to Wikimedia Commons (free license requirement is too restrictive for many research use cases). ▶ Clarify roles and responsibilities with regard to deployment, operations, and maintenance (SLAs). Key Learnings
- 11. Berner Fachhochschule | Haute école spécialisée bernoise | Bern University of Applied Sciences ▶ For R&D projects, we need reliable partner organizations. ▶ The respective movement entities need to be empowered for this role. ▶ Maintenance and development of important tools & services need to be backed-up by an organizational commitment. ▶ We should become experts in navigating between the “volunteer” and the “paid staff” worlds; we need to develop a shared professional culture englobing both of them. Reflections with regard to the wider Community Discussion Movement Strategy
- 12. Berner Fachhochschule | Haute école spécialisée bernoise | Bern University of Applied Sciences ▶ Include Structured Data on Commons in the SWITCH Research Data Connectome ▶ Fix the remaining bugs of the enhanced version of the ISA Tool ▶ Further Develop the Metadata-to-Concept Algorithm (students’ project at University of Virginia – School of Data Science) Outlook
- 13. Berner Fachhochschule | Haute école spécialisée bernoise | Bern University of Applied Sciences Beat Estermann Bern Academy of the Arts Representative Open Science Board & Digital Humanities beat.estermann@hkb.bfh.ch Opendata.ch Member of the Board beat.estermann@opendata.ch Contact