Exploring Machine Learning for Libraries and Archives: Present and Future
1 like568 views
The document discusses several ways machine learning (ML) could be used to enhance the work of libraries and archives. It describes projects that used ML for expediting archival processing through tasks like document segmentation and classification. It also discusses using ML to expand descriptive metadata by generating image classifiers and facilitating tagging of photograph collections. Further, it outlines a project aiming to generate and manage rich metadata for audiovisual materials at scale using ML techniques like speech recognition.
Exploring Machine Learning for Libraries and Archives: Present and Future
- 1. Exploring Machine Learning (ML) for Libraries & Archives: Present & Future Bohyun Kim Chief Technology Officer & Professor, University of Rhode Island, USA Twitter: @bohyunkim | Web: http://www.bohyunkim.net/blog/ Bite-sized Internet Librarian International 2021, September 22, 2021.
- 2. Machine Learning (ML) Image from : https://labelbox.com/image-segmentation-overview
- 3. Images from : https://towardsdatascience.com/what-is-machine- learning-a-short-note-on-supervised-unsupervised-semi- supervised-and-aed1573ae9bb
- 5. Discussing ML for Libraries/Archives Images from : https://medium.com/@dhartidhami/machine-learning-basics-model-cost-function-and-gradient-descent-79b69ff28091 ; https://www.analyticsvidhya.com/blog/2020/02/cnn-vs-rnn-vs-mlp-analyzing-3-types-of-neural-networks-in-deep-learning/
- 6. Can ML be a useful tool for the work of libraries & archives? Image from : https://dancohen.org/2019/09/17/humane-ingenuity-3-ai-in-the-archives/
- 7. Q. What would practical application of ML for libraries/archives look like?
- 8. Things I will be going over today: Q. What would practical application of ML for libraries/archives look like? I. ML for Expediting archival processing II. ML for Expanding descriptive metadata III. ML for Enhancing metadata IV. ML for Full content extraction Image from: https://www.analyticsvidhya.com/blog/2020/02/cnn-vs-rnn-vs-mlp-analyzing-3-types-of-neural-networks-in-deep-learning/
- 9. I. ML for Expediting Archival Processing • “Processing” refers to the arrangement, description, and housing of materials for storage and use. • Unprocessed materials pose an impediment to researchers because they cannot be easily identified, accessed, and used. • Machine learning can expedite the processing of archival materials.
- 11. The AIDA Project Explored: • Document Segmentation; • Graphic Element Classification & Text Extraction; • Document Type Classification; • Digitization Type Differentiation; • Document Image Quality Assessment; and • Document Clustering. Source: Elizabeth Lorang et al., “Digital Libraries, Intelligent Data Analytics, and Augmented Description: A Demonstration Project” (University of Nebraska - Lincoln, January 10, 2020), https://digitalcommons.unl.edu/libraryscience/396.
- 12. (a) Identify/Classify Graphical Content & Pull Text with Image Zoning and Segmentation • The AIDA team tried to identify and classify graphical content (e.g. figures, illustrations, and cartoons) and extract text in historical newspaper page images, using ML’s image zoning and segmentation (U-NeXt). • 2 sets of newspaper images were used for ML model training/evaluation. • One set from the Library of Congress’s Beyond Words project, which crowdsourced the graphical content zone demarcation and the text transcription of historical newspaper images. • 1,532 images and corresponding data from graphical content zones; 1,226 images used for training and 306 for evaluation. • The other set from the Europeana Newspapers Project. • A set of 481 page images already zoned and segmented, with the segments classed as background, text, figure, separator or table.
- 13. Source: Elizabeth Lorang et al., “Digital Libraries, Intelligent Data Analytics, and Augmented Description: A Demonstration Project” (University of Nebraska - Lincoln, January 10, 2020), https://digitalcommons.unl.edu/libraryscience/396.
- 14. What is Computer Vision(CV)? • Computer vision is a field of artificial intelligence (AI) that enables computers and systems to derive meaningful information from digital images, videos and other visual inputs — and take actions or make recommendations based on that information.
- 15. Computer Vision Concepts • Image filtering, matching, segmentation, and recognition • Segmentation divides whole images into pixel groupings which can then be labelled and classified. Images from : https://ai.stanford.edu/~syyeung/cvweb/tutorial1.html
- 16. Tasks Performed by Computer Vision • Image classification • Object detection • Object tracking • Content-based image retrieval • Semantic segmentation • Instance segmentation
- 17. (b) Natural Image / Document Classification • The AIDA project team applied natural image and document classification methods in machine learning to a collection of minimally processed manuscript materials to see whether handwritten, printed, and mixed (both handwritten and printed) documents can be distinguished from one another. • The team generated their own model by using the VGG method with 16 categories (VGG-16) pre-trained on the RVL_CDIP dataset. • 999 images were used in total.
- 18. Figure 9. Prediction failure cases. In the left example, the model classified the document as handwritten rather than Mixed. Note that the printed regions are very small compared to the handwritten content in the image. In the right example, the model classified the document as printed rather than mixed. Here, the handwritten region is very small compared to the printed region in the image. Source: Elizabeth Lorang et al., “Digital Libraries, Intelligent Data Analytics, and Augmented Description: A Demonstration Project” (University of Nebraska - Lincoln, January 10, 2020), https://digitalcommons.unl.edu/libraryscience/396.
- 19. (c) Detecting Digitization Type • The AIDA project team also applied machine learning techniques to both classifying manuscript collection images as digitized from original vs. from microform and analyzing their image quality. • 36,103 images were used as the ground truth classification Figure 13. A digital image of a photograph, digitized from the original item, and classified by the classifier as having been digitized from a microform source. Source: Elizabeth Lorang et al., “Digital Libraries, Intelligent Data Analytics, and Augmented Description: A Demonstration Project” (University of Nebraska - Lincoln, January 10, 2020), https://digitalcommons.unl.edu/libraryscience/396.
- 20. Benefits of ML for Archival Processing • There is no doubt that being able to identify graphical content contained in archival materials and to peruse the text in them would make browsing and navigating those materials a lot more manageable to users. And if ML can process archival materials this way, it would surely improve their discoverability and usefulness. • Automating the classification of materials as handwritten, printed, and mixed can surely save archivists’ and librarians’ time spent on performing such classification manually. • Automatically assessing the quality of images and assigning their original source type can help archivists and librarians identify materials at higher-risk and take appropriate preservation measures. Source: Bohyun Kim, “Machine Learning for Libraries and Archives.” Online Searcher 45(1): 39-41, 2021.
- 21. Two Challenges • The results from these explorations indicate that these machine learning methods are not ready to be immediately put to use in archival processing. • “Training a classifier to learn information from rare classes is very hard.” • The project report highlights two challenges: • (a) general scarcity of training data sets; • (b) overall difficulty in obtaining or establishing ground truth (= the expected result against which the performance of a machine learning model is measured). Source: Elizabeth Lorang et al., “Digital Libraries, Intelligent Data Analytics, and Augmented Description: A Demonstration Project” (University of Nebraska - Lincoln, January 10, 2020), https://digitalcommons.unl.edu/libraryscience/396.
- 22. Need for More Data and Ground Truth • Without a sufficiently large amount of training data, a machine learning model for libraries/archives cannot be created. • Without ground-truth datasets, it is difficult to tell how well a machine learning model for libraries/archives works.
- 23. II. ML for Expanding Descriptive Metadata • Mass digitization projects created many digital objects of historical materials, and many more born-digital items are being added to archival collections. • Unfortunately, libraries and archives lack resources to manually create rich metadata for all digital objects in a timely manner. • Lacking such metadata, those digital objects are not easily discovered, identified, or navigated. • ML can advance the discovery, navigation, and clustering of these materials by generating more detailed metadata.
- 24. (a) Image Classifier ML Algorithm for the Frick Collection • The Frick Art Reference Library in New York together with Stanford University, Cornell University, and the University of Toronto developed an image classifier algorithm that applies a local classification system based on visual elements to the library’s digitized Photoarchive. • Tens of thousands of already-labeled images & hundreds of thousands of images that require labeling. • Hierarchical classification headings: • “American School: Portraits: Men: Without hands (Without hats): Head to left”
- 25. Process • The Cornell/Toronto/Stanford team modified the popular ResNet152 network34—with 152 layers and six million parameters—and trained it on a collection of 45,857 classified reproductions of American paintings provided by the Frick Art Reference Library’s Photoarchive. • The majority were portraits and represented 98 unique classification headings. • After training, the network was fed images from the unlabeled portion of the library’s Photoarchive, and the network predicted a classification heading for each one. Source: Ellen Prokop et al., “AI and the Digitized Photoarchive: Promoting Access and Discoverability,” Art Documentation: Journal of the Art Libraries Society of North America 40, no. 1 (March 1, 2021): 1–20, https://doi.org/10.1086/714604.
- 26. Vetting & Prospect • These images were then annotated with the predictions and shown to the library’s photoarchivists through an application to vet the predictions. • The library staff vetted 8,661 images in the year August 2019 - August 2020 and agreed with the network on the classification headings applied to 5,829 (67 %) of the images. • Yet even the incorrect predictions were, in general, “almost correct” with only one tag incorrect or missing. • Holds the potential to improve the staff’s efficiency in in metadata creation and image retrieval.
- 27. (b) CAMPI (Computer-Aided Metadata Generation for Photo archives Initiative) https://www.cmu.edu/news/stories/archives/2020/october/computer-vision-archive.html
- 28. Expanding Item-Level Description Metadata • The CAMPI project team built a prototype application that uses computer vision (CV), in order to aid archivists and metadata editors for Carnegie Mellon University Archives’ General Photograph Collection (GPC) of +20,000 images with metadata at some hierarchy (e.g. job; directory). • This collection of photographic prints, 35mm negatives, and born digital images was specifically chosen for the project, because: • those images lack item-level description and • are stored in and accessed through a shared network drive by archivists and metadata editors, with no browsing interface. Source: Matthew Lincoln et al., “CAMPI: Computer-Aided Metadata Generation for Photo Archives Initiative” (Carnegie Mellon University Libraries, October 9, 2020), https://kilthub.cmu.edu/articles/preprint/CAMPI_Computer- Aided_Metadata_Generation_for_Photo_archives_Initiative/12791807.
- 29. Visual Similarity Search Powered by an “Off- the-shelf” Image Search Model • The CAMPI prototype application offered archivists and metadata editors visual similarity search using CV techniques in ML. • For this visual similarity search capability, a pre-trained “off-the-shelf” image search model was used. • The team used the second-to-last pooled layer of the InceptionV3 trained neural network as the feature source. For each image, this produced an array of 512 numbers, allowing them to treat the entire collection of images as a set of points in 512-dimensional space. • Then, the team used an approximate nearest neighbor search algorithm to create an index of that feature space, allowing it to be queried.
- 30. The prototype allows staff to search similar images to reduce repeated metadata generation and remove noise from the eventual public discovery interface.
- 31. Speeding Up Tagging • The application provided a browsing and CV- assisted tagging interface faceted by the existing job and directory metadata associated with the photographs. • Using this application, archivists and metadata editors were able to start with a tag, find a seed photo, and retrieve similar photos via visual similarity search that specifies the number of neighbors requested. • Then, they were able to quickly tag those retrieved photos with tags from the seed photo when appropriate.
- 33. ==---==
- 35. ML for Metadata Workflow • In their white paper, the CAMPI team reported that of the more than 43,000 tagging decisions made, a little over 1 in 5 were able to be automatically distributed across a set, saving metadata editors more than 9,000 decisions. • They also noted that the prototype application identified around 28% of the collection as sets with duplicates. • This prototype had the effect of rapidly streamlining and speeding up item-level metadata creation. • These results demonstrate how machine learning can be integrated into the existing metadata creation and editing workflow at libraries and archives. Source: Matthew Lincoln et al., “CAMPI: Computer-Aided Metadata Generation for Photo Archives Initiative” (Carnegie Mellon University Libraries, October 9, 2020), https://kilthub.cmu.edu/articles/preprint/CAMPI_Computer- Aided_Metadata_Generation_for_Photo_archives_Initiative/12791807.
- 36. Image Browsing/Filtering UI • What is interesting about the CAMPI project is that even though the use of computer vision was minimal in their prototype application, this project produced a successful outcome because it combined machine learning with a well-designed user interface that took advantage of the existing archival structure, thereby effectively augmenting archivists’ and metadata experts’ existing work. • At libraries and archives, a browsing and filtering interface is primarily created to provide patrons with public access to archival materials. • However, such an interface significantly increases the overall productivity of metadata-related work, if it fits in well with the way the staff perform their work.
- 37. Need for More Training Sets • “There is a field-wide need for specialized computer vision training sets based on historical, nonborn-digital photograph collections.” Source: Matthew Lincoln et al., “CAMPI: Computer-Aided Metadata Generation for Photo Archives Initiative” (Carnegie Mellon University Libraries, October 9, 2020), https://kilthub.cmu.edu/articles/preprint/CAMPI_Computer- Aided_Metadata_Generation_for_Photo_archives_Initiative/12791807.
- 38. III. ML for Enhancing Metadata • The Audiovisual Metadata Platform Pilot Development (AMPPD) Project is run by Indiana University Libraries, University of Texas at Austin School of Information, New York Public Library, and AVP, a data management solutions and consulting company. • Started in 2018 and continuing. • The AMPPD team intends to create automated metadata generation mechanisms and integrate them with the human metadata generation process. • The goal is to generate and manage metadata for audiovisual materials at scale for libraries and archives.
- 40. (a) Generate and Manage Rich Metadata for Audiovisual Materials at Scale • AMPPD Project’s Priority Tasks • First: Detecting silence, speech, and music; speech-to-text / speaker diarization; named entity recognition, such as people, geographic locations, and topics; • Next: Video and structured data OCR (metadata, transcript, subject terms); detecting scene and shot; detecting music genre in audiovisual materials. Source: Shawn Avercamp and Julie Hardesty, “AI Is Such a Tool: Keeping Your Machine Learning Outputs in Check” (Code4Lib 2020 Conference, Pittsburgh, PA, March 11, 2020), https://2020.code4lib.org/talks/AI-is- such-a-tool-Keeping-your-machine-learning-outputs-in-check.
- 41. Annotating AV Materials • It is easy to see that their first-priority items alone will annotate audiovisual materials in such a way that library and archive users can quickly identify the exact content they are looking for and better grasp what kinds of content each audiovisual material contains. • If the AMP can automatically and reliably extract broad subject terms by recognizing named entities within the content, that will also greatly enhance the discoverability and accessibility of those materials.
- 42. ML for Better Discovery, Identification, and Navigation of AV Materials • Extracting the full content of digital AV objects as machine-accessible and interpretable data is a big step towards enabling better discovery, identification, and navigation, because it makes full-text query possible.
- 43. (b) Summarization of Textual Documents • The Catholic Pamphlets collection of over 5,500 PDFs at the University of Notre Dame had sparse summaries. While half of the collection had a MARC summary field, most of these summaries were a few words at most. • 3 NLP (Natural Language Processing) automated summarization techniques of ML were tested to create more robust summaries by combining the summaries with LC subject headings. • 4 NLP algorithms in ML were used: Rapid Automatic Keyword Extraction (RAKE); a raw TextRank implementation algorithm; Genism; Bert Extractive Summarizer (BERT; uses DL) Source: Jeremiah Flannery, “Using NLP to Generate MARC Summary Fields for Notre Dame ’s Catholic Pamphlets,” International Journal of Librarianship 5, no. 1 (July 23, 2020): 20–35, https://doi.org/10.23974/ijol.2020.vol5.1.158.
- 44. Process & Result • Of the 5,500 texts, 51 had sufficient summaries already, and another 50 summaries were written. • The automated summaries were generated after feeding the pamphlets as .pdf files into an OCR pipeline. • Computer-generated summaries of the pamphlets were compared against human-written summaries. • The BERT Extractive technique yielded the best score - an average Rouge F1 score of 0.239. The Gensim python package implementation of TextRank scored at .151. A hand-implemented TextRank algorithm created summaries that scored at 0.144.
- 45. Source: Jeremiah Flannery, “Using NLP to Generate MARC Summary Fields for Notre Dame ’s Catholic Pamphlets,” International Journal of Librarianship 5, no. 1 (July 23, 2020): 20–35, https://doi.org/10.23974/ijol.2020.vol5.1.158.
- 46. Source: Jeremiah Flannery, “Using NLP to Generate MARC Summary Fields for Notre Dame ’s Catholic Pamphlets,” International Journal of Librarianship 5, no. 1 (July 23, 2020): 20–35, https://doi.org/10.23974/ijol.2020.vol5.1.158.
- 47. Challenges in Summarization & Prospect • Latin text and OCR errors have persisted despite choosing more sophisticated algorithms and better text preprocessing. • Better OCR correction & the exclusion of Latin text and sentence fragments remain as challenges. • “For collections in which their text does not need OCR and all appears in English, the BERT Extractive Summarizer likely could provide summaries with few enough errors that catalogers would be comfortable adding the computed-generated summaries with minimal human oversight.” Source: Jeremiah Flannery, “Using NLP to Generate MARC Summary Fields for Notre Dame ’s Catholic Pamphlets,” International Journal of Librarianship 5, no. 1 (July 23, 2020): 20–35, https://doi.org/10.23974/ijol.2020.vol5.1.158.
- 48. IV. ML for Full Content Extraction • Archival materials are given a set of metadata either at the item, collection, or some other unit level, so that they can be searched online. • But their full content is not always readily available in digital form as machine-Interpretable data. This hinders their discoverability and use, such as running full-text or full-content searches and performing other computational analyses. • Machine learning can be utilized for content extraction.
- 49. ML for text, auditory, visual, and tabular data • OCR (an ML method of image recognition) for text documents. • Auditory data can be extracted using machine learning’s speech- recognition method. • From materials such as photographs and video recordings, visual information can be extracted by image recognition techniques.
- 50. ML for text, auditory, visual, and tabular data • Tabular data in digitized images are highly problematic since they cannot be easily identified and utilized as structured data. • But they can also be extracted using table detection, cell recognition, and text extraction algorithms in machine learning. • Extracting tabular data along with their data structure will make historical data immediately useful to researchers. Ryan Cordell, “Machine Learning + Libraries” (Library of Congress, July 14, 2020), https://labs.loc.gov/static/labs/work/reports/Cordell-LOC-ML-report.pdf?loclr=blogsig.
- 51. Future Focuses for Libraries/Archives • Need for more data & ground truth for ML algorithm training and evaluation • Exploration of pre-trained off-the-shelf ML models • e.g. AWS Rekognition (aws.amazon.com/rekognition) and Google Cloud Vision AI (cloud.google.com/vision). • Need for continued digitization
- 52. Thank you! Bohyun Kim Chief Technology Officer & Professor, University of Rhode Island, USA Twitter: @bohyunkim | Web: http://www.bohyunkim.net/blog/ Bite-sized Internet Librarian International 2021, September 22, 2021.