Poster: Using Open Source Tools to Improve Access to Oral History Collections
2 likes682 views
The document discusses the Miami Stories Oral History Project, which began in 2005 and collects personal recollections from the university community. It highlights the use of OpenCalais, a natural language processing tool, for automating metadata extraction to improve access to extensive oral history collections, while addressing challenges in data consistency and indexing. The project aims to migrate its contents to new platforms, enhance data quality, and explore better integration methods for improved user access.
Poster: Using Open Source Tools to Improve Access to Oral History Collections
- 1. Using Open Source Tools to Improve Access to Oral History CollectionsBecky Yoose, Bibliographic Systems Librarian and Jody Perkins, Metadata LibrarianMiami University Libraries, Miami University, Oxford, OHMiami Stories OralHistory ProjectBegin in 2005, coordinated by University Archives, CONTENTdm collection maintained by Digital Initiatives
- 2. Current and former students, faculty, and staff, as well as friends of the University share recollections of their Miami years
- 3. 100 videotaped interviewsAverage length of interview: 2 hoursHalf have been fully transcribed OpenCalaisReleased in 2008, used by various companies, news agencies, and publishers
- 4. Uses natural language processing and machine learning to extract categorized metadata (in RDF format) from full text documents
- 5. API, modules, applications available for different platforms DrupalPopular Open Source content management system (CMS) built with PHP
- 6. Used widely for web sites and blogs
- 7. Flexible and customizable, over 8,000 modulesMiami Stories OpenCalais PilotHuman metadata creation workflow: Each interviewer had a cover sheet to list key terms and topics relevant to the interview. These terms were entered as keywords into item records and supplemented with FAST (Faceted Application of Subject Terminology) headings - a controlled vocabulary based on Library of Congress Subject Headings and related LC Authority files. Human metadata creation issues: Data collected on cover sheets varied with the amount of time and number of staff available for a given interview – the sheets varied from having no data to over 50 topics for a single interview. Name entries even for interviewees were inconsistent. The Libraries did not have the staff to manually go through 60+ interview transcripts to manually extract metadata.Pilot project goal: Experiment with applications that automatically generated index terms from full text as more efficient way to provide access points for this collection at the item level. The OpenCalais API offered a number of advantages that made it ideal for this purpose. The faceting framework it employs makes use of categories that are essential to these kinds of historical collections – names, places and locations in particular.OutcomeOC can provide substantial efficiencies when working with large volumes of full text especially for collections where terms representing people, organizations, facilities and locations that are deemed critical access points.Possible Next Steps:Data quality study: Measure data quality using established criteria for:Aboutness / substantive coverage
- 8. Accuracy
- 9. Completeness
- 10. Context
- 11. Consistency
- 12. Interoperability
- 13. UsabilityIntegration, display, and sharing: Currently the Oral History Project is hosted on CONTENTdm; however, the Libraries are in the process of migrating several collections to DSpace. In light of this move, the metadata generated from this project, along with the videos, transcripts, and descriptive metadata, might be calling one of the following platforms “home” in the near future:Drupal
- 14. Omeka http://omeka.org/OpenWMShttp://rucore.libraries.rutgers.edu/open/projects/openwms/ Want to learn more about the technical details of this project? Scan the QR codeor visit http://bit.ly/hYlEHDfor more information!Migrating transcripts into Drupal & batch processing using the Calais Drupal modulein six [oversimplified] easy steps!Step 1: Export XML from CONTENTdm to MySQLStep 2: Import MySQL table using Table WizardStep 3: Migrate content into Drupal using Migrate+Step 4: Edit Calais node settingsNB: We set the Relevancy Threshold to return the maximum number of terms for the project.Step 6: Profit!Step 5: Batch process transcripts using Calais ObservationsOC generates a much larger number of access points, but OC results also included a larger number of false hits/inaccuracies
- 15. OC categories provide a less granular browsing structure
- 16. Terms representing contextual and relational information are lacking in OC results
- 17. Certain aspects of the OC schema don’t suit the content (many irrelevant categories) and there are numerous gaps when compared to the cataloger created metadata
- 18. Meaning of many OC categories is ambiguous making index terms difficult to interpret
- 19. Preservation and genre metadata not captured (since OC only processes text)
- 20. Subject indexing seems to be a weakness of OC – it only generated a few very broad terms, though it did so with a great deal of accuracy
- 21. Name indexing (people, organizations, facilities and locations) seems to be a real strength of OCFor further informationMiami Stories Oral History Projecthttp://doyle.lib.muohio.edu/cdm4/mustories/OpenCalaishttp://www.opencalais.com/ Drupalhttp://drupal.org/ Becky Yoose, Bibliographic Systems Librarian yoosebj@muohio.eduJody Perkins, Metadata Librarianperkintj@muohio.edu Sample of assigned subjectsSample of name entries issuesCONTENTdm topics Anti-Vietnam War protests
- 22. Black Student Action Committee
- 24. Faculty leaving the University
- 25. Faculty Senate
- 27. Long hair and beards on men
- 29. Voices of ReasonOpenCalais tags War
- 30. The Organ (false hit)
- 31. Music
- 32. Education
- 34. Politics
- 35. Religion
- 36. Technology (false hit)Even though the following table shows some of the name entries issues from OC, it should be noted that OC on average created more name entries than catalogers.