Digitisation at Scale: Automating the mass acquisition of digitised content
Download as PPT, PDF1 like1,831 views
The document discusses the Wellcome Library's five-year plan to digitize its collections to make them accessible globally, detailing the challenges and solutions related to scaling digitization processes. It emphasizes the use of the Goobi system for managing and automating the digitization process while highlighting the importance of efficient workflow design and vendor collaboration. The document also reflects on lessons learned from the automation process, such as the need for metadata and human oversight.
Digitisation at Scale: Automating the mass acquisition of digitised content
- 1. Digitisation at Scale: Automating the mass acquisition of digitised content IS&T Archiving Conference, Washington, April 2016 Dave Thompson Digital Curator, Wellcome Library
- 2. The Wellcome Library • Part of Wellcome Collection, astonishing public venue in London developed by the Wellcome Trust. Where people can learn more about medicine through the ages & across cultures • Five-year plan for transforming the Wellcome Library.
- 3. Driver for digitisation • To make our collections available to anyone, anywhere, we are digitising as much of our physical collection as we can, for both our website and the websites of other organisations. We are also digitising and hosting collections from partners that complement our holdings Transforming the Wellcome Library: 2009-2014. http://wellcomelibrary.org/what-we-do/library-strategy-and-policy/transforming- the-wellcome-library/
- 4. The problem • How to scale systems & processes to deliver on our ambition • How to design & build new high volume systems & processes for; acquisition, storage, processing, access • How to manage volumes of data during creation/acquisition
- 5. Process design – sources of content Goobi (METS/OCR) Preservica In-house Institutions Contractors Harvesting TIFF or JP2 TIFF or JP2 HD & ftp TIFF or JP2 Normalises TIFF to JP2 Manual Automatic Jpylyzer validates JP2 Auto harvesting of JP2 & DMD Grey literature PDF Ingest Officer / Digital Curator Snagging Snagging
- 6. The approach • (Re)Use/develop existing systems were possible, e.g. bibliographic system Sierra, Preservica EE repository • Identify where new systems would be required, e.g. workflow middle ware • Take a practical approach & accept that it would be iterative learning as we go
- 7. The solution was to use Goobi
- 8. Why Goobi? • Dedicated to digitisation • Flexibility & process control • Adaptable & scalable • Vendor expertise/support http://www.inspirelancs.org.uk/interested-in-volunteering-family-carers-volunteers-wanted/
- 9. Role of Goobi • Role of Goobi is overall management & tracking of processes • Initiate ingest into our DAM Preservica • Reporting & statistics
- 10. Role of humans • Working at volume did not imply more staff, it implied efficiency • Also implied automation • Human work was focussed on tasks machines couldn't do http://planetivy.com/gaming/25273/natural-selection-2-gaming-evolution-in-action/
- 11. System & process design • High volume doesn’t imply use of many systems • Requires design to be as simple as possible, with as few moving parts as possible • Processes need to be efficient & scalable, human as well as system http://www.nivenswealthstrategies.com/keeping-it-simple/
- 12. Partnership for scalable digitisation • Relationship with Internet Archive digitising our Library content • High volume long term project • Content harvested from Internet Archive website & processed automatically • Dedicated Goobi process for fully automated harvesting
- 13. Harvesting from Internet Archive Content processed automatically, including creation of METS & ALTO. Goobi has a ‘repository’ of IA identifiers for searching/harvesting. Goobi harvests data from Internet Archive website. Content available in the player. Content stored in Preservica. DDS creates JSON for the player & pre- caches some content.
- 14. Challenges - M&Ms • Multi volume works • No metadata to support their union • Have to construct them manually, but process can be simplified • Time consuming, still to be fully automated
- 15. Challenges – Working with partners • Changes to Internet Archive website broke our harvesting • For automated ftp to work 3rd parties need to follow instructions • Creation of JPEG2000 images/video • Incorrect identifiers trips up processes
- 16. Opportunities • Working with IT, flexibility of virtualised environment • Working with Intranda, brings in vendor expertise • Distributed system brings in feedback from many users • Small team simplifies decision making • Success leads to success
- 17. Life cycle management • Good place with regard to life cycle management • Consistent processes based on common workflows • Goobi outputs consistent & predictable • Unified data set easier to manage in the future
- 18. Has automation been successful? • Yes with a but • Automation can be complex, easy to make mistakes • Automation requires metadata to be available • Automated processes still require a human minder
- 19. The scale of things
- 21. Lessons learned • Complexity Vs simplicity • Iterative approaches work but are time consuming • Vendor support/input crucial when starting from scratch • Process design essential
- 22. Be bold. Sometimes it’s the way we work that has to change
- 23. Thank you Questions now, questions later…? Dave Thompson, Digital Curator Wellcome Library d.thompson@wellcome.ac.uk @d_n_t http://wellcomelibrary.org/