The Improving Access to Text (IMPACT) project and other European initiatives
1 like1,220 views
The IMPACT project, supported by the European Community under the FP7 ICT Work Programme and coordinated by the National Library of the Netherlands, aims to improve mass digitization of historical printed texts through innovative OCR software and capacity building across Europe. It addresses the significant challenge faced by Europe, where only 1% of books in national libraries have been digitized, and focuses on enhancing OCR accuracy, collaboration, and the entire post-scanning workflow. The project involves a consortium of 15 partners, including libraries and universities, and has a total budget of EUR 15.5 million, with EUR 11.5 million funded by the EU.
The Improving Access to Text (IMPACT) project and other European initiatives
- 1. IMPACT is supported by the European Community under the FP7 ICT Work Programme. The project is coordinated by the National Library of the Netherlands. The Improving Access to Text (IMPACT) project and other European initiatives Michael Day UKOLN, University of Bath m.day@ukoln.ac.uk http://www.ukoln.ac.uk/ JISC Workshop: OCR for the Mass Digitisation of Textual Materials, University of Bath 24 September 2009
- 2. IMPACT is supported by the European Community under the FP7 ICT Work Programme. The project is coordinated by the National Library of the Netherlands. Presentation outline Contexts – Some European digitisation activity – Digitisation challenges The IMPACT project – The consortium and project structure – Major project activities JISC Workshop: OCR for the Mass Digitisation of Textual Materials, University of Bath, 24 September 2009 2
- 3. IMPACT is supported by the European Community under the FP7 ICT Work Programme. The project is coordinated by the National Library of the Netherlands. Digitisation activity in Europe (1) European Commission – i2010 digital libraries initiative Launched September 2005 Bringing together European cultural heritage online – Europeana portal Many projects dealing with the digitisation of texts in Europe – Many at large-scale, with selectivity at collection level or higher (industrial-scale mass digitisation) – Content holders often work with commercial providers (e.g., outsourcing of conversion processes, partnering with Google Books) – However, "Europe is facing a very important cultural and economic challenge: Only some 1% of the books in Europe's national libraries have been digitised so far, leaving an enormous task ahead of us" (Viviane Reding and Charlie McCreevy, EU Commissioners, September 2009) JISC Workshop: OCR for the Mass Digitisation of Textual Materials, University of Bath, 24 September 2009 3
- 4. IMPACT is supported by the European Community under the FP7 ICT Work Programme. The project is coordinated by the National Library of the Netherlands. Digitisation activity in Europe (2) Europeana - Europe's digital library – Website: http://europeana.eu/portal/ – Launched in November 2008 – Hosted by the National Library of the Netherlands; run by the European Digital Library Foundation – Part funded by the EU's eContent plus programme – A portal providing access to ca. 4.6 million items – Mixed content: Books, newspapers, photographs, maps, film clips Books included are mainly those in the public domain – EC public consultation on Europeana and the digitisation of books, open until 15 November 2009 http://ec.europa.eu/information_society/newsroom/cf/itemlongdetail.cf m?item_id=5181 JISC Workshop: OCR for the Mass Digitisation of Textual Materials, University of Bath, 24 September 2009 4
- 5. IMPACT is supported by the European Community under the FP7 ICT Work Programme. The project is coordinated by the National Library of the Netherlands. JISC Workshop: OCR for the Mass Digitisation of Textual Materials, University of Bath, 24 September 2009 5
- 6. IMPACT is supported by the European Community under the FP7 ICT Work Programme. The project is coordinated by the National Library of the Netherlands. JISC Workshop: OCR for the Mass Digitisation of Textual Materials, University of Bath, 24 September 2009 6
- 7. IMPACT is supported by the European Community under the FP7 ICT Work Programme. The project is coordinated by the National Library of the Netherlands. JISC Workshop: OCR for the Mass Digitisation of Textual Materials, University of Bath, 24 September 2009 7
- 8. IMPACT is supported by the European Community under the FP7 ICT Work Programme. The project is coordinated by the National Library of the Netherlands. Digitisation challenges (1) Large-scale digitisation – Mostly based on the image front searching technique (pioneered by projects like JSTOR) Scan physical item to create digital images of pages Subject those pages to OCR Combine OCR output with the images, OCR output considered good enough for searching, but any ambiguous results are able to be compared with page images “The strategy of linking page images with OCR enables us to make effective use of large corpora of relatively cheaply scanned books and was, in large measure, effective because it points backwards to the limitations of print: search gets human readers to the page and leaves them to parse out its meaning” (Many More than a Million seminar report, CLIR, November 2007: http://www.clir.org/activities/digitalscholar/Nov28final.pdf) JISC Workshop: OCR for the Mass Digitisation of Textual Materials, University of Bath, 24 September 2009 8
- 9. IMPACT is supported by the European Community under the FP7 ICT Work Programme. The project is coordinated by the National Library of the Netherlands. Digitisation challenges (2) Current generations of OCR tools do not always provide satisfactory results for historical documents – Main focus of tools is on modern documents – Not always fit for historic material with archaic fonts, obsolete characters, complex layouts, warped or degraded pages, language variation, etc. – Manual post-correction has a role, but is slow and expensive Example of OCR errors – From Australian Newspapers (National Library of Australia): http://newspapers.nla.gov.au/ – "The text in the left panel has been electronically translated by a computer. Computers are not as good at reading as humans, and often make mistakes“ – This system permits users to correct the OCR output – Article by Rose Holley in D-Lib Magazine, March/April 2009: http://www.dlib.org/dlib/march09/holley/03holley.html JISC Workshop: OCR for the Mass Digitisation of Textual Materials, University of Bath, 24 September 2009 9
- 10. IMPACT is supported by the European Community under the FP7 ICT Work Programme. The project is coordinated by the National Library of the Netherlands. JISC Workshop: OCR for the Mass Digitisation of Textual Materials, University of Bath, 24 September 2009 10
- 11. IMPACT is supported by the European Community under the FP7 ICT Work Programme. The project is coordinated by the National Library of the Netherlands. Extremely simplified text digitisation workflow Linguistic Named entity tools identification Access Printed package item Image Post correction processing + OCR OCR enhancement output Image Imaging Structural analysis and segmentation Metadata JISC Workshop: OCR for the Mass Digitisation of Textual Materials, University of Bath, 24 September 2009 11
- 12. IMPACT is supported by the European Community under the FP7 ICT Work Programme. The project is coordinated by the National Library of the Netherlands. The IMPACT project Research project funded by the European Commission – Large-scale Integrating Project – Funded from January 2008, for four years – Coordinated by the National Library of the Netherlands (KB) – Total budget: EUR 15.5M; EU funding: EUR 11.5M – Consortium of 15 partners Libraries Universities and research centres Industrial partners JISC Workshop: OCR for the Mass Digitisation of Textual Materials, University of Bath, 24 September 2009 12
- 13. IMPACT is supported by the European Community under the FP7 ICT Work Programme. The project is coordinated by the National Library of the Netherlands. The IMPACT consortium • Libraries • Universities and research – National Library of the centres Netherlands (coordinator) – Dutch Institute for Lexicology – The British Library – National Centre for Scientific – Bibliothèque nationale de France Research - Demokritos – German National Library – University of Salford – Bavarian State Library – University of Munich – Goettingen State and University – University of Innsbruck Library – University of Bath (UKOLN) – Austrian National Library – University of Innsbruck Library • Industrial partners – ABBYY – IBM Haifa Research Lab JISC Workshop: OCR for the Mass Digitisation of Textual Materials, University of Bath, 24 September 2009 13
- 14. IMPACT is supported by the European Community under the FP7 ICT Work Programme. The project is coordinated by the National Library of the Netherlands. IMPACT project objectives Aims to significantly improve the mass digitisation of historical printed text by – Innovating OCR software and language technology – Sharing expertise and building capacity across Europe – Ensuring that tools and services will be sustained after the end of the project Specific principles: – Reduce effort and enhance speed and results of mass digitisation (speed and scalability) – Focus on the whole post scanning workflow: image processing, OCR processing (including dictionaries), OCR correction, and document formatting – All research and development to be grounded in the needs of libraries – Working with other centres of competence JISC Workshop: OCR for the Mass Digitisation of Textual Materials, University of Bath, 24 September 2009 14
- 15. IMPACT is supported by the European Community under the FP7 ICT Work Programme. The project is coordinated by the National Library of the Netherlands. IMPACT project approach (1) Project structure – 22 work packages Four sub-projects – Technical and research based: TR (Text Recognition) focused on the extraction of text in a digital form from an image (OCR) EE (Enhancement and Enrichment) using linguistic technologies to make the results of full-text digitisation more accurate and accessible – Strategic: OC (Operational Context) guiding the direction of the project from the libraries' perspective CB (Capacity Building) stimulating the uptake of results in the museums, libraries and archives communities JISC Workshop: OCR for the Mass Digitisation of Textual Materials, University of Bath, 24 September 2009 15
- 16. IMPACT is supported by the European Community under the FP7 ICT Work Programme. The project is coordinated by the National Library of the Netherlands. IMPACT project approach (2) Operational Context Requirements, Benchmarking and Metrics Best Practices and Guidelines Technical Framework and Interoperability Text Recognition Enhancement and Enrichment Pre-processing and segmentation Collaborative Correction Adaptive and experimental OCR Historical Lexica Models and dictionaries Structural Metadata Capacity Building Packaging of resources Training and support Demonstration JISC Workshop: OCR for the Mass Digitisation of Textual Materials, University of Bath, 24 September 2009 16
- 17. IMPACT is supported by the European Community under the FP7 ICT Work Programme. The project is coordinated by the National Library of the Netherlands. IMPACT tools and services (1) Text Recognition – Technologies for supporting the extraction of text from the page – Adaptive OCR engine, integrating: Image enhancement toolkit Segmentation toolkit Post-correction modules Other OCR engines – Experimental prototypes Typewritten OCR Wordspotting Inventory extraction JISC Workshop: OCR for the Mass Digitisation of Textual Materials, University of Bath, 24 September 2009 17
- 18. IMPACT is supported by the European Community under the FP7 ICT Work Programme. The project is coordinated by the National Library of the Netherlands. IMPACT tools and services (2) Enhancement and enrichment – Focus on making OCR results more accurate and accessible – Collaborative correction Web based, linked to OCR engine – Tools and content General and named entities lexica for Dutch, German and English, general support for lexicon building in other languages Dealing with historical languages Collaborative environments for managing named entities – Structural metadata Functional Extension Parser, for the automatic detection and tagging of structural metadata of scanned material JISC Workshop: OCR for the Mass Digitisation of Textual Materials, University of Bath, 24 September 2009 18
- 19. IMPACT is supported by the European Community under the FP7 ICT Work Programme. The project is coordinated by the National Library of the Netherlands. IMPACT tools and services (3) Strategic tools and services – Website (http://www.impact-project.eu/) – Decision support tools, to support the initiation, organisation, management of mass-digitisation projects – A set of learning resources providing guidance on the digitisation of texts and the implementation of project tools – Training and support Helpdesk Training programme (events) – Demonstration of the tools (case studies) – IMPACT Centre of Competence JISC Workshop: OCR for the Mass Digitisation of Textual Materials, University of Bath, 24 September 2009 19
- 20. IMPACT is supported by the European Community under the FP7 ICT Work Programme. The project is coordinated by the National Library of the Netherlands. JISC Workshop: OCR for the Mass Digitisation of Textual Materials, University of Bath, 24 September 2009 20
- 21. IMPACT is supported by the European Community under the FP7 ICT Work Programme. The project is coordinated by the National Library of the Netherlands. Thank you for your attention! Any questions? Additional information: – The IMPACT project: Website: http://www.impact-project.eu/ Project office: impact@kb.nl – Europeana: http://www.europeana.eu/portal/ – Workshop Materials: http://www.ukoln.ac.uk/events/ocr-2009/ JISC Workshop: OCR for the Mass Digitisation of Textual Materials, University of Bath, 24 September 2009 21