![Top-Level Links Over Time [here]
www.bl.uk 11](https://image.slidesharecdn.com/tdm-2014-11-27-141127060742-conversion-gate02/85/Digging-into-the-Web-Archive-at-the-British-Library-2014-11-27-11-320.jpg)
Digging into the Web Archive at the British Library 2014-11-27
- 1. Digging into the Web Archive at the British Library Andrew Jackson UK Web Archive Technical Lead
- 2. Collections & Scale • Three collections: – By permission (2004-2013) • c. 200 million URLs – Legal Deposit (2013 onwards) • c. 2 billion URLs/year (30TB/y) – JISC/IA Historical (1996-2013) • c. 6 billion URLs (57TB) • Use data-mining to support: – Access – Search – Preservation – Web science www.bl.uk 2
- 4. Web Archive Architecture www.bl.uk 4
- 5. Search & Analytical Access • ‘Title-level’ search: – Millions of homepages found via metadata • Full-text search: – Billions of resources – Dedicated faceted search service • Analytical access: – Combine faceted full-text search with: • Trend analysis • Visualisation tools – Working with modern historians to drive development www.bl.uk 5
- 6. Longitudinal Analysis (Prime Ministers) www.bl.uk 6
- 8. Secondary Datasets • Facts about content, including: – Crawl index – Geo-index – Format profiles – Link graphs • Facilitate independent research • Can be made available under CC0 • Hosted at http://data.webarchive.org.uk/opendata/ www.bl.uk 8
- 9. Exploring Links Between Hosts www.bl.uk Courtesy of Peter Webster, Rainer Simon and Jules Mataly 9
- 10. Links From 1996 www.bl.uk 10
- 11. Top-Level Links Over Time [here] www.bl.uk 11
- 12. Access Service Spectrum • Single-item retrieval • ‘Title-level’ search • Full-text search • Analytics & visualisation (at full scale) • Secondary datasets • Remote analysis of datasets (an API, e.g. SPARQL) • Full computational access service (internal only right now) • Not just the web archive? www.bl.uk 12
- 13. Thank you! Email: Andrew.Jackson@bl.uk Twitter: @anjacks0n UK Web Archive: http://www.webarchive.org.uk Blog: http://britishlibrary.typepad.co.uk/webarc hive/ Twitter: @ukwebarchive www.bl.uk 13
Editor's Notes
- #2: 20 min presentation on - wa activities and status - Usage etc. stats in talk for LDLs. - inc. research projects for sept 19th, should also include document harvest as they are expecting to take that on too (CEDAR?).
- #5: Well, it’s just a bunch of files.
- #8: Supporting research has helped us improve how we crawl material. Also helps us understand the collection, so we can take care of it. And furthermore, helps us understand how to leverage this collection.
- #10: Paper accepted for WebSci’14: Mapping the UK Webspace: Fifteen Years of British Universities on the Web MA thesis by Jules Mataly: The Three Truths of Margaret Thatcher: Creating and Analysing Link graph mostly explored secondary dataset 1996 visualisation of the UK web by Rainer Simon Peter Wester’s analysis and visualisation of hosts within the dataset over time link to Rowan Williams’ website as archbishop of Canterbury, and how this relates to the public reaction to his lecture in February 2008 on the interaction between English family law and Islamic shari’a law Student from HK