Building Web Archiving Technology, Together
- 1. Building Web Archiving Technology, Together Nicholas Taylor Web Archiving Service Manager Stanford University Libraries Web Archives 2015: Capture, Curate, Analyze November 13, 2015
- 2. overview • why build together? • community for collaborative work • APIs for collaborative work “LAX on take off” by Doug under CC BY-NC-ND 2.0
- 3. not a programmer “Bug” by Randall Munroe under CC BY-NC 2.5
- 4. aspiring OSS contributor GitHub: “nullhandle (Nicholas Taylor)”
- 5. studying the landscape “2010 Grand Canyon Celebration of Art 172” by Grand Canyon National Park under CC BY 2.0
- 6. a centralized enterprise 60% 25% 14% 63% 20% 16% 0% 10% 20% 30% 40% 50% 60% 70% External Local Both 2011 2013 NDSA: “Web Archiving in the U.S.: A 2013 Survey”
- 7. a centralized enterprise 0 0 1 0 2 0 1 0 1 0 3 3 1 2 4 2 6 4 1 0 2 0 0 1 1 0 1 3 5 3 4 2 2 5 6 15 0 2 4 6 8 10 12 14 16 18 20 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 Number of organizations Archive-It Partner as of 2013 NDSA: “Web Archiving in the U.S.: A 2013 Survey”
- 8. minimal local preservation 19% 81% 20% 80% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% Transferred Haven't transferred 2011 2013 NDSA: “Web Archiving in the U.S.: A 2013 Survey”
- 9. evolving web “Light Writing - Spider Web” by oz dean under CC BY-ND 2.0
- 10. opportunities for preservation “standing out” by kenda bustami under CC BY 2.0
- 11. opportunities for research “Exploring the Canadian Political Interest Group and Political Parties Web Sphere” by Ian Milligan under Standard YouTube License
- 12. not the only one HUL: “Web Archiving Environmental Scan Home” CDL: “Announcing a New Partnership”
- 13. a response
- 14. COMMUNITY “Why we love Peckham, P1020468crop” by Eye magazine under CC BY-NC-SA 2.0
- 15. community analysis SAA Web Archiving Roundtabl e Archive-It Partners IIPC NDSA: “Web Archiving in the U.S.: A 2013 Survey”
- 16. Archive-It Archive-It: “Archive-It 5.0 Feature Requests”
- 17. IIPC Open HUB: “Open Wayback”
- 18. models of software production (irrespective of license) • sole source – single developer • closed source – team/corporate dev; no outside contributions • club source – pool resources for solo/team/corporate dev • community source – direct and distributed community participation • open source – grassroots, democratic, meritocratic participation Tom Cramer: “Collaborative Open Source Software Production & APIs”
- 19. club source examples • Archivematica, AtoM (Artefactual) • ArchivesSpace (Lyrasis) • Bitcurator (Educopia) • Fedora (DuraSpace) • JHOVE (OPF) • LOCKSS (Stanford University) • Omeka (George Mason University)
- 21. community architecture • privileges community over code • recognizes distribution of investment • embraces community diversity • models open processes and governance • encourages varied contributions • serves community needs
- 22. STANDARDS “P1050827” by Rebecca Siegel under CC BY 2.0
- 23. success of a standard • capture: DeDuplicator, Heritrix, python- heritrix, SiteStory, WAIL, WARCreate, WarcMITMProxy, WarcProxy, Webrecorder, wget, Wpull • access: OpenWayback, pywb, warc-proxy, WarcManager, Wayback Machine, Web Archive Discovery, WebArchivePlayer • utilities: JHOVE2, JWAT, Megawarc, pylibwarc, WARCAT, Warcbase, warctools, Web Archive Commons
- 24. web archiving lifecycle Internet Archive: “The Web Archiving Life Cycle Model”
- 25. missed opportunities? Appraisal and Selection Scoping Data Capture Storage and Organization QA and Analysis Metadata / Description Access / Use / Reuse Preservation Risk Management ACT Archive-It AtN BCWeb CDL WAS DigiBoard Islandora WARC Solution Pack Netarchive Suite PageFreezer UNT Nomination Tool WCT
- 26. smaller, modular components “Giant Rubik's Cube” by Francois Lamotte under CC BY 2.0
- 27. smaller projects do better small projects (<$1 million) large projects (>$10 million) on time/budget challenged failedon time/budget challenged failed Standish Group: “Chaos Manifesto 2013: Thing Big, Act Small”
- 28. IIPC community interest in APIs contribution type % of respondents # of respondents help define functional requirements 94% 15 contribute use cases 81% 13 help define technical details 69% 11 help schedule and run meetings 19% 3 implement and test 6% 1 Andrea Goethals: “Results of the Web Archiving API Survey of IIPC Members”
- 29. API candidates • capture tool/proxy interconnect • capture tool management • data import/export • query + extraction • integrity audit + repair • descriptive metadata • logs + analytics • renderings/derivative formats • federated data delivery • federated replay • federated full-text search
- 30. let’s combine forces “Stages of flow” by Peter Thoeny under CC BY-NC-SA 2.0