Archiving Web-Based #musetech for Institutional Memory
Download as PPTX, PDF0 likes64 views
The document discusses the importance of web archiving for museums, detailing methods for collecting and preserving online content such as websites and social media. It outlines the various types of web archiving techniques and tools available and emphasizes the need for archiving to meet institutional memory requirements and document online events. A case study of the Indianapolis Museum of Art illustrates the web archiving process and offers guidelines for creating preservable websites.
Archiving Web-Based #musetech for Institutional Memory
- 1. Out with the Old: Archiving Web-Based #musetech for Institutional Memory SAMANTHA NORLING, DIGITAL COLLECTIONS MANAGER, NEWFIELDS NOVEMBER 19, 2018
- 2. 2 Out with the Old? Archiving Web-Based #musetech for Institutional Memory Samantha Norling Digital Collections Manager, Newfields Museum Computer Network, November 15, 2018
- 3. 3 What is Web Archiving? Web archiving is the process of collecting, preserving, and enabling access to content on the Web, such as: • Websites • Blogs • Social Media pages For museums, web-based content may include: • Exhibition-related websites • Gallery interactives • Tour applications …all of which tend to be time-based.
- 4. 4 Why Should Museums Archive the Web? • Meet records retention requirements or capture important institutional content as part of an archives program • Document online events and/or public responses to the museum and its activities • Collect web-native resources within your collection scope* • Combat link rot * Though this is not the focus of this presentation, there are many interesting projects being led by museums to collect this material.
- 5. 5 Who Currently Archives the Web? Source: Web Archiving: A 2016 Survey (National Digital Stewardship Alliance)
- 6. 6 Source: Web Archiving: A 2016 Survey (National Digital Stewardship Alliance) What is being Archived?
- 7. 7 Types of Web Archiving • Client-side • Archiving any site that is freely- available on the web • Crawlers start from a “seed” page and navigate through hyperlinks, based on parameters set • Capture documents, text pages, data, images, audio and video files • Can be blocked by robots.txt exclusions on websites Source: Web Archiving Guidance, National Archives UK (2011) http://www.nationalarchives.gov.uk/documents/information-management/web-archiving-guidance.pdf
- 8. 8 Types of Web Archiving • Transaction-based • Requires access to the web server • Records transactions between users of a site and the server • Content that is never viewed will never be archived • Record exactly what was seen and when Source: Web Archiving Guidance, National Archives UK (2011) http://www.nationalarchives.gov.uk/documents/information-management/web-archiving-guidance.pdf
- 9. 9 Types of Web Archiving • Server-side • Direct copy of files from web server • Challenge to make the content usable as a navigable archived site • Dynamically-generated content difficult to recreate • Faithful reproduction requires replication of the environment in which the live site was run • Benefit: archive parts of sites that are inaccessible to web crawlers Source: Web Archiving Guidance, National Archives UK (2011) http://www.nationalarchives.gov.uk/documents/information-management/web-archiving-guidance.pdf
- 10. 10 Client-side web archiving in a (super simplified) nutshell:
- 11. 11 THE WAYBACK MACHINE WEB ARCHIVING TOOLS AND SERVICES https://archive.org/web
- 12. 12 ARCHIVE-IT WEB ARCHIVING TOOLS AND SERVICES https://archive-it.org
- 14. 14 Source: Web Archiving: A 2016 Survey (National Digital Stewardship Alliance) Local Web-Archiving Tool Usage
- 15. 15 WEB CRAWLERS • HTTrack (file directory directory) https://www.httrack.com/ • Heretrix https://github.com/internetarchive/heritrix3 • Warcprox http://github.com/internetarchive/warcprox • wget https://www.petekeen.net/archiving-websites-with- wget WEB ARCHIVING TOOLS AND SERVICES
- 16. 16 WEB ARCHIVE PLAYERS • OpenWayback https://github.com/iipc/openwayback/wiki • pywb (Python Wayback) https://github.com/webrecorder/pywb • oldweb.today http://oldweb.today/ • Webrecorder Player https://github.com/webrecorder/webrecorder-player WEB ARCHIVING TOOLS AND SERVICES
- 17. 17 WEBRECORDE R WEB ARCHIVING TOOLS AND SERVICES https://webrecorder.io (online) https://github.com/webrecorder/webrecorder (local)
- 18. 18 CASE STUDY INDIANAPOLIS MUSEUM OF ART AT NEWFIELDS WEB ARCHIVES
- 19. 19 SETTING THE STAGE A PLACE FOR NATURE & THE ARTS Sutphin Mall at Newfields,with Robert Indiana’s LOVE and Roy Lichtenstein’s Five Brushstrokes.
- 20. 20 imamuseum.org
- 21. 21 Site Spider, Mark II
- 22. 22 RESULTS • 13,663 pages total • 7,327 IMA Blog pages • 2,003 JPG images • 455 PDF files • 3 mp3 recordings • 828 HTTP Status 404 • 2 The Island residency blogs • Exhibition microsites Site Spider, Mark II
- 26. 26 WEBRECORDE R WEB ARCHIVING TOOLS AND SERVICES https://webrecorder.io (online) https://github.com/webrecorder/webrecorder (local)
- 28. 28 Retroactive Web Archiving – IMA Archives
- 29. 29 Guidelines for Preservable Websites • Provide a standard link to all website content • Avoid proprietary formats for important content, especially the home page • Include a user and/or site map • Omit robots.txt exclusions, or limit them to areas of the site not needed for archiving • Maintain stable URLs and redirect when necessary • Correctly identify character set encoding Source: Columbia University Libraries
- 30. 30 Samantha Norling Digital Collections Manager snorling@discovernewfields.org @SamiNorling Contact Information
Editor's Notes
- #3: Set the stage for the presentation here. Introduction, context (not an expert in web archiving, but a former archivist who found myself in a position to need to react quickly in anticipation of a major loss with the retirement of the previous site). Following the successful archiving of that site, now pursuing an expanded web archiving program, primarily of institutionally-created content, both current and past, but also including some external content.
- #5: Essentially, web archiving can help you to maintain a reliable point of access to your most important records.
- #8: The robots exclusion standard is a tool used by a webmaster to direct a web crawler not to crawl all or specified parts of their website. The webmaster places their request in the form of a robots.txt file that is easily found on their website (ex. example.com/robots.txt). Archive-It (like Google and most other search engines) uses a robot to crawl and archive web pages.
- #9: The robots exclusion standard is a tool used by a webmaster to direct a web crawler not to crawl all or specified parts of their website. The webmaster places their request in the form of a robots.txt file that is easily found on their website (ex. example.com/robots.txt). Archive-It (like Google and most other search engines) uses a robot to crawl and archive web pages.
- #10: The robots exclusion standard is a tool used by a webmaster to direct a web crawler not to crawl all or specified parts of their website. The webmaster places their request in the form of a robots.txt file that is easily found on their website (ex. example.com/robots.txt). Archive-It (like Google and most other search engines) uses a robot to crawl and archive web pages.
- #11: A web crawler (also known as a web spider or webrobot) is a program or automated script which browses the World Wide Web in a methodical, automated manner, harvesting content from web servers. This process is called Web crawling or spidering. Many legitimate sites, in particular search engines, use spidering as a means of providing up-to-date data. Crawlers used for web archiving navigate through live websites and download their source code into the WARC format (web archive), which provides an archival snapshot of the live website at the time of the crawl. “Replay technologies” allow for viewing of WARC files. (Graphic shows going from WARC to live website).
- #12: Launched in 2001, major collecting activity of the Internet Archive. 341 Billion web pages saved over time – 25 petabytes.
- #13: Find image to represent Archive-It. Launched in 2006, Archive-It is a subscription-based web archiving service with over 400 partner (subscriber) organizations. This includes the New York Art Resources Consortium (NYARC)
- #14: NYARC - The New York Art Resources Consortium (NYARC) consists of the research libraries and archives of three leading art museums in New York City: The Brooklyn Museum, The Frick Collection, and The Museum of Modern Art. With funding from The Andrew W. Mellon Foundation, NYARC was formed in 2006 to facilitate collaboration that results in enhanced resources to research communities.
- #16: List: HTTrack, Heritrix, warcprox, wget
- #17: List: Webrecorder Player, Wayback Machine, OpenWayback, pywb (Python Wayback), oldweb.today (check that all of these are still active, and provide URLs)
- #20: Tascha and Sami introductions.
- #21: Known content on imamuseum.org included the main site and its sub-pages, as well as the IMA Blog (active 2007 – 2014)
- #23: Find image to represent warc players. List: Webrecorder Player, Wayback Machine, OpenWayback, pywb (Python Wayback), oldweb.today (check that all of these are still active, and provide URLs)
- #30: https://library.columbia.edu/bts/web_resources_collection/guidelines_for_preservable_websites.html