InterPlanetary Wayback: Peer-To-Peer Permanence of Web Archives
3 likes1,652 views
This document proposes InterPlanetary Wayback (IPWB), a system to archive web pages using the InterPlanetary File System (IPFS) in a peer-to-peer manner. It extracts HTTP headers and payloads from WARC files and stores them as objects in IPFS, generating corresponding CDXJ indexes. Pages can then be replayed by dereferencing the CDXJ locators through IPFS. While a proof of concept, IPWB has potential to solve archival permanence through distributed storage, though performance requires further optimization for large-scale use.
InterPlanetary Wayback: Peer-To-Peer Permanence of Web Archives
- 1. InterPlanetary Wayback Peer-to-Peer Permanence of Web Archives Mat Kelly, Sawood Alam, Michael L. Nelson, Michele C. Weigle Old Dominion University Web Science and Digital Libraries Research Group Norfolk, Virginia, USA @WebSciDL TPDL 2016 Hannover, Germany September 7, 2016 http://github.com/oduwsdl/ipwb
- 2. Background - IPFS ● Hypermedia distributed protocol ● IPFS entity hashes are content addressed ○ Content changes → different hash produced ○ Inherent potential for de-duplication of content ● Files accessible via HTTP: http://ipfs.io/<hash> ● Built on trust chains for provenance
- 3. Content addressing http://foo.com/spaceDog.jpg http://example.org/yuri.jpg QmZAD4xeeNeYF3TmwWgBXypLKTiCGwGRMXHW7MtheWKtw4 QmZAD4xeeNeYF3TmwWgBXypLKTiCGwGRMXHW7MtheWKtw4 === $ ipfs cat QmZAD4xeeNeYF3TmwWgBXypLKTiCGwGRMXHW7MtheWKtw4 > doge.jpg
- 4. Background - WARC WARC response record HTTP resp header HTTP resp payload Warc-response header WARCs also contain: ● HTTP requests ● warc-info ● warc-metadata records ● etc. uses only warc-response records
- 5. Background - Wayback Archival Indexer Archival Index (e.g., CDXJ) Replay Engine processes outputs reads (file, offset) read archived content Present WARC content to user
- 6. Motivation ● Persistence of archived web data dependent on resilience of organization and availability of data ● Remove massive redundancy in web archive files of exact duplicate content ● Determine feasibility of pushing WARCs into IPFS
- 7. Indexing
- 8. WARC Creation HTTP Header & Payload Extraction Push to IPFS Generate CDXJ WARC-CDXJ correspondence
- 9. HTTP HEADER BLOCK HTTP PAYLOAD BLOCK WARC Creation HTTP Header & Payload Extraction Push to IPFS Generate CDXJ WARC-CDXJ correspondence
- 10. QmcN9eWwRF73dZj5BgT4x8jeEcFr xurX1hot8QwCbMi9PB Qmczh9YnB4U1ptPeqxcaTZA4aMm uNUswTLTWzXntvbp9sL HEADER DIGEST PAYLOAD DIGEST HTTP HEADER BLOCK HTTP PAYLOAD BLOCK WARC Creation HTTP Header & Payload Extraction Push to IPFS Generate CDXJ WARC-CDXJ correspondence
- 11. QmcN9eWwRF73dZj5BgT4x8jeEcFrxurX1hot8QwCb Mi9PB Qmczh9YnB4U1ptPeqxcaTZA4aMmuNUswTLTWzX ntvbp9sL HEADER DIGEST PAYLOAD DIGEST ipwb.example.com)/ 20160905022013 { "locator":"urn:ipfs/QmcN9eWwRF73dZj5BgT4x8jeEcFrxurX1hot8QwCbMi9PB/Qmczh9YnB4U1ptPeqxca TZA4aMmuNUswTLTWzXntvbp9sL", "mime_type": "text/html", "status_code": 200, “other_fields”: “other values...” } CDXJ: http://ws-dl.blogspot.com/2015/09/2015-09-10-cdxj-object-resource-stream.html WARC Creation HTTP Header & Payload Extraction Push to IPFS Generate CDXJ Record WARC-CDXJ correspondence
- 12. ipwb.example.com)/ 20160905022013 {"locator": "urn:ipfs/QmcN9eWwRF73dZj5BgT4x8jeEcFrxurX1hot8QwCbMi9PB/ Qmczh9YnB4U1ptPeqxcaTZA4aMmuNUswTLTWzXntvbp9sL", "mime_type": "text/html", "status_code": "200"} ipwb.example.com)/style.css 20160905022013 {"locator": "urn:ipfs/QmU1k71bT6ibZBSdxBL35cQXwovTih8cTB4CXfrjyMfZxE/Q mbvUAo9U31wSdvARjvbPeVBTAwCjN1kyPhQ4ho3n8TAZo", "mime_type": "text/css", "status_code": "200"} ipwb.example.com)/ipwb.png 20160905022013 {"locator": "urn:ipfs/QmTjfMxFGvbP4nwFoq3tNYDPW6gC99i5njrqsXSw6QRvHa/ QmYMKZbnk53kuPJirahJHGevCCy2afLyePRdX38TukFUwd", "mime_type": "image/png", "status_code": "200"} ipwb.example.com)/fileduration.png 20160905022013 {"locator": "urn:ipfs/QmaCj6LNngxwqxaLmfp1xCyxcwDt2Uzqf8gCG6bVyQppYC/ QmdgtMcGprTF8bqv7ytgMwtoi5BhRxfuvBjD6Vj2U7ohz1", "mime_type": "image/png", "status_code": "200"} ipwb.example.com)/filesize.png 20160905022013 {"locator": "urn:ipfs/QmNPjrSVY31oGDooMiA18ZDNHfkLnEg3j5gRj1dFdrqmS4/ Qmb4heB8PU58nkWt6w5tBgMfpeLTKuU7iuxg9tFdoPsF1B", "mime_type": "image/png", "status_code": "200"} WARC Creation HTTP Header & Payload Extraction Push to IPFS Generate CDXJ WARC-CDXJ correspondence
- 13. Replay
- 14. ipwb.example.com)/ 20160905022013 {"locator": "urn:ipfs/QmcN9eWwRF73dZj5BgT4x8jeEcFrxurX1 hot8QwCbMi9PB/Qmczh9YnB4U1ptPeqxcaTZA4a MmuNUswTLTWzXntvbp9sL", "mime_type": "text/html", "status_code": "200"} ipwb.example.com)/style.css 20160905022013 {"locator": "urn:ipfs/QmU1k71bT6ibZBSdxBL35cQXwovTih8cT B4CXfrjyMfZxE/QmbvUAo9U31wSdvARjvbPeVBTA wCjN1kyPhQ4ho3n8TAZo", "mime_type": "text/css", "status_code": "200"} ipwb.example.com)/ipwb.png 20160905022013 {"locator": "urn:ipfs/QmTjfMxFGvbP4nwFoq3tNYDPW6gC99i5 njrqsXSw6QRvHa/QmYMKZbnk53kuPJirahJHGevC Cy2afLyePRdX38TukFUwd", "mime_type": "image/png", "status_code": "200"} ... http://ipwb.example.com Replay reference via CDXJ Dereference via IPFS Reconstruction from IPFS
- 15. ipwb.example.com)/ 20160905022013 {"locator": "urn:ipfs/ QmcN9eWwRF73dZj5BgT4x8jeEcFrxurX1hot8QwCbMi9PB/Qmczh9YnB4U1ptPeqxcaTZA4aMmuNUswTLTWzXntvbp9sL", "mime_type": "text/html", "status_code": "200"} ... HTTP HEADER BLOCK HTTP PAYLOAD BLOCK Replay reference via CDXJ Dereference via IPFS Reconstruction from IPFS
- 16. HTTP HEADER BLOCK HTTP PAYLOAD BLOCK Reconstruct Replay reference via CDXJ Dereference via IPFS Reconstruction from IPFS
- 17. Data Flow
- 18. Evaluation
- 19. ● Reported IPFS slowness https://github.com/ipfs/go-ipfs/issues/1216 ○ Has since been fixed, subsequent to IPWB-TPDL 570 files per minute~10% overhead
- 20. Replay Time ● 600 requests in 222 seconds ● Slower than PyWB (which took 5.26 seconds) ● File vs. rich object based retrieval ● Never expiring cache
- 21. Future Works ● Evaluate the improved IPFS on large dataset ● Evaluate deduplication ● Implement an index-free collaborative archiving system ● Utilize IPNS to reference URI-Rs
- 22. Conclusions ● A proof of concept system to leverage a novel approach to archiving and retrieval ● Evaluated storage and time costs and qualitative analysis ● It can only work for small archives in it’s current state ● A path to answer “who will archive the archives?” question
- 23. InterPlanetary Wayback Peer-to-Peer Permanence of Web Archives @WebSciDL http://github.com/oduwsdl/ipwb Support: NSF #1624067 via the Archives Unleashed Hackathon Mat Kelly, Sawood Alam, Michael L. Nelson, Michele C. Weigle