MementoMap: A Web Archive Profiling Framework for Efficient Memento Routing

Sawood Alam <@ibnesayeed>
Advisor: Michael L. Nelson
Members: Michele C. Weigle, Jian Wu, Sampath Jayarathna, and Erika F. Frydenlund
MementoMap
A Web Archive Profiling Framework
for Efficient Memento Routing
Doctoral Dissertation Defense, December 04, 2020
Old Dominion University, Norfolk, Virginia - 23529 (USA)

@ibnesayeed | @WebSciDL
● Introduction and Motivation
● Research Questions
● MementoMap Framework
● RQ1: Understanding Web Archives
○ Archival Holdings
○ Archival Voids
● RQ2: Serialization and Dissemination
● RQ3: Memento Routing
● Contributions, Future Work, and Conclusions
Outline
2

@ibnesayeed | @WebSciDL 4
Homepage of the Smithsonian Institution (SI)

An Early Memento of SI in Wayback Machine
https://web.archive.org/web/19971210203441/http://www.si.edu/newstart.htm
A missing image

The Earliest Memento of SI in Arquivo.pt
https://arquivo.pt/wayback/19961013204418/http://www.si.edu/

List of SI Mementos in the Two Web Archives
https://web.archive.org/web/*/http://si.edu/ https://arquivo.pt/wayback/*/http://si.edu/

List of SI Mementos From an Aggregator
http://timetravel.mementoweb.org/list/19950101000000/http://si.edu/
Mementos from 7
different web archives.
Arquivo.pt has the
first memento.

$ curl -s https://web.archive.org/web/timemap/link/http://si.edu/
<http://www.si.edu:80/>; rel="original",
<https://web.archive.org/web/http://si.edu/>; rel="timegate",
<https://web.archive.org/web/timemap/link/http://si.edu/>;
rel="self"; type="application/link-format"; from="Fri, 02 May 1997 11:07:51 GMT",
<https://web.archive.org/web/19970502110751/http://www.si.edu:80/>;
rel="first memento"; datetime="Fri, 02 May 1997 11:07:51 GMT",
rel="memento"; datetime="Fri, 02 May 1997 11:07:51 GMT",
rel="memento"; datetime="Mon, 28 Jul 1997 07:58:21 GMT",
rel="memento"; datetime="Wed, 10 Dec 1997 20:36:35 GMT",
[...TRUNCATED...]
TimeMap of SI From Wayback Machine
Original URI (URI-R)
Memento URI (URI-M)

$ curl https://arquivo.pt/wayback/timemap/link/http://si.edu/
<https://arquivo.pt/wayback/timemap/link/http://si.edu/>;
rel="self"; type="application/link-format"; from="Sun, 13 Oct 1996 20:44:18 GMT",
<https://arquivo.pt/wayback/http://si.edu/>; rel="timegate",
<http://si.edu/>; rel="original",
<https://arquivo.pt/wayback/19961013204418mp_/http://www.si.edu/>;
rel="memento"; datetime="Sun, 13 Oct 1996 20:44:18 GMT"; collection="$root",
rel="memento"; datetime="Sat, 25 Oct 2008 15:15:19 GMT"; collection="$root",
rel="memento"; datetime="Thu, 16 Jul 2009 05:32:58 GMT"; collection="$root",
rel="memento"; datetime="Wed, 14 Oct 2009 12:15:40 GMT"; collection="$root",
rel="memento"; datetime="Sat, 29 May 2010 16:54:54 GMT"; collection="$root",
[...TRUNCATED...]
TimeMap of SI From Arquivo.pt

$ curl https://memgator.cs.odu.edu/timemap/link/http://si.edu/
<http://si.edu/>; rel="original",
<https://memgator.cs.odu.edu/timemap/link/http://si.edu/>;
rel="self"; type="application/link-format",
rel="first memento"; datetime="Sun, 13 Oct 1996 20:44:18 GMT",
<https://webarchive.loc.gov/all/19970502110751/http://www.si.edu/>;
<https://wayback.archive-it.org/all/19970502110751/http://www.si.edu/>;
[...TRUNCATED...]
TimeMap of SI From a Memento Aggregator

$ memgator -f cdxj http://si.edu/ | grep -v "^!" | cut -d'/' -f3 | sort | uniq -c | sort -nr
13263 web.archive.org
3590 wayback.archive-it.org
1202 web.archive.bibalex.org
651 webarchive.loc.gov
321 arquivo.pt
32 wayback.vefsafn.is
11 web.archive.org.au
3 archive.is
1 www.webarchive.org.uk
1 swap.stanford.edu
1 perma.cc
$ memgator -f cdxj http://odu.edu/ | grep -v "^!" | cut -d'/' -f3 | sort | uniq -c | sort -nr
3071 web.archive.org
796 wayback.archive-it.org
751 web.archive.bibalex.org
99 webarchive.loc.gov
26 arquivo.pt
2 archive.is
1 wayback.vefsafn.is
Cross-Archive Memento Lookup With MemGator
Although there are
13k+ mementos in IA,
there are also
mementos in 10 other
public web archives.
https://github.com/oduwsdl/MemGator
ODU is less popular, but
there are mementos in 7
different web archives.

Who Would Have Thought to Lookup in the
Icelandic Web Archive for odu.edu Mementos?
13
http://wayback.vefsafn.is/wayback/20100810032449/http://odu.edu/

Why Aggregate Small Archives?
● One trillion+ mementos in IA’s Wayback Machine do not
cover everything
● Archives often have unique mementos (small overlap)
● Linguistic and geolocation diversity
● High-quality curated collections
● Restricted resources and private archives
14

MemGator Broadcasting
15

16

17

18

19

20

MemGator Log Responses From Various Archives
21
93% of the requests
made from MemGator
to upstream archives
were wasteful.
Only about one third
of the requests to the
largest web archive
(IA) were a hit.

Aggregation Is Great, But Broadcasting Is Wasteful
22
What do we want? Aggregate all archives, large or small
What’s the problem? Broadcasting is wasteful and problematic
What’s the solution? Selectively poll archives that are likely to
return good results for a lookup URI
How to identify those? Profile web archives
How to profile archives? MementoMap Framework

Memento Lookup Routing
23
Let us fix the broadcasting issue
with a more informed routing.

Archive Profiling Strategies
● Complete URI-R Profiling (1 URI-R = 1 Profile Key) [Sanderson et al., TPDL 2012]
○ bbc.co.uk/images/logo.png?w=90
○ cnn.com/2014/03/15/?id=128734
● TLD-Only Profiling (1 TLD = 1 Profile Key) [AlSum, et al., TPDL 2013]
○ *.com
○ *.uk
● Middle Ground
○ *.cnn.com
○ *.co.uk
○ *.bbc.co.uk
○ bbc.co.uk/images/*
24
We explore
these strategies
in this work.
Top three archives after
IA produce full TimeMaps
52% of the time.

Related Work
25
Archive Profiling
● Sanderson et al., IIPC 2012
● AlSum et al., IJDL 2014
● Bornand et al., JCDL 2016
● Klein et al., JCDL 2019
Query Routing
● Gravano et al., SIGMOD 1997
● Callan et al., SIGMOD 1999
● Lu et al., CIKM 2003
● Meng et al., CSUR 2002
Bloom Filters
● Bloom, CACM 1970
● Majkowski, Cloudflare 2020
● Broder et al., Internet
Mathematics 2003
Web Archive Searching
● Gomes et al., TempWeb 2013
● Costa et al., TempWeb 2013
● Kanhabua et al., TPDL 2016
Archival Web Coverage
● Ainsworth et al., JCDL 2011
● Alkwai et al., ToIS 2017
● SalahEldeen et al., TPDL 2012
● Kelly et al., JCDL 2018
On-Premise Indexing
● Hammer et al., IJCIS 2000
● Kumar et al., RAIT 2016
Surface Web Crawling
● WorldWideWebSize.com
● Lawrence et al., Science 1998
● Alarifi et al., SRE 2012
● Khabsa et al., PLOS ONE 2014
Deep/Hidden Web Crawling
● Raghavan et al., VLDB 2001
● Ntoulas et al., JCDL 2005
● Wu et al., ICDE 2006
● Sheng et al., VLDB 2012
Focused Crawling
● Micarelli et al., The Adaptive Web
2007
● Bergmark et al., ECDL 2002
● Li et al., WI-IAT 2012

● RQ1: Understanding Web Archives
a. How to Learn an Archive’s Holdings?
b. How to Learn an Archive’s Voids?
● RQ2: How to Summarize and Serialize Archival Holdings
and Voids for Dissemination?
● RQ3: How to Utilize MementoMaps for Memento Routing?
Research Questions
27

RQ1a: How to Learn an Archive’s Holdings?
● Content-based Profiling
○ CDX Profiling [Alam, et al., TPDL 2015; Alam, et al., IJDL 2016]
○ Fulltext Search Profiling [Alam, et al., TPDL 2016]
● Usage-based Profiling
○ Sample URI Profiling
○ Response Cache Profiling [Bornand, et al., JCDL 2016]
28

RQ1b: How to Learn an Archive’s Voids?
● Content-based Profiling
○ Collection Exclusion Policies
○ Access Control Lists (ACLs)
● Usage-based Profiling
○ Archive’s Access Log Profiling
○ Aggregator’s Access Log Profiling
29

RQ2: How to Summarize and Serialize Archival
Holdings and Voids for Dissemination?
● Generation and Compaction
● Updates and Merger
○ Incremental updates
○ Distributed generation
● Pagination
○ Small chunks for storage and transportation
○ Time- or TLD-based organization
○ Holdings and Voids segregation
● Dissemination and Discovery
30

RQ3: How to Utilize MementoMaps for Memento
Routing?
● Inverted Index
○ Precomputed index of MementoMaps
● Routing Score Estimation
○ Rank ordering each candidate archive
○ Routing to top-k archives
○ Routing to archives with score above certain threshold
● Machine Learning-Based Classifier
○ Models for individual web archives
31

MementoMap Framework Components
● Ingestion (RQ1)
○ CDX files/API
○ Fulltext search
○ Access logs
○ Sample URIs
● Summarization and Serialization (RQ2)
○ Resource constraints
○ Application-specific variants
● Memento Routing (RQ3)
○ Integration with aggregators
33

Evaluation Plan
● Cost
○ Time
○ Storage space
○ Network bandwidth
○ Periodic updates
● Accuracy
○ Targeting for fewer false positives and false negatives in individual MementoMap
● Freshness
○ How often MementoMaps of a web archive need to be updated?
● Routing Efficiency
○ Accuracy of inverted index across multiple web archives
34

What is Archived in Arquivo.pt?
What is Accessed from MemGator?
35
2B URI-Rs that have
1-9 mementos each in
Arquivo.pt were never
requested from ODU’s
MemGator server.
43 URI-Rs were
requested thousands
of times each, but
had zero mementos
in Arquivo.pt.
45 URI-Rs had tens
of mementos each
that were requested
hundreds of times.

What is Archived in Arquivo.pt?
What is Accessed from MemGator?
36
Blind spot of a
usage-based
profile
Blind spot of a
content-based
profile

Who Bears the Cost of Bad Routing Decisions?
37
Actual
Present in the Archive Not in the Archive
Predicted
Routed to the
Archive
True Positive (TP) False Positive (FP)
Not Routed to
the Archive
False Negative (FN) True Negative (TN)
FP: Wasteful (Infrastructure suffers)
FN: Disuse (Users suffer)

Recall and Accuracy
38
Recall = TP / (TP + FN)
We do not report Precision because it does not capture TNs, which are crucial in Memento routing.
How many lookup URIs are routed among URIs that are present in an archive?
How many lookup URIs are correctly routed or not routed?
Accuracy = (TP + TN) / All

RQ1a:
How to Learn an Archive’s Holdings?
39

URI Canonicalization and SURT
40
https://news.bbc.co.uk/images/Logo.png?width=200&height=80&rotate=90%C2%B0#top
http://www.news.BBC.co.uk/images/Logo.png?width=200&height=80&rotate=90%c2%b0#top
http://www.news.bbc.co.uk/images/Logo.png?rotate=90%c2%B0&width=200&height=80
http://NEWS.BBC.CO.UK:80//images//Logo.png?height=80&width=200&rotate=90%c2%b0#top
news.bbc.co.uk/images/Logo.png?height=80&rotate=90%C2%B0&width=200
uk,co,bbc,news,)/images/logo.png?height=80&rotate=90%c2%b0&width=200
Canonicalization
SURT

CDX/CDXJ Summarization
41http://archive.org/web/researcher/cdx_file_format.php

URI-Key Generation and Static Profiling Policies
42
● HmPn Policy (maximum “m” host segments and “n” path segments)
○ H3P0 (3 host and 0 path segments): uk,co,bbc,)/
○ HxP1 (All host and 1 path segments): uk,co,bbc,news,)/images
● DLim Policy (“RegisteredDomain[#SubDomain[/#Paths[/#Queries[/PathInitial]]]]”)
○ DDom (up to domain name): uk,co,bbc,)/
○ DIni: (up to path initial): uk,co,bbc,)/1/2/3/i

Archives Dataset
43
Archive URI-Rs URI-Ms Index Size
Archive-It 1.9B 5.3B 1.8TB
UKWA 0.7B 1.7B 0.5TB
Stanford 12M 25M 8.3GB

Sample Query URI Sets
44
Sample
(1M URIs Each)
In
Archive-It
In
UKWA
In
Stanford
Union
{AIT, UK, SU}
DMOZ 4.097% 3.594% 0.034% 7.575%
MementoProxy 4.182% 0.408% 0.046% 4.527%
IAWayback 3.716% 0.519% 0.039% 4.165%
UKWayback 0.108% 0.034% 0.002% 0.134%
Alam et al., “Web Archive Profiling Through CDX Summarization”, IJDL 2016

CDX Size vs URI-M (UKWA 10 Years)
45
Alpha: 175 bytes per CDX line

URI-M vs URI-R (UKWA 10 Years)
46
Gamma: 2.46 K: 2.686
Beta: 0.911

Relative Space Cost (UKWA 7 Years)
47
Phi: 8.5e-07 -- 0.70583

Time Cost (UKWA 7 Years)
48
Tau: 5.7e-05 -- 6.2e-05
CDX: 45GB
URI-Ms: 181M
URI-Rs: 96M
Time: 3 hours

Resource Requirement
49Alam et al., “Web Archive Profiling Through CDX Summarization”, IJDL 2016

Cost vs. Accuracy: UKWA
50
Archive-It and Stanford archives have similar trends.

Profile Policy Groups: Cost vs. Accuracy
51
Group Group Relative Cost Accuracy
G1 H1P0/TLD Bound by # of TLDs ≈ 0.01
G2 H3P0, DDom, DSub, DPth, DQry < 0.01 ≈ 0.78
G3 DIni ≈ 2 * G2 ≈ 0.88
G4 HxP1 ≈ 5 * G3 ≈ 0.94
G5 Higher HmPn 0.4 - 0.7 Not Explored
G6 URIR 1.0 1.0

Collecting CDX Is Difficult
52
https://memgator.cs.odu.edu/
MemGator Service at ODU
currently aggregates 16
web archives, but we have
CDX data only from 4.
However, some of these
archives have fulltext
search support, so we can
learn about their holdings.
⭐
⭐
⭐
⭐

Who Knows Term Frequency for Estonian Nouns?
53
https://en.wiktionary.org/wiki/Category:Estonian_nouns

Fulltext Search Profiling
54
Top Nouns
time
year
people
way
man
day
thing
child
mr
government
Random Dict
analogies
unbolt
consonant
coils
stolidly
cigar
decrepit
rhododendron
cannibal
honeydew
Dynamic Words Discovery
the ‫وﻛﺎﻟﺔ‬ war
angry ‫أﻧﺑﺎء‬ the
arab ‫اﻟﻌرﺑﻲ‬ middle
news ‫اﻟﻐﺎﺿب‬ east
service on arabic
a politics poetry
source war art

Random Searcher Model (RSM)
55
Search for a word

56
http://jeffreyhill.typepad.com/english/
http://www.nc-net.info/english.php
http://english.aljazeera.net/
http://twitter.com/AJEnglish
https://vimeo.com/248815105
http://www.bridge.edu/
http://www.wordreference.com/
https://www.facebook.com/aljazeera
http://www.elizabethangardens.org/
...
Collect resulting links

57
...
Load a random result link

58
Teaching Resources Adjunct Toolkit NC NET Academy PD Planning Tools Regional Centers Campus Liaisons Nontraditional Careers College Tech Prep NC
ACCESS Co op Education Green Technology You are here NC NET Teaching Resources Discipline Specific English English Self Paced Modules Writing
Across the Curriculum NC NET Western Center Incorporating Visuals in Workplace Documents Sections 1 2 Wake Tech Community College Incorporating
Visuals in Workplace Documents Section 3 Wake Tech Community College All self paced modules can be accessed through the NC NET Blackboard server
Log in with the user name faculty and the password nc net Once connected you can view the courses by topic or alphabetically by title English
Webliography North Carolina Community College System 2012
...
Extract words from the page

59
...
Select a word to search

60
...

RSM Modes Costs
61
Mode HTTP Cost Remarks
Static C
Suitable for specialized collection with known top
keywords
PopularityBiased 2C Human like model, but costly
EqualOpportunity 2C Human like model, but costly
Conservative C + 𝛿
(where 𝛿 << C)
Suitable for any collection and works without any
supplementary materials with very little overhead
“C” is the cost/number of search queries.

Search Needed vs. Coverage
62
100% in 11K searches
100% in 27K searches
100% in 337K searches 100% in 1.9M searches
Alam et al., “Web Archive Profiling Through Fulltext Search”, TPDL 2016

Accuracy, Recall, and Coverage (10-100%)
63
DMOZ IA Wayback
UK WaybackMemento Proxy
Low Accuracy (high FP) =>
Archives & Aggregator suffer
Low Recall (high FN) =>
Users suffer
Alam et al., “Web Archive Profiling Through Fulltext Search”, TPDL 2016

64
RQ1b:
How to Learn an Archive’s Voids?

Why Profile Archival Voids?
65
$ curl -I https://web.archive.org/web/https://quora.com/
HTTP/1.1 403 FORBIDDEN
Server: nginx/1.15.8
Date: Wed, 02 Dec 2020 20:39:33 GMT
Content-Type: text/html; charset=utf-8
Connection: keep-alive
Server-Timing: captures_list;dur=0.150497
X-App-Server: wwwb-app58
X-ts: 403
The Internet Archive has
many “*.com” domains,
but it may not want to
capture or replay some.

Archival Voids Profiles Reduce False Positives
66
org,arxiv)/abs/a 40
org,arxiv)/abs/b 23
org,arxiv)/abs/c 17
org,arxiv)/format/a 15
org,arxiv)/format/b 20
org,arxiv)/format/c 10
org,arxiv)/search/a 30
...
org,arxiv)/abs/* 80
org,arxiv)/format/* 45
org,arxiv)/search/* 60
org,arxiv)/* 185
org,arxiv)/abs/d
False Positive org,arxiv)/pdf/a
org,arxiv)/pdf/b
org,arxiv)/pdf/c
False Positive
org,arxiv)/* 185
org,arxiv)/pdf/* 0
How about summarizing frequently
accessed URIs an archive does not hold?

Arquivo.pt Access Log Dataset
67

Most Frequently Accessed URIs
68
Most of the traffic to
“fccn.pt” is originated
from the UptimeRobot
Always returned a “404 Not Found” response.

404-Only Frequencies and Request Savings
69
An archival voids profile of 2.4k URIs, that were accessed hundreds of
times each or more, could have saved about 8.4% of wasted requests.

Archival Voids Recommendations
70
● Keep archival voids profiles separate from archival holdings
● Update often
● Use specific keys with only high confidence
● Profile only resources that are high in demand
● Archives themselves are better sources of truth than
external observers

RQ2:
How to Summarize and Serialize
Archival Holdings and Voids for
Dissemination?
71

If Only Archives Could Tell What to Ask Them For
● Websites advertise their holdings using sitemap.xml, why can’t archives?
○ Archives have billions or even trillions of URI-Ms
○ Such exhaustive lists would go stale very quickly
● How about robots.txt?
○ It is compact, but is exclusion format, it does not tell what the site has
○ It assumes a single domain, patterns are for paths (not the domain name)
● How about well-known URIs?
○ Good for automated discovery of domain-specific metadata resources
● How about combining these ideas?
○ Introducing MementoMap!
72

A MementoMap Example
73
!context ["http://oduwsdl.github.io/contexts/ukvs"]
!id {uri: "http://archive.example.org/"}
!fields {keys: ["surt"], values: ["frequency"]}
!meta {type: "MementoMap", name: "A Test Web Archive", year: 1996}
!meta {updated_at: "2018-09-03T13:27:52Z"}
* 54321/20000
com,* 10000+
org,arxiv)/ 100
org,arxiv)/* 2500~/900
org,arxiv)/pdf/* 0
uk,co,bbc)/images/* 300+/20-
https://github.com/oduwsdl/ORS/blob/master/ukvs.md
Goodbye HmPn/DLim static profiling policies, thanks to our SURT with wildcard.

SURT Representation With Wildcard
74
Original SURTs did not have wildcards.
We introduced it for dynamic profiling.
In practice the common “http://(” prefix
is removed.

Arquivo.pt Index Statistics
75Alam et al., “MementoMap Framework for Flexible and Adaptive Web Archive Profiling”, JCDL 2019
The Internet Archive is
about 150 times bigger
than Arquivo.pt.

Top Arquivo.pt TLDs
Arquivo.pt was created to
archive sites of interest to
the Portuguese people.
One third of the Arquivo.pt
is other than *.pt pages,
hence routing based on top
TLDs would miss a lot.

Who Would Have Thought
Arquivo.pt Has 10K+ .онлайн Sites?
77
“.онлайн”
(encoded as “xn--80asehdb”)
is an IDN gTLD which means
“.online”

Cumulative Growth of URI-Ms and URI-Rs in Arquivo.pt
50% mementos
were captured in
the last two active
years alone.

Shape of HxPx Key Tree of Arquivo.pt

Incremental Children Reduction Rate

Processed Lines vs. Compacted MementoMap Growth
81
com,example)/a/1/x
com,example)/a/2
com,example)/a/3
com,example)/b/1
com,example)/b/2
com,example)/c/1
com,example)/a/*
com,example)/b/1
com,example)/b/2
com,example)/c/1
com,example)/*
Alam et al., “MementoMap Framework for Flexible and Adaptive Web Archive Profiling”, JCDL 2019

MementoMap Generation, Compaction, and Lookup
1.5% Relative Cost yields 60% Accuracy.
Arquivo.pt can save 60% wasted traffic by
publishing a 119MB summary file!

Dissemination and Discovery Methods
83
GET /.well-known/mementomap HTTP/1.1
Host: arquivo.pt
Link: <https://arquivo.pt/path/to/mementomap.ukvs>;
rel="mementomap"
<link href="https://arquivo.pt/path/to/mementomap.ukvs"
rel="mementomap">
Well-known URI
Link Header
Link HTML Element

RQ3:
How to Utilize MementoMaps for
Memento Routing?
84

Putting Archival Holdings and Voids to Work
85
● Combine MementoMaps (holdings and/or voids) of various
web archives
● Create an inverted index for efficient cross-archive lookup
● Define scores based on reported data in MementoMaps
● Predict whether a lookup URI should be routed to an archive

How Densely Is a URI Subtree Archived?
* /10000000000+
edu,odu)/* /10000~
86
Where to go?
https://www.odu.edu/academics/graduation-commencement/graduation/FAQs
* /10000+
edu,odu)/* /1

Density Score
87
A normalized score to assess how actively an archive is capturing a given URI space.
* /1000000
org,example)/a/b/c/* 200/100org,example)/a/b/c/d/e
µ = log(1 + 100) / log(1 + 1000000) ≈ 0.334

How Close Are a Lookup URI and Its Corresponding
URI Key?
88
edu,odu,)/academics/graduation-commencement/* /100
Where to go?
edu,odu)/* /100

Closeness Score
89
A normalized score to assess how closely the longest URI Key prefix matches with the lookup URI.
org,example)/a/b/c/*org,example)/a/b/c/d/e
χ = min(log(1 + 5), 1) / (1 + 7 − 5) ≈ 0.259
Ll
= 7 Lk
= 5

How Likely Is the Lookup URI to Be Found
in Each of the Archives?
90
Which ones to select?
1.00
0.75
0.50
0.25
0.00
1.00
0.75
0.50
0.25
0.00
1.00
0.75
0.50
0.25
0.00
1.00
0.75
0.50
0.25
0.00
Cut-off
Threshold

Routing Score
91
A normalized score to assess the likelihood of finding the lookup URI in an archive.
ρ = µ * χ ≈ 0.334 * 0.259 ≈ 0.087
Weighted Relative Routing ScoreRelative Routing Score

MementoMaps of Different Archives
92

Inverted Index of MementoMaps
93

Inverted Index Lookup Result
94
Not profiled (default routing score)
Archival Void (“0” routing score)
Relative Routing Scores of all
archives add up to “1.0”

Web Archive Routing Evaluation Dataset
95
Created from the MemGator access logs

Archival Collection Diffusion
96
Simple TLD-based Memento routing is insufficient.

Baseline Routing
97
Large Recall values and small request costs of top-1 and top-2 policies
show the effectiveness of our heuristic Routing Score.

Cut-off Threshold Routing
98
Inclusion of Voids
profiles improve
Savings more
prominently than
Accuracy due to
frequency bias.
Poor prevalence of
sample URIs in
UKWA and Stanford
hurts their scores.

Machine Learning-Based Routing
99
Classifier is biased towards Accuracy at the cost of poor Recall
due to poor prevalence of positive cases in the dataset.

Contributions, Future Work, and
Conclusions
100

Tools Contributions in the Web Archiving Ecosystem
101
Open-Source Tools/Scripts
❖ https://github.com/oduwsdl/ipwb
❖ https://github.com/oduwsdl/Reconstructive
❖ https://github.com/oduwsdl/MemGator
❖ https://github.com/oduwsdl/archive_profiler
❖ https://github.com/oduwsdl/accesslog-parser
❖ https://github.com/oduwsdl/MementoMap
❖ https://github.com/oduwsdl/ORS
❖ https://jekyll.github.io/classifier-reborn/

Contributions: Algorithms
● Random Searcher Model (RSM)
○ Utilize fulltext search interface to sample archival holdings
○ Supports multiple modes of operation
○ Discovery: 10% => Accuracy: 0.8, Recall: 0.9
● MementoMap Generation, Compaction, and Merger
○ Consumes a sorted list of URIs in SURT format
○ Allows configuration options to control compaction
○ Linear, single-pass, small constant memory footprint
(irrespective of the input size)
○ Accuracy > 0.6, Recall: 1.0, Relative Cost < 1.5%
102

● URI Key: Extended SURT with the wildcard support to describe subtrees of the URI
space in the form of URI prefixes
● Archival Holdings: A measure to describe holdings of an archive
● Archival Voids: A measure to describe what an archive is missing
● Relative Cost: The ratio of the number of URI keys used to describe summarized
holdings of an archive over the total number of unique URI-Rs in the archive
● Frequency Score: A means to represent the number of URI-Ms and/or URI-Rs
under a URI key
● Density Score: A normalized score derived from the frequency score to describe
the archiving activity under a URI key
● Closeness Score: A normalized score to describe how similar or different two URI
keys are
● Routing Score: A normalized score to represent how likely it is that an archive has
a URI
Contributions: Terminologies and Metrics
103

Contributions: Publications
104
RQ2
● TPDL 2015
● TCDL 2015
● IJDL 2016
● JCDL 2016
● TPDL 2016
● TPDL 2016
● JCDL 2017
● TCDL 2017
● JCDL 2018
● JCDL 2019 ⭐
● IJDL 2021
● RFC
● RFC
RQ3
● JCDL 2016
● TCDL 2017
● ESCAPE 2019
● JCDL 2021
● IJDL 2021
🥇 Best paper/poster award ⭐ Best paper nomination Italics = Planned/in progress
RQ1
● TPDL 2015
● TCDL 2015
● IJDL 2016
● TPDL 2016
● JCDL 2017 🥇
● JCDL 2018 🥇
● JCDL 2019
● WADL 2019
● ESCAPE 2019
● WADL 2020
● JCDL 2021
● IJDL 2021

Future Work
● URI Keys for collection seed summarization and collection diversity measure
● Archival Voids to assess crawl jobs
● Explore UKVS file format use cases in web archiving and beyond
● Profiling other dimensions
○ Datetime
○ Language
● Standards for cross-archive collection exploration
● Alternate approaches to URI subtree rollups
● Other ML and Neural Net-based techniques for Memento routing
● Hierarchical network Memento routing
● Adoption of MementoMap framework by archives and aggregators
105

Future Work: MementoMap Adoption Path
● PWA, UKWA, and NLA have shown interest
● PyWB archival replay system is open for implementation
● MemGator and LANL’s Time Travel service are interested
● Big web archives can start with publishing archival voids
○ No need to profile IA
● Archives with access restrictions can have multiple
MementoMaps
● Third parties can create and publish MementoMaps of the
rest of the archives while they catch up
106

107
<salam@cs.odu.edu> <sawood@archive.org>

● Introduced challenges in Memento aggregation
○ Broadcasting can be evil
○ Profiling is desired for collection understanding and better Memento routing
○ Aggregation is useful, even for the small web archives
● MementoMap framework addresses three research questions
○ RQ1: How to learn about a web archive’s holdings and voids?
■ Holdings: CDX and fulltext search profiling
■ Voids: Access log profiling
○ RQ2: How to summarize and serialize archival holdings and voids?
■ MementoMap/UKVS serialization and dissemination
○ RQ3: How to utilize MementoMaps for informed Memento routing?
■ Inverted Index, Routing Score, Classifier
Conclusions
108
Over 96% Accuracy with 89% Recall or 68% Accuracy with 99% Recall
via a 120MB MementoMap file for an archive with 2B+ unique URI-Rs.

MementoMap: A Web Archive Profiling Framework for Efficient Memento Routing

More Related Content

What's hot (10)

Similar to MementoMap: A Web Archive Profiling Framework for Efficient Memento Routing (20)

More from Sawood Alam (20)

Recently uploaded (20)

MementoMap: A Web Archive Profiling Framework for Efficient Memento Routing