Carbon Dating The Web: Estimating the Age of Web Resources

Carbon Dating the Web:
Estimating the Age of Web Resources
Hany SalahEldeen & Michael Nelson Carbon Dating the Web
Hany M. SalahEldeen & Michael L. Nelson
Old Dominion University
Department of Computer Science
Web Science and Digital Libraries Lab.

Motivation
In our research in social media,
resource sharing, and user
intention a question emerged…
Hany SalahEldeen & Michael Nelson 1 Carbon Dating the Web
When did a certain resource first
appear on the web?

First thought: Last Modified
Response Header
$ curl -I http://ws-dl.blogspot.com/2012/02/2012-02-11-losing-my-revolution-year.html
HTTP/1.1 200 OK
Content-Type: text/html; charset=UTF-8
Expires: Wed, 08 May 2013 14:18:49 GMT
Date: Wed, 08 May 2013 14:18:49 GMT
Cache-Control: private, max-age=0
Last-Modified: Wed, 08 May 2013 08:03:02 GMT
ETag: "e419d850-22ae-4fe6-a0f4-8ab9477f0c0d"
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block

The server responds with the last
modified date …
HTTP/1.1 200 OK
Date: Wed, 08 May 2013 14:18:49 GMT
Real Creation date
Current Server datetime
Last modified date (Incorrect)

Lacks accuracy
HTTP/1.1 200 OK
Date: Wed, 08 May 2013 14:18:49 GMT
Real Creation date
Current Server datetime
Last modified date (Incorrect)
Problematic as it is inaccurate in a large percentage of cases.
08 May 2013 ≈ 2012-02-11

Last modified date header is not
available
% curl -I http://temporalweb.net/
HTTP/1.1 200 OK
Set-Cookie: 60gpBAK=R1224192509; path=/; expires=Sat, 11-May-2013 03:45:10 GMT
Date: Sat, 11 May 2013 02:37:55 GMT
Content-Type: text/html
Connection: keep-alive
Set-Cookie: 60gp=R152135972; path=/; expires=Sat, 11-May-2013 03:36:44 GMT
Server: Apache/2.2.X (OVH)
Accept-Ranges: bytes
Vary: Accept-Encoding
Sometimes it is not present in the response headers.

Second thought: Timestamp on
the page

But the timestamp is highly
inconsistent

… and dependent on the page’s
style/scheme.

So as its location on the page

Pages’ Timestamps Differ
Very dependent on the page’s scheme/style
Not consistent
Non-existent sometimes

Shortcomings of using timestamp
extraction
• M. Inoue and K. Tajima. Noise robust detection of the emergence and spread of topics
on the web. In Proceedings of the 2nd Temporal Web Analytics Workshop, TempWeb
'12, pages 9 {16, New York, NY, USA, 2012. ACM
M. Inoue and K. Tajima developed a technique of extracting
creation timestamps on web pages.
Shortcomings:
• Ambiguity (12/07 is it the 12th of July or the 7th of December?).
• Non generalizable.
• Highly dependent on the specific CMS
• Highly dependent on the most prominent timestamp patterns.

But what if the resource itself
doesn’t exist any more?

Third thought: First existence in
public archives
Timestamp of the first memento

Shortcomings:
1- The page is not archived

Shortcomings:
2- Delay between page creation and archive’s first crawl.

Shortcomings:
3- A page is published then deleted before it is archived.

Shortcomings:
4- The archive’s quarantine (12 month- 2 weeks)

Goal
Create a tool that estimates with
generality the creation date of the
resource without relying on
specific infrastructures

Target Specification
• Doesn’t rely on the infrastructure of the
hosting web server.
• Doesn’t rely on the state and template of the
resource.
• Highly generic.
• Fast response with no quarantine periods.
• High accuracy, getting close estimates to real
creation date.

Idea
Moving objects leave trails…

Idea
Moving objects leave trails…
Or:
Foo  If you were Aussie
Chad  if you were British

Idea
Web pages leave trails as well since
the day they were created…

Web Trails
A web page could leave a trail of one of the
following denoting its existence:
– References
– Links (anchors)
– Social media likes and interactions.
– URL shortening.
– Backlinks

The Assumptions
We can propose reasonable assumptions that:
1. We have no prior knowledge of the resource or
its hosting web server.
2. The creation date and the publishing date of a
resource coincide.
 Ex.: When you write a blog, you publish it as
soon as you create it.

Idea
The creation date of any of the associated
events/trails could be an estimate of the
creation date.
Web
Resource

Scenario
Let’s consider the following scenario, on
Saturday night on the 11th of February of last
year I wrote a blog post about my work on the
research group’s blog page.

After creating the post I tweeted
about it …
https://twitter.com/hanysalaheldeen/status/168704224488730625

Then it picked up some speed on
Twitter and Facebook …
http://topsy.com/http://ws-dl.blogspot.com/2012/02/2012-02-11-losing-my-revolution-year.html

The timeline of the resource

Given the events linked to the
existence of the resource we will
examine ways to extract first
observations

Age Estimation Methods
1. Resource and server analysis.
2. Backlinks analysis.
a) Web page backlinks.
b) Social media backlinks.
3. Archiving analysis.
4. Search engine indexing analysis

Resource and Server Analysis
HTTP/1.1 200 OK
Date: Wed, 08 May 2013 14:18:49 GMT
Examine the server response and extract the last
modified date from the header if exists.

Observations recorded:
1. Last modified date from the response header.

Backlinks Analysis
• We use Google search API to discover backlinks of A.
• B & C were created after A was created.
• But this assumption is not completely true.
• Page B or C could be modified later to its creation of A
A
(The resource)
B C

Time Magazine
Ex.: If the front page of Time magazine decided
to finally feature me as “Person of the Year”
In this case page B (Time magazine’s front page)
was modified to point to my page A
Hany’s
Website
Time
Magazine

When did the link first appear?
To solve this problem:
1. We extract the timemap of the archived mementos of B.
2. Perform binary search to allocate the first appearance of the
link to A in B.
3. Get the timestamp of that first memento.
time
I first appeared here!

2. First Appearance of a backlink.

Social Media Backlinks
• Similarly, you create a social backlink when you
tweet about a page

Topsy Otter API
Upto500Tweets

Topsy Otter API
Different shortened versions

Topsy Otter API
Break ties via the API epoch

3. First Tweet published.

URL Shortening
http://ws-dl.blogspot.com/2012/02/2012-02-11-
losing-my-revolution-year.html
http://bit.ly/losing_revolution
Extract number of clicks
Creation Date of the Bitly

4. First Bitly Shortened URL created.

Archives Analysis
• Furthermore, if the original headers exist for the first memento we extract
the original last modified date.
Get timestamp of first memento
Download the memento timemaps of the resource

5. Time stamp of first memento in the archives.

Search Engine Index Analysis
• We use Google’s search API to extract the last crawled date
• Relatively short time between resource creation and search engine discovery.
• Drawback: Granularity is by day not by time.
Last crawled dates

5. Time stamp of first memento in the archives.
6. Date of the last crawl by the search engine.

Ok, now we have a collection of
sources that return creation dates,
what will we do next?

Timestamps Accumulation
• We collect the obtained dates and get the
leftmost creation date recorded.

Timestamps Accumulation

Next step: Verifying our methods

Estimated Age Verification
1. Collect a dataset of webpages of known
creation/publishing date.
2. Compare the estimated results from our
method and the actual dates recorded.

Gold Standard Data Collection
We collect the pages from 4 difference
categories of collections to ensure variation.
1. News Sites.
2. Social Media and Blogs.
3. Long Standing Domains.
4. Manual Random Extraction.

News Sites
Using RSS and Atom feeds or XML sitemaps we
extracted numerous pages along with their
respective publishing dates.
1. Google News (29,154 pages)
2. BBC (3,703 pages)
3. CNN (18,519 pages)
4. Yahoo News (34,588 pages)
5. The Hollywood Gossip (6,859 pages)

Social Sites
We randomly selected different resources with
no regard to popularity to avoid the inherent
bias:
1. Pinterest (55,463 posts)
2. Tumblr (52,513 posts)
3. Youtube (78,000 posts)
4. Word Press (2,405,901 posts)
5. Blogger (32,417 posts)

Long Standing Domains
• Extract the top 500
domains from
Alexa.com
• Query their DNS
registry dates.
• Were able to extract
167 dates.

Manual Random Extraction
• We extracted 90 different random URLs
obtained from random walks on the web,
visually inspected them to extract the creation
date.
• The 10 URLs analyzed by Jatowt et al.*
* A. Jatowt, Y. Kawai, and K. Tanaka. Detecting age of page content. In Proceedings of the 9th
annual ACM international workshop on Web information and data management, WIDM '07,
pages 137--144, New York, NY, USA, 2007. ACM.

Gold Standard Data Collection
 From each we randomly selected 100 unique URLs to create our gold standard
dataset

Evaluation
• Applied our 6 methods on 1200 resources.
• Get leftmost estimation.
Number of Resources Percentage
An estimation found 910 76%
Exact matching estimation 393 33%
No estimation found 290 24%
Total Resources 1200 100%

Evaluation

Actual Vs. Estimated Dates

So what happens if one of these 6
methods failed?

Isolation and Elimination

Carbon Date API

http://cd.cs.odu.edu/cd/<Your URL Here>

Carbon Date API on GitHub
• Due to the slow response we advise that you
download the module and install it on your
machine.
• https://github.com/HanySalahEldeen/CarbonDate

Extra Slides
Hany SalahEldeen & Michael Nelson Carbon Dating the Web

Without Bitly

Without Google

Carbon Dating The Web: Estimating the Age of Web Resources

More Related Content

Viewers also liked (16)

More from heinestien (8)

Recently uploaded (20)

Carbon Dating The Web: Estimating the Age of Web Resources