From Publisher To Platform: How The Guardian Used Content, Search, and Open Source To Build a Powerful New Business Model

1

From publisher to platform
How the guardian used content, search, and open source to
build a powerful new business model
Stephen Dunn, Guardian News and Media

Apache Lucene EuroCon 21 May 2010

The publishing era

Apache Lucene EuroCon 21 May 2010 2

We started a long
time ago:


“To secure the ﬁnancial and editorial independence of
To secure the financial and editorial
The Guardian in perpetuity.”
independence of the Guardian in perpetuity.
Topromote freedom in the press press and liberal
“To promote freedom in the and liberal journalism
journalism globally.
globally.”

2010


2010
Keyword page

Live blogs
iPhone app Mobile site

Twitter updates
Swine flu Comment

Content partnerships

Newspapers

Audio

Video Data API


1996


1999


01-> 06


2009
1.5M pages
and counting

250M+ pages/
month

30M visitors/
month

4x Webby
award winner
(best
newspaper
site)


Part of the Web


1. Permanent

http://www.ﬂickr.com/photos/fstorr/

• “A cool URI is one that does not change” Tim Berners-Lee 1998
• 1.5 million resources redirected to new scheme

2. Addressable
★ Resources are “about” something - ready for the
social web.

★ We live in “the age of point-at-things” (Coates 2005)


3. Discoverable

★ Multiple routes
to content

★ Tagging drives
discovery


The hackable guardian.co.uk
http://www.guardian.co.uk/....



/technology/internet

/technology/all

/environment/climatechange



/technology/internet

/technology/all

/environment/climatechange +business/globaleconomy



/technology/internet/rss

/technology/all/rss

/environment/climatechange +business/globaleconomy/rss


Results...


Site trafﬁc growth Final Release

Unique Users

First release



Unique Users
30,000,000

26,250,000 First release

22,500,000
Unique Users

18,750,000

15,000,000

11,250,000

7,500,000

3,750,000

Sep 2005 Feb 2006 Jul 2006 Dec 2006 May 2007 Oct 2007 Mar 2008 Aug 2008 Jan 2009



Unique Users
30,000,000


22,500,000
Unique Users

Pre - project
18,750,000

15,000,000

11,250,000

7,500,000

3,750,000




Unique Users
30,000,000


22,500,000
Unique Users

Pre - project
18,750,000

15,000,000

11,250,000
36M
7,500,000

3,750,000



However...


1 Billion+
Internet
Users!


....”How I stopped
worrying about
my website and
learned to love
the whole
Internet.”

Matt McAlister


The Open Strategy
OPEN IN OPEN OUT

Bring in data and Enable partners to
apps from the build applications
Internet using Guardian
content and services
for other digital
platforms


"Our most interesting experiments lie in combining
what we know with the experience, opinions and
expertise of the people who want to participate
rather than passively receive.”

TA
BE

The Open Platform


TA
BE

OPEN IN OPEN OUT

Bring in data and apps Allow partners to build
from the Internet applications using
Guardian content and
services for other digital
platforms


TA
BE

The suite of services
enabling partners to build
applications with the
Guardian

TA
BE


TA
BE

CONTENT API DATA STORE POLITICS API

A service for A directory of Open database of
selecting and useful data candidates, voting
collecting content curated by records,
from the Guardian Guardian constituencies,
editors election results,
for re-use
live data on
election day


TA
BE
Your App Here!

CONTENT API
A service for selecting REST API
and collecting content
from the Guardian for
re-use

Search engine
CMS

Guardian
database

TA
BE


• Stamen Design - APIMaps.org

Text


TA
BE

DATA STORE
A directory of
useful data curated
by Guardian
editors


TA
BE

POLITICS API
Open database of
candidates, voting
records, constituencies,
election results, live
data on election day


TA
BE

POLITICS API
Open database of
candidates, voting
records, constituencies,


TA
BE

Open for Business


Open for Business


1 3 Tiers of access, 3 Revenue models

BESPOKE: Take, reformat, augment our content. Same access as
Guardian. Revenue model to be negotiated. Combination of Media,
Fees, Downloads.

APPROVED: Take our full article content, with an advert.
Guardian keeps ad revenue, you keep rest-of-page revenue

KEYLESS: Take our headlines. You keep associated revenues


What this means
OPEN OUT: Developers can now access our full content APIs
on demand with keys post-approved.

We are now positioning the platform as a place to do
business with us.

So, rapid scalability, reliability, performance, are now core
requirements


2 Open In
CONTENT API DATA STORE POLITICS API
A service for selecting A directory of Open database of
and collecting content useful data curated candidates, voting
from the Guardian for by Guardian records,
re-use editors constituencies,


2 Open In
CONTENT API DATA STORE POLITICS API MICROAPPS
A service for selecting A directory of Open database of A framework for
and collecting content useful data curated candidates, voting integrating 3rd party
from the Guardian for by Guardian records, applications into
re-use editors constituencies, guardian.co.uk.


OPEN OUT
OPEN IN
Allow partners to build
Bring in data and apps
applications using
from the Internet
Guardian content and
services for other digital
platforms


App showcase


What this means
Open In: Partners can now more easily integrate
into our core

The Open Platform will become key to our
commercial future.


Evolving the
architecture


From Publisher to Platform
★Seeking massive growth, but no longer only
broadcasting content

★User/partner engagement & contribution on
★journalism
★data
★software
★applications
★revenue and ads

★ Support developers and partners with data and
APIs, need scalability, reliability, speed

Web server Web server Web server

App server App server App server

Memcached

Oracle

CMS


Web server Web server Web server

Why RDBMS?
App server App server App server
5 years ago, fewer alternatives

Understand operations procedures
Memcached
Can easily recruit DBAs / devs

Developer/ops tools
Oracle
Business critical system: a safe
choice

CMS Data feeds

Scaling


Unique Users


30,000,000
Unique Users
26,250,000

22,500,000
Unique Users

18,750,000

15,000,000

11,250,000

7,500,000

3,750,000



Unique Users


28,000,000
25,750,000 Unique Users
23,500,000
21,250,000
19,000,000
16,750,000
14,500,000
12,250,000

May 2008 Jul 2008 Sep 2008 Nov 2008 Jan 2009

Whatʼs going on?
★We tag our content
(multifaceted)

★Guardian.co.uk is a faceted
browse through our tag-
space, with editorial teams
“spotlighting” key resources
on selected nodes.

★Can apply multiple facets in
queries faster in a search-like
architecture, than an RDBMS


“Related content” from search engine


5

Your App Here!

CONTENT API
A service for selecting REST API
and collecting content
from the Guardian for
re-use

Search engine
CMS

Guardian
database

We used Solr/Lucene
Can perform complex queries, including full text search

We can change the schema with no downtime.

On our dataset most queries are of a similar cost

Scales very well horizontally

Replication makes it easy to work in the cloud


Core

Web servers

App server

Memcached

rdbms

CMS


Core

Content API
Web servers

Solr
App server

Solr

Memcached
Solr

rdbms Solr
Solr

Solr
CMS

Cloud, EC2


Open in?

Simple REST/ HTTP framework
MICROAPPS allows lightweight development

A framework for Applications proxied for
integrating 3rd party performance
applications into
guardian.co.uk. Apps generally hosted in the
cloud, hot deployment into
production


Core

Apps
Web servers

Proxy
App
App server

App
Memcached
App

App rdbms

App

App CMS

external hosting
app engine etc

OPEN IN OPEN OUT

Web servers

Solr
Proxy
App App servers

App Memcached Solr

App Solr
App CMS Solr
Solr
App
Solr
App rdbms

Cloud, EC2
external hosting
app engine etc

C
I O

CONTENT

r
external Clo
C
I O

???????

r
external Clo

Thank you
http://www.guardian.co.uk/open-platform
Twitter: @openplatform
@cuica (Stephen Dunn)


From Publisher To Platform: How The Guardian Used Content, Search, and Open Source To Build a Powerful New Business Model

More Related Content

Similar to From Publisher To Platform: How The Guardian Used Content, Search, and Open Source To Build a Powerful New Business Model (14)

Recently uploaded (20)

From Publisher To Platform: How The Guardian Used Content, Search, and Open Source To Build a Powerful New Business Model