2013 10-03-semantics-meetup-s buxton-mark_logic_pub

Copyright © 2013 MarkLogic® Corporation. All rights reserved.
© COPYRIGHT 2013 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.
Stephen Buxton
Lotico meetup, NYC
October 3rd, 2013
Semantics is the future of Search is the future of Semantics

Semantics is the future of Search is the future of Semantics
 Introduction: what does the title mean?
 MarkLogic: who's that?
 Imagine a World with Search AND Semantics
 Rich Search Applications … plus Semantics
 Semantics Applications … plus Search
 Bringing it all together: Use Cases
 Where To Next?

Two Hemispheres, One Brain

Two Hemispheres, One Brain
Triples:
 Highly structured
 Atomic
 Do one thing well
XML:
 Flexible structure
 Rich documents
 Rich applications

The MarkLogic Advantage
Search & Query
ACID Transactions
High Availability / Disaster Recovery
Government-grade Security
Semantics
Elasticity
Cloud Deployment
Hadoop for Storage & Compute
The Only Enterprise NoSQL Database

Real Value From Big Data
Make The World More Secure
Provide Access To Valuable Information
Create New Revenue Streams
Gain Insights to Increase Market Share
Reduce Bottom Line Expense

Dynamic Semantic Publishing
BBC Sports
 Size and Complexity:
 # of athletes
 # of teams
 # of assets (match
reports, statistics, etc.)
 # of relations (facts)
 Rich user experience
 See information in context
 Personalize content
 Easy navigation
 Intelligently serve ads
(outside of UK)
 Manageable
 Static pages?
Too many, too fast-changing
 Limited number of journalists
 Automate as much as possible
The Challenge Goals

Dynamic Semantic Publishing

Dynamic Semantic Publishing: A Solution
 Store, manage documents
 Stories
 Blogs
 Feeds
 Profiles
 Store, manage values
 Statistics
 Full-Text search
 Performance, scalability
 Robustness
 Metadata about documents
 Tagged by journalists
 Added (semi-)automatically
 Inferred
 Facts reported by journalists
 Real-world facts from the
Open Data Web
Document Database Triple Store

 At query time, dynamically aggregate stories, blogs, feeds,
images, profiles, results, statistics, videos for a particular
concept such as "West Ham".
(See Jem Rayfield, BBC, http://bbc.in/I1NdkB)
 we are not publishing pages, but publishing
content as assets which are then organized by the
metadata dynamically into pages
(John O'Donovan, BBC and PA)
Document Database Triple Store

 At query time, dynamically aggregate stories, blogs, feeds,
images, profiles, results, statistics, videos for a particular
concept such as "Chelsea".
(See Jem Rayfield, BBC, http://bbc.in/I1NdkB)
 we are not publishing pages, but publishing
content as assets which are then organized by the
metadata dynamically into pages
(John O'Donovan, BBC and PA)
Document Database Triple Storewith

Semantics Architecture
TRIPLE
XQY XSLT SQL SPARQL
GRAPH
SPARQL

Why Semantic Technologies?
 Triples are atomic – easy to create, manage, combine
 Semantic Web shares data as triples
 A natural choice for metadata and real-world facts
 .. and facts embedded in a document
 Standards encourage tools and sharing
 Graph model – easy to follow links
 Ontologies – infer new facts
Because …

Why Semantics and Search?
 Many use cases need documents, triples, and data
 One database means a simple, efficient, powerful architecture
 Combination queries – query documents, triples, data in a
single query – open up new possibilities
Because …

2013 10-03-semantics-meetup-s buxton-mark_logic_pub

Rich Search Applications .. Made Richer
Rich Search Application:
• 60 million docs indexed
• Links to email messages
• Snippets about e-mail
• Sender data
• Facets
• Search trends

Rich Search Applications .. Made Richer
Name: John Smith
Affiliation: IBM
Timezone: PST
Committer: Hadoop

Search With Real-World Context

The World of Triples
Linked Open Data
(Free semantic facts available to anyone)
Proprietary Semantic Facts
(Facts and Taxonomies in your organization) SemanticWorld
DocumentWorld
Facts from Free-Flowing Text
(Derived from semantic enrichment)
MarkLogic
Facts in Documents
(Part of metadata or added with authoring tools)

Triples from free-flowing text
assumed risk of
acquired by

Semantics + Search [1]

How do you deal with …
 Provenance – what's the source? And its source? And so on ..
 Reification – what's my confidence? Author? Date?
 Bi-Temporal – two date ranges
Answer questions like:
 Which countries did Nixon visit?
 .. before 1974?
 .. only show me answers where I have at least 80% confidence
 .. and the source is AP Newswire OR BBC

Triples and Documents
Documents can contain triples
<article>
<meta>
<title>Man bites dog</title>
<sem:triple>
<sem:subject>http://example.org/news/42</sem:subject>
<sem:predicate>http://example.org/published</sem:predicate>
<sem:object>2013-09-10</sem:object>
</sem:triple>
…

Triples can be annotated in documents
<source>AP Newswire</source>
<sem:triple date="1972-02-21" confidence="100">
<sem:subject>http://example.org/news/Nixon</sem:subject>
<sem:predicate>http://example.org/wentTo</sem:predicate>
<sem:object>China</sem:object>
</sem:triple>
…

Documents can contain triples
<article>
<meta>
<title>Man bites dog</title>
<sem:triple>
<sem:subject>http://example.org/news/42</sem:subject>
<sem:predicate>http://example.org/published</sem:predicate>
<sem:object>2013-09-10</sem:object>
</sem:triple>
… Triples can be annotated in documents
<source>AP Newswire</source>
<sem:triple date="1972-02-21" confidence="100">
<sem:subject>http://example.org/news/Nixon</sem:subject>
<sem:predicate>http://example.org/wentTo</sem:predicate>
<sem:object>China</sem:object>
</sem:triple>
…

import module namespace sem = "http://marklogic.com/semantics"
at "/MarkLogic/semantics.xqy";
sem:sparql('
SELECT ?country
WHERE {
<http://example.org/news/Nixon> <http://example.org/wentTo> ?country
}
',
(),
(),
cts:and-query( (
cts:path-range-query( "//sem:triple/@confidence", ">", 80) ,
cts:path-range-query( "//sem:triple/@date", "<", xs:date("1974-01-01")),
cts:or-query( (
cts:element-value-query( xs:QName("source"), "AP Newswire" ),
cts:element-value-query( xs:QName("source"), "BBC" )
) )
) )
)
Which countries did Nixon visit?
 .. before 1974?
 .. only show me answers where I have at least 80% confidence
 .. and the source is AP Newswire OR BBC

 Combination queries – simple, easy, efficient, powerful
 Query across triples, documents, data
 Leverage context
Answer questions like:
 I have a partial match on a license plate number –
a witness to a crime says it starts with ABC
 Are there any incident reports around the time of the crime,
around the location of the crime, that mention a black van?
 If yes, show me the names and addresses of the license holders

Combination Query: Example
<SAR>
<title>Suspicious vehicle…Suspicious vehicle near stadium
<date>
<type>
<threat>
2012-11-12Z
observation/surveillance
<type>suspicious activity
<category>suspicious vehicle
<location>
<lat>37.497075
<long>-122.363319
<subject>IRIID
<subject>IRIID
<predicate>
<predicate>
isa
value
<triple>
<triple>
<object>license-plate
<object>ABC 123
<description>A black van…A black van with license plate ABC 123 was observed parked behind the airport sign…
</title>
</date>
</type>
</type>
</category>
</threat>
</lat>
</long>
</location>
</subject>
</subject>
</predicate>
</predicate>
</object>
</object>
</description>
</SAR>
</triple>
</triple>
XML document – a SAR report

Combination Query: Example
<SAR>
<title>
Suspicious vehicle…
<date>
2012-11-12Z
<type>
<threat>
suspicious activity
<category>
suspicious vehicle
<location>
<lat>
37.497075
<long>
-122.363319
<description>
A black van…
<subject>
<subject>
<predicate>
<object>
IRIID
IRIID
isa
value
license-plate
ABC 123<predicate>
<object>
observation/surveillance
<type>
<triple>
<triple>

Combination Queries
 SPARQL -centric:
 SPARQL with XQuery built-in functions
 SPARQL with a search argument
 SPARQL with variable bindings
 SPARQL with forest-ids
 XQuery -centric:
Inside an XQuery program:
 sem:sparql( sparql query, search criteria )
 cts:triples( subject, predicate, object , search criteria)
 cts:triple-range-query( subject, predicate, object , [=,<,>] )

Semantics and Search – Use Cases
 Publishers and media organizations look to use the combined
Enterprise NoSQL database and triple store as a way to speed
implementation of Dynamic Semantic Publishing.
 Financial institutions query triples in the context of a document
to manage Reference Data and understand risk.
 Intelligence, law enforcement, fraud investigators and analysts
are able to discover connections and patterns in facts and
documents to find bad guys faster.
 Procurement agents are creating decision support tools to
rationalize purchasing decisions, vendors and bid management.
 Pharmaceutical companies and governing organizations are
using facts and documents together to assess risks and to
decide which drug trials to invest in.

Search: Understand, Discover, Make decisions
Search Fetch
documents
Extract relevant
facts
Analyze
Old
Search Fetch facts, data,
and documents
in context
Analyze and
annotate
Fetch supporting
facts, data, and documents
As
needed
New

Understand, Discover, Make Decisions

http://www.marklogic.com/summit-series/

Any Questions?
stephen.buxton@marklogic.com

2013 10-03-semantics-meetup-s buxton-mark_logic_pub

More Related Content

What's hot (6)

Similar to 2013 10-03-semantics-meetup-s buxton-mark_logic_pub (20)

Recently uploaded (20)

2013 10-03-semantics-meetup-s buxton-mark_logic_pub

Editor's Notes