Introduction to question answering for linked data & big data

© Copyright 2009 Digital Enterprise Research Institute. All rights reserved.
Digital Enterprise Research Institute www.deri.ie
Question Answering over
Linked Data & Big Data
André Freitas
LdB – Semantics: Theory, Technologies and Applications
Bari, September 2013

Goal of this Talk
 Understand the changes on the database landscape in
the direction of more heterogeneous data scenarios.
 Understand how Question Answering (QA) fits into
this new scenario.
 Give the fundamental pointers to develop your own
QA system from the state-of-the-art.
 Coverage over depth.
2

Outline
 Motivation & Context
 Big Data & QA
 Are NLIs useful for Databases?
 The Anatomy of a QA System
 Evaluation of QA Systems
 QA over Linked Data
 Treo: Detailed Case Study
 Do-it-yourself (DIY): Core Resources
 QA over Linked Data: Roadmaps
 From QA to Semantic Applications
(Deriving Patterns)
 Take-away Message
3

Motivation &
Context
4

Why Question Answering?
5
 Humans are built-in with
natural language
communication
capabilities.
 Very natural way for
humans to communicate
information needs.

What is Question Answering?
 A research field on its own.
 Empirical bias: Focus on the development of
automatic systems to answer questions.
 Multidisciplinary:
 Natural Language Processing
 Information Retrieval
 Knowledge Representation
 Databases
 Linguistics
 Artificial Intelligence
 Software Engineering
 ...
6

What is Question Answering?
7
QA
System
Knowledge
Bases
Question: Who is the
daughter of Bill Clinton
married to?
Answer: Marc
Mezvinsky

QA vs IR
 Keyword Search:
 User still carries the major efforts in interpreting the data.
 Satisfying information needs may depend on multiple search
operations.
 Answer-driven information access.
 Input: Keyword search
– Typically specification of simpler information needs.
 Output: documents, data.
 QA:
 Delegates more ‘interpretation effort’ to the machines.
 Query-driven information access.
 Input: natural language query
– Specification of complex information needs.
 Output: direct answer.
8

QA vs Databases
 Structured Queries:
 A priori user effort in understanding the schemas behind
databases.
 Effort in mastering the syntax of a query language.
 Satisfying information needs may depend on multiple
querying operations.
 Input: Structured query
 Output: data records, aggregations, etc
 QA:
 Delegates more ‘interpretation effort’ to the machines.
 Input: natural language query
 Output: direct answer
9

When to use?
 Keyword search:
 Simple information needs.
 Predictable search behavior.
 Vocabulary redundancy (large document collections, Web)
 Structured queries:
 Precision/recall guarantees.
 Small & centralized schemas.
 More data volume/less semantic heterogeneity.
 QA:
 Specification of complex information needs.
 More automated semantic interpretation.
10

QA: Vision
11

12
QA: Reality (FB Graph Search)

13
QA: Reality (Watson)

14
QA: Reality (Watson)

15
QA: Reality (Siri)

To Summarize
 QA is usually associated with the delegation of
more of the‘interpretation effort’ to the machines.
 QA supports the specification of more complex
information needs.
 QA, Information Retrieval and Databases are
complementary.
16

Big Data & QA
17

Big Data
 Big Data: More complete data-based picture of the
world.
18

From Rigid Schema to Schemaless
10s-100s attributes
1,000s-1,000,000s attributes
 Heterogeneous, complex and large-scale
databases.
 Very-large and dynamic “schemas”.
19
circa 2000
circa 2013

Mutiple Interpretations
 Multiple perspectives (conceptualizations) of the reality.
 Ambiguity, vagueness, inconsistency.
20

Big Data & Dataspaces
 Franklin et al. (2005): From Databases to Dataspaces.
 Helland (2011): If You Have Too Much Data, then
“Good Enough” Is Good Enough.
 Fundamental trends:
 Co-existence of heterogeneous data.
 Semantic best-effort queries.
 Pay-as-you go data integration.
 Co-existent query/search services.
21

Big Data & NoSQL
 From Relational to NoSQL Databases.
 NoSQL (Not only SQL).
 Four trends (Emil Eifrem)
 Trend 1: Data set size.
 Trend 2: Connectedness.
 Trends 3 & 4: Semi-structured data & decentralized
architecture.
22
Eifrem, A NOSQL Overview And The Benefits Of Graph Database (2009)

Trend 1: Data size
23
Gantz & Reinsel, The Digital Universe in 2020 (2012).

Trend 2: Connectedness
24
Eifrem, A NOSQL Overview And The Benefits Of Graph Database (2009).

Trends 3 & 4: Semi-structured data
 Individualization of content.
 Decentralization of content generation.
25

Emerging NoSQL Platforms
 Key-value stores
 Data model: collection of key values (K/V) pairs.
 Example: Voldemort.
 BigTables clones
 Data model: Big Table.
 Example: Hbase, Hypertable.
 Document databases
 Data Model: Collections of K/V collections.
 Example: MongoDB, CouchDB.
 Graph databases
 Data Model: nodes, edges.
 Example: AllegroGraph, VertexDB, Neo4J, Semantic Web DBs.
26

NoSQL Databases
27
Size
Complexity
Key-value
stores
Bigtable clones
Document databases
Graph databases
Eifrem, A NOSQL Overview And The Benefits Of Graph Database (2009).

Big Data
 Volume
 Velocity
 Variety
28
- The most interesting but usually neglected
dimension.

Big Data & Linked Data
 Linked Data is Big Data (Variety)
 Entity-Attribute-Value (EAV) Data Model.
 Entities:
 Identifiers
 Attributes
 Attribute Values
 Linked Data as a special type of the EAV Data Model:
 EAV/CR: Classes and Relations.
 URIs as a superkey.
 Dereferentiable URIs.
 Standards-based (RDF/HTTP).
 EAV as an anti-pattern.
29
Idehen, Understanding Linked Data via EAV Model (2010).

Big Data & Linked Data
30

Big Data: Structured queries
31
 Big Problem: Structured queries are still the primary way to query
databases.

Big Data: Structured queries
10-100s
attributes
103
-106
s
attributes
32
Query construction size x schema size

Query/Search Spectrum
Adapted from Kauffman et al (2009)
Structured
queries
33

Vocabulary Problem for Databases
Who is the daughter of Bill Clinton married to?
Schema-agnostic queriesSemantic Gap
Possible representations
34

Who is the daughter of Bill Clinton married to ?
Semantic Gap
Lexical-level
Abstraction-level
Structural-level
35

Semantic Gap
Lexical-level
Abstraction-level
Structural-level
Query:
Data
36

Semantic Gap
Lexical-level
Abstraction-level
Structural-level
Query:
Data
37
Popescu et al. (2003): Semantic tractability

38
Big Data & QAs
QA Big Data
semantic gap,
vocabulary gap
QA Big Data
distributional
semantics

To Summarize
 Schema size and heterogeneity represent a
fundamental shift for databases.
 Addressing the associated data management
challenges (specially querying) depends on the
development of principled semantic models for
databases.
 QA/Natural Language Interfaces (NLIs) as schema-
agnostic query mechanisms.
39

Are NLIs useful for
Databases?
40

41
NLI & QAs
 Natural Language Interfaces (NLI)
 Input: Natural language queries
 Output: Either processed or unprocessed queries
– Processed: Direct answers.
– Unprocessed: Database records, text snippets, documents.
NLI
QA

42
 Kaufmann & Bernstein (2007).
 User study based on 4 different types of
interfaces:
1. NLP-Reduce: Simple keyword-based NLI
– Domain-independent
– Bag of Words
– WordNet-based query expansion
2. Querix: More complex NLI
– Domain-independent
– Query parsing
– WordNet-based query expansion
– Clarification dialogs
Comparative study

43
 Kaufmann & Bernstein (2007).
 User study based on 4 different types of
interfaces:
3. Ginseng: Guided input NLI
– Grammar-based suggestions
4. Semantic Crystal: Visual query interface
– Graphically displayable and clickable
Comparative study

44
NLP-Reduce
Kaufmann & Bernstein, How Useful are Natural Language Interfaces to the Semantic Web for Casual
End-users? (2007).

45
Querix
End-users? (2007).

46
Ginseng
End-users? (2007).

47
Semantic Crystal
End-users? (2007).

48
 48 subjects (different areas and ages).
 4 questions.
 Query metrics + System Usability Scale (SUS).
 Tang & Mooney Dataset.
User study
End-users? (2007).

49
 48 subjects (different areas and ages).
 4 questions.
 Query metrics + System Usability Scale (SUS).
 Tang & Mooney Dataset.
User study
Brooke, SUS - A quick and dirty usability scale.

50
 Querix: Full English question input was judged to
be the most useful and best-liked query interface.
 Users appreciated the “freedom” given by full
natural language queries.
 Possible interpretations:
 No need to convert natural language to keyword search.
 More complete expression of information needs.
 Limitations:
 Number of queries
 Database size
Results

The Anatomy of a QA
System
51

Basic Concepts & Taxonomy
 Categorization of questions and answers.
 Important for:
 Understanding the challenges before attacking
the problem.
 Scoping the system.
 Based on:
 Chin-Yew Lin: Question Answering.
 Farah Benamara: Question Answering Systems: State of
the Art and Future Directions.
52

Terminology: Question Phrase
 The part of the question that says what is
being asked:
 Wh-words:
– who, what, which, when, where, why, and how
 Wh-words + nouns, adjectives or adverbs:
– •“which party …”, “which actress …”, “how long …”,
“how tall …”.
53

Terminology: Question Type
 Useful for distinguishing different processing
strategies
 FACTOID: “Who is the wife of Barack Obama?”
 LIST: “Give me all cities in the US with less than 10000
inhabitants.”
 DEFINITION: “Who was Tom Jobim?”
 RELATIONSHIP: “What is the connection between Barack
Obama and Indonesia?”
 SUPERLATIVE: “What is the highest mountain?”
 YES-NO: “Was Margaret Thatcher a chemist?”
 OPINION: “What do most Americans think of gun control?”
 CAUSE & EFFECT: “Why did the revenue of IBM drop?”
54

Terminology: Answer Type
 The class of object sought by the question
 Person (from “Who …”)
 Place (from “Where …”)
 Process & Method (from “How …”)
 Date (from “When …”)
 Number (from “How many …”)
 Explanation & Justification (from “Why …”)
55

Terminology: Question Focus & Topic
 Question focus is the property or entity that is
being sought by the question
 “In which city Barack Obama was born?”
 “What is the population of Galway?”
 Question topic: What the question is generally about :
 “What is the height of Mount Everest?” (geography, mountains)
 “Which organ is affected by the Meniere’s disease?” (medicine)
56

Terminology: Data Source Type
 Structure level:
 Structured data (databases)
 Semi-structured data (e.g. comment field in
databases, XML)
 Free text
 Data source:
 Single (Centralized)
 Multiple
 Web-scale
57

Terminology: Domain Type
 Domain Scope:
 Open Domain
 Domain specific
 Data Type:
 Text
 Image
 Sound
 Video
 Multi-modal QA
58

Terminology: Answer Format
 Long answers
 Definition/justification based.
 Short answers
 Phrases.
 Exact answers
 Named entities, numbers, aggregate, yes/no
59

Answer Quality Criteria
 Relevance: The level in which the answer addresses
users information needs.
 Correctness: The level in which the answer is
factually correct.
 Conciseness: The answer should not contain
irrelevant information.
 Completeness: The answer should be complete.
 Simplicity: The answer should be simple to be
interpreted by the data consumer.
 Justification: Sufficient context should be provided
to support the data consumer in the determination of
the query correctness.

Answer Assessment
 Right: The answer is correct and complete.
 Inexact: The answer is incomplete or incorrect.
 Unsupported: the answers does not have an
appropriate justification.
 Wrong: The answer is not appropriate for the
question.

Answer Processing
 Simple Extraction: cut and paste of snippets from
the original document(s) / records from the data.
 Combination: Combines excerpts from multiple
sentences, documents / multiple data records,
databases.
 Summarization: Synthesis from large texts / data
collections.
 Operational/functional: Depends on the application
of functional operators.
 Reasoning: Depends on the an inference process.
62

Complexity of the QA Task
 Semantic Tractability (Popescu et al., 2003):
Vocabulary distance between the query and the
answer.
 Answer Locality (Webber et al., 2003): Whether
answer fragments are distributed across different
document fragments/documents or datasets/dataset
records.
 Derivability (Webber et al, 2003): Dependent if the
answer is explicit or implicit. Level of reasoning
dependency.
 Semantic Complexity: Level of ambiguity and
discourse/data heterogeneity.
63

Main Components
64
Question
Analysis
Search
Answer
Extraction
Question Keyword
query
Query
features
Data
records/
Passages
Answers
Documents/
Datasets

Main Components
 Question Analysis: Includes question parsing,
extraction of core features (NER, answer type, etc).
 Search (i.e. passage retrieval): Pre-selection of
fragments of text (sentences, paragraphs, documents)
and data (records, datasets) which may contain the
answer.
 Answer Extraction: Processing of the answer based
on the passages.
65

Evaluation of
QA Systems
66

Evaluation Campaigns
 Test Collection
 Questions
 Datasets
 Answers (Gold-standard)
 Evaluation Measures

Recall
 Measures how complete is the answer set.
 The fraction of relevant instances that are retrieved.
 Which are the Jovian planets in the Solar System?
 Returned Answers:
– Mercury
– Jupiter
– Saturn
 Gold-standard:
– Jupiter
– Saturn
– Neptune
– Uranus
Recall = 2/4 = 0.5

Precision
 Measures how accurate is the answer set.
 The fraction of retrieved instances that are relevant.
– Mercury
– Jupiter
– Saturn
 Gold-standard:
– Jupiter
– Saturn
– Neptune
– Uranus
Recall = 2/3 = 0.67

Mean Reciprocal Rank (MRR)
 Measures the ranking quality.
 The Reciprocal-Rank (1/r) of a query can be defined as the rank r
at which a system returns the first relevant entity.
– Mercury
– Jupiter
– Saturn
 Gold-standard:
– Jupiter
– Saturn
– Neptune
– Uranus
rr = 1/2 = 0.5

Mean Average Interpolated Precision
(MAiP)
 Computes the Interpolated-Precision at a set of n standard recall
levels (1%, 10%, 20%, etc).
 Average-Interpolated-Precision (AiP) is a single-valued measure
that reflects the performance of a search engine over all the
relevant results.
 Mean-Average-Interpolated-Precision (MAiP) that reflects the
performance of a system over all the results.

Normalized Discounted
Cumulative Gain (NDCG)
 Discounted-Cumulative-Gain (DCG) uses a graded relevance scale
to measure the gain of a system based on the positions of the
relevant entities in the result set.
 This measure gives a lower gain to relevant entities returned in
the lower ranks to that of the higher ranks.

Test Collections
 Question Answering over Linked Data (QALD-CLEF)
 INEX Linked Data Track
 BioASQ
 SemSearch

QALD

QALD-1, ESWC (2011)
 Datasets:
 Dbpedia 3.6 (RDF)
 MusicBrainz (RDF)
 Tasks:
 Training questions: 50 questions for each dataset
 Test questions: 50 questions for each dataset
http://greententacle.techfak.uni-bielefeld.de/~cunger/qald/index.php?x=challenge&q=1

QALD-1, ESWC (2011)

QALD-1, ESWC (2011)
 Which presidents were born in 1945?
 Who developed the video game World of Warcraft?
 List all episodes of the first season of the HBO television series The
Sopranos!
 Who produced the most films?
 Which mountains are higher than the Nanga Parbat?
 Give me all actors starring in Batman Begins.
 Which software has been developed by organizations founded in
California?
 Which companies work in the aerospace industry as well as on nuclear
reactor technology?
 Is Christian Bale starring in Batman Begins?
 Give me the websites of companies with more than 500000 employees.
 Which cities have more than 2 million inhabitants?

QALD-2, ESWC (2012)
 Datasets:
 Tasks:
 Training questions: 100 questions for each dataset
 Test questions: 50 questions for each dataset
http://greententacle.techfak.uni-bielefeld.de/~cunger/qald/index.php?x=challenge&q=1

QALD-3, CLEF (2013)
 Datasets:
 Tasks:
 Multilingual QA
– Given a RDF dataset and a natural language question or set
of keywords in one of six languages (English, Spanish,
German, Italian, French, Dutch), either return the correct
answers, or a SPARQL query that retrieves these answers.
 Ontology Lexicalization
http://greententacle.techfak.uni-bielefeld.de/~cunger/qald/index.php?x=task1&q=3

INEX Linked Data Track (2013)
 Focuses on the combination of textual and structured data.
 Datasets:
 English Wikipedia (MediaWiki XML Format)
 DBpedia 3.8 & YAGO2 (RDF)
 Links among the Wikipedia, DBpedia 3.8, and YAGO2 URI's.
 Tasks:
 Ad-hoc Task: return a ranked list of results in response to a search
topic that is formulated as a keyword query (144 search topics).
 Jeopardy Task: Investigate retrieval techniques over a set of natural-
language Jeopardy clues (105 search topics – 74 (2012) + 31
(2013)).
https://inex.mmci.uni-saarland.de/tracks/lod/

INEX Linked Data Track (2013)

SemSearch Challenge
 Focuses on entity search over Linked Datasets.
 Datasets:
 Sample of Linked Data crawled from publicly available
sources (based on the Billion Triple Challenge 2009).
 Tasks:
 Entity Search: Queries that refer to one particular entity. Tiny
sample of Yahoo! Search Query.
 List Search: The goal of this track is select objects that match
particular criteria. These queries have been hand-written by
the organizing committee.
http://semsearch.yahoo.com/datasets.php#

SemSearch Challenge
 List Search queries:
 republics of the former Yugoslavia
 ten ancient Greek city
 kingdoms of Cyprus
 the four of the companions of the prophet
 Japanese-born players who have played in MLB where the
British monarch is also head of state
 nations where Portuguese is an official language
 bishops who sat in the House of Lords
 Apollo astronauts who walked on the Moon

SemSearch Challenge
 Entity Search queries:
 1978 cj5 jeep
 employment agencies w. 14th street
 nyc zip code
 waterville Maine
 LOS ANGELES CALIFORNIA
 ibm
 KARL BENZ
 MIT

 Datasets:
 PubMed documents
 Tasks:
 1a: Large-Scale Online Biomedical Semantic Indexing
– Automatic annotation of PubMed documents.
– Training data is provided.
 1b: Introductory Biomedical Semantic QA
– 300 questions and related material (concepts, triples and
golden answers).

Baseline
Balog & Neumayer, A Test Collection for Entity Search in Dbpedia (2013).

Going Deeper
 Metrics, Statistics, Tests - Tetsuya Sakai
 http://www.promise-noe.eu/documents/10156/26e7f254-1feb-41
 Building test Collections (IR Evaluation - Ian Soboroff)
 http://www.promise-noe.eu/documents/10156/951b6dfb-a404-46

QA over
Linked Data
88

The Semantic Web Vision
2001:
Software which is able to
understand meaning (intelligent,
flexible)
Leveraging the Web for
information scale

The Semantic Web Vision
 What was the plan to
achieve it?
 Build a Semantic Web
Stack
 Which covers both
representation and
reasoning

Reality Check
 Adoption:
 No significant data
growth
 Ontologies are not
straightforward to
build:
 People are not
familiriazed with the
tools and principles
 Difficult to keep
consistency at Web scale
 Scalability

Reasoning
 Problems:
 Consistecy
 Scalability
Logic World Web World

Linked Data
 The Web as a Huge Database
 Fundamental step for data
creation
2006:

No Reasoning, no Fun?
 Where are the intelligence and
flexibility?
 We will be back to this point
in a minute

From which university did the wife of
Barack Obama graduate?
Consuming/Using Linked Data
 With Linked Data we are still in the DB world

Consuming/Using Linked Data
 With Linked Data we are still in the DB world
 (but slightly worse)

Linked Data
 Data Model Features:
 Graph-based data model
 Extensible schema
 Entity-centric data integration
 Specific Features:
 Designed over open Web standards
 Based on the Web infrastructure (HTTP, URIs)

Linked Data
 Positives:
 Solid adoption in the Open Data context
(eGovernment, eScience, etc,...)
 Existing data is relevant (you can build real
applications)
 Negatives:
 Data consumption is a problem
 Data generation beyond databases
mapping/triplification is also a problem
 Still far from the Semantic Web vision

QA for Linked Data
 Addresses practical problems of data
accessibility in a data heterogeneity
scenario.
 A fundamental part of the original
Semantic Web vision.
99

QA4LD Requirements
 High usability:
 Supporting natural language queries.
 High expressivity:
 Path, conjunctions, disjunctions, aggregations, conditions.
 Accurate & comprehensive semantic matching:
 High precision and recall.
 Low maintainability:
 Easily transportable across datasets from different domains
(minimum adaptation effort/low adaptation time).
 Low query execution time:
 Suitable for interactive querying.
 High scalability:
 Scalable to a large number of datasets (Organization-scale, Web-
scale).
100

Exemplar QA Systems
 Aqualog & PowerAqua (Lopez et al. 2006)
 ORAKEL (Cimiano et al, 2007)
 QuestIO & Freya (Damljanovic et al. 2010)
 Treo (Freitas et al. 2011, 2012)
101

PowerAqua (Lopez et al. 2006)
 Key contribution: semantic similarity
mapping.
 Terminological Matching:
 WordNet-based
 Ontology Based
 String similarity
 Sense-based similarity matcher
 Evaluation: QALD (2011).
 Extends the AquaLog system.
102

WordNet
103

Sense-based similarity matcher
 Two words are strongly similar if any of the following
holds:
 1. They have a synset in common (e.g. “human” and “person”)
 2. A word is a hypernym/hyponym in the taxonomy of the
other word.
 3. If there exists an allowable “is-a” path connecting a synset
associated with each word.
 4. If any of the previous cases is true and the definition
(gloss) of one of the synsets of the word (or its direct
hypernyms/hyponyms) includes the other word as one of its
synonyms, we said that they are highly similar.
104
Lopez et al. 2006

PowerAqua (Lopez et al. 2006)
105
Lopez et al. 2006

ORAKEL (Cimiano et al. 2007)
 Key contribution:
 FrameMapper lexicon engineering approach.
 Parsing strategy: Logical Description Grammars (LDGs).
– LDG is inspired by Lexicalized Tree Adjoining Grammars (LTAGs).
– An important characteristic of these trees is that they encapsulate
all syntactic/semantic arguments of a word.
 FrameMapper
 Evaluation:
 Dataset: domain-specific
 Dimensions: Relevance, Performance, lexicon construction
106

 More sophisticated parsing strategy: Logical
Description Grammars (LDGs).
 LDG is inspired by Lexicalized Tree Adjoining
Grammars (LTAGs).
 An important characteristic of these trees is that
they encapsulate all syntactic/semantic
arguments of a word.
107

108

109

Freya (Damljanovic et al. 2010)
 Key contribution: user interaction (dialogs)
for terminological matching.
 WordNet & Cyc (synonyms)
 String similarity (Monge Elkan + Sundex)
 User feedback
 Extends the QuestIO system.
 Evaluation: Mooney (2010), QALD (2011)
110

Freya (Damljanovic et al. 2010)
111
Ontology-based
Gazeteer (GATE)
SPARQL
Generation
Ontology
Query
Analysis
Synonym
(WordNet + Cyc)

Other QA Approaches
 Unger et al. (2012), Template-based
Question Answering over RDF Data.
 Cabrio et al. (2012), QAKIS: An Open
Domain QA System based on Relational
Patterns.
112

Treo: Detailed Case
Study
113

Treo (Irish): Direction, path

Treo (Freitas et al. 2011, 2012)
 Key contribution:
 Distributional semantic relatedness matching model
 Distributional relational space.
 Explicit Semantic Analysis (ESA)
 WordNet
 String similarity + node cardinality
 Evaluation: QALD (2011)
115

Statistical
analysis
Linked Data

Query Pre-Processing
(Question Analysis)
 Transform natural language queries into
triple patterns
“Who is the daughter of Bill Clinton married to?”

(Question Analysis)
 Step 1: POS Tagging
 Who/WP
 is/VBZ
 the/DT
 daughter/NN
 of/IN
 Bill/NNP
 Clinton/NNP
 married/VBN
 to/TO
 ?/.

(Question Analysis)
 Step 2: Core Entity Recognition
 Rules-based: POS Tag + TF/IDF
(PROBABLY AN INSTANCE)

(Question Analysis)
 Step 3: Determine answer type
 Rules-based.
(PERSON)

(Question Analysis)
 Step 4: Dependency parsing
 dep(married-8, Who-1)
 auxpass(married-8, is-2)
 det(daughter-4, the-3)
 nsubjpass(married-8, daughter-4)
 prep(daughter-4, of-5)
 nn(Clinton-7, Bill-6)
 pobj(of-5, Clinton-7)
 root(ROOT-0, married-8)
 xcomp(married-8, to-9)

(Question Analysis)
 Step 5: Determine Partial Ordered Dependency
Structure (PODS)
 Rules based.
– Remove stop words.
– Merge words into entities.
– Reorder structure from core entity position.
Bill Clinton daughter married to
(INSTANCE)
Person
ANSWER
TYPE
QUESTION FOCUS

(Question Analysis)
 Step 5: Determine Partial Ordered Dependency
Structure (PODS)
 Rules based.
– Remove stop words.
– Merge words into entities.
– Reorder structure from core entity position.
Bill Clinton daughter married to
(INSTANCE)
Person
(PREDICATE) (PREDICATE) Query Features

Query Planning
 Map query features into a query plan.
 A query plan contains a sequence of:
 Search operations.
 Navigation operations.
(INSTANCE) (PREDICATE) (PREDICATE) Query Features
 (1) INSTANCE SEARCH (Bill Clinton)
 (2) DISAMBIGUATE ENTITY TYPE
 (3) GENERATE ENTITY FACETS
 (4) p1 <- SEARCH RELATED PREDICATE (Bill Clintion, daughter)
 (5) e1 <- GET ASSOCIATED ENTITIES (Bill Clintion, p1)
 (6) p2 <- SEARCH RELATED PREDICATE (e1, married to)
 (7) e2 <- GET ASSOCIATED ENTITIES (e1, p2)
 (8) POST PROCESS (Bill Clintion, e1, p1, e2, p2)
Query Plan

Core Entity Search
 Entity Index:
 Construction of entity index (instances, classes and complex
classes).
 Extract terms from URIs and index the terms using an inverted
index.
 Search instances by keywords
 Entity Search (Instance Example):
 Input: Keyword search (Bill Clinton).
 Ranking by: string similarity and entity cardinality.
 Output: List of URIs.
 Core rationale:
 Prioritize the matching of less ambiguous/polysemic entities.
 Prioritize more popular entities.

Core Entity Search
Bill Clinton daughter married to Person
:Bill_Clinton
Query:
Linked
Data:
Entity Search

Core Entity Search
 Entity Disambiguation:
 Get entity types.
 Compute distributional semantic relatedness between entity
types:
– Explicit Semantic Analysis (ESA).
– semantic relatedness (Bill Clinton, President) = 0.11
– semantic relatedness (Bill Clinton, Politician)= 0.052
– semantic relatedness (Bill Clinton, Actor) = 0.011
 Generate Entity Types facets.
http://vmdeb14.deri.ie:8080/esa/service?task=esa&term1=bill%20clinton&term2=politician

Distributional Semantic Search
:Bill_Clinton
Query:
Linked
Data:
:Chelsea_Clinton
:child
:Baptists
:religion
:Yale_Law_School
:almaMater
...
(PIVOT ENTITY)
(ASSOCIATED
TRIPLES)

:Bill_Clinton
Query:
Linked
Data:
:Chelsea_Clinton
:child
:Baptists
:religion
:Yale_Law_School
:almaMater
...
sem_rel(daughter,child)=0.054
sem_rel(daughter,child)=0.004
sem_rel(daughter,alma mater)=0.001
Which properties are semantically related to ‘daughter’?

:Bill_Clinton
Query:
Linked
Data:
:Chelsea_Clinton
:child

Distributional Semantic Relatedness
 Computation of a measure of “semantic proximity”
between two terms.
 Allows a semantic approximate matching between
query terms and dataset terms.
 It supports a commonsense reasoning-like behavior
based on the knowledge embedded in the corpus.

 Use distributional semantics to semantically match
query terms to predicates and classes.
 Distributional principle: Words that co-occur together
tend to have related meaning.
 Allows the creation of a comprehensive semantic model from
unstructured text.
 Based on statistical patterns over large amounts of text.
 No human annotations.
 Distributional semantics can be used to compute a
semantic relatedness measure between two words.

:Bill_Clinton
Query:
Linked
Data:
:Chelsea_Clinton
:child
(PIVOT ENTITY)

:Bill_Clinton
Query:
Linked
Data:
:Chelsea_Clinton
:child
:Mark_Mezvinsky
:spouse

Semantic Relatedness
 Computation of a measure of “semantic proximity”
between two terms
 Allows a semantic approximate matching between
query terms and dataset terms
 It supports a reasoning-like behavior based on the
knowledge embedded in the corpus

 Use distributional semantics to semantically match
query terms to predicates and classes
 Distributional principle: Words that co-occur together
tend to have related meaning
 Allows the creation of a comprehensive semantic model from
unstructured text
 Based on statistical patterns over large amounts of text
 No human annotations
 Distributional semantics can be used to compute a
semantic relatedness measure between two words

Results

Post-Processing

Second Query Example
What is the highest mountain?
(CLASS) (OPERATOR) Query Features
mountain - highest PODS

Entity Search
Mountain highest
:Mountain
Query:
Linked
Data:
:typeOf
(PIVOT ENTITY)

Extensional Expansion
Mountain highest
:Mountain
Query:
Linked
Data:
:Everest
:typeOf
(PIVOT ENTITY)
:K2:typeOf
...

Distributional Semantic Matching
Mountain highest
:Mountain
Query:
Linked
Data:
:Everest
:typeOf
(PIVOT ENTITY)
:K2:typeOf
...
:elevation
:location
...
:deathPlaceOf

Get all numerical values
Mountain highest
:Mountain
Query:
Linked
Data:
:Everest
:typeOf
(PIVOT ENTITY)
:K2:typeOf
...
:elevation
:elevation
8848 m
8611 m

Apply operator functional definition
Mountain highest
:Mountain
Query:
Linked
Data:
:Everest
:typeOf
(PIVOT ENTITY)
:K2:typeOf
...
:elevation
:elevation
8848 m
8611 m
SORT
TOP_MOST

From Exact to Approximate
 Semantic approximation in databases (as in any IR
system): semantic best-effort.
 Need some level of user disambiguation,
refinement and feedback.
 As we move in the direction of semantic systems
we should expect the need for principled dialog
mechanisms (like in human communication).
 Pull the the user interaction back into the system.

From Exact to Approximate

User Feedback

Treo Architecture

Core Elements of Treo
 Hybrid model: database/IR/QA.
 Ranked query results.
 A distributional VSM for representing and semantically
processing relational data was formulated: Ƭ-Space.
 Similar in motivation to Cohen’s predication space.
 Distributional semantic relatedness as a primitive
operation.
150

-SpaceƬ
ESA
EAV
vector field
151

Simple Queries (Video)

More Complex Queries (Video)

Vocabulary Search (Video)

Treo Answers Jeopardy Queries (Video)

Video Links:
 Introducing Treo: Talk to your Data
 http://www.youtube.com/watch?v=Zor2X0uoKsM
 Treo: Do it your own (DIY) Jeopardy Question
Answering Engine
 http://www.youtube.com/watch?v=Vqh0r8GxYe8
 Treo: Semantic Search over Schema & Vocabularies
 http://www.youtube.com/watch?v=HCBwSV1mTdY

Evaluation: Relevance
Relevance
Avg. Precision Avg. Recall MRR % of queries
answered
0.62 0.81 0.49 80%
 Test Collection: QALD 2011.
 DBpedia 3.7 + YAGO.
 102 natural language queries.
Dataset: 45,767 predicates, 5,556,492 classes and 9,434,677
instances
157

Evaluation: Treo Evolution
Version Avg.
Precision
Avg.
Recall
MRR % of queries
answered
QA Query
exec. time
# of Queries
0.1 0.39 0.45 0.42 56% No QA > 2 min 50
0.2 0.48 0.49 0.52 58% No QA > 2 min 50
0.3 0.62 0.81 0.49 80% QA 8,530 s >102
158

Evaluation: Terminological Matching
Avg.
Precision@5
Avg.
Precision@10
MRR % of queries
answered
0.732 0.646 0.646 92.25%
Approach % of queries answered
ESA 92.25%
String matching 45.77%
String matching + WordNet QE 52.48%
159

Performance & Maintainability
160

Do-it-yourself (DIY):
Core Resources
161

Corpora: Wikipedia
 High domain coverage:
 ~95% of Jeopardy! Answers.
 ~98% of TREC answers.
 Wikipedia is entity-centric.
 Curated link structure.
 Where to use:
 Construction of distributional semantic models.
 As a commonsense KB
 Complementary tools:
 Wikipedia Miner

Linked Datasets
 DBpedia: Instances and data.
 YAGO: Classes and instances.
 Freebase: Instances.
 CIA Factbook: Data.
<http://dbpedia.org/class/yago/19th-centuryPresidentsOfTheUnitedStates>
<http://dbpedia.org/class/yago/4th-centuryBCGreekPeople>
<http://dbpedia.org/class/yago/OrganizationsEstablishedIn1918>
<http://dbpedia.org/class/yago/PeopleFromToronto>
<http://dbpedia.org/class/yago/JewishAtheists>
<http://dbpedia.org/class/yago/CountriesOfTheMediterraneanSea>
<http://dbpedia.org/class/yago/TennisPlayersAtThe1996SummerOlympics>
<http://dbpedia.org/class/yago/OlympicGoldMedalistsForTheUnitedStates>
<http://dbpedia.org/class/yago/WorldNo.1TennisPlayers>
<http://dbpedia.org/class/yago/PeopleAssociatedWithTheUniversityOfZurich>
<http://dbpedia.org/class/yago/SwissImmigrantsToTheUnitedStates>
<http://dbpedia.org/class/yago/1979VideoGames>

Linked Datasets
 DBpedia: Instances and data.
 YAGO: Classes and instances.
 Freebase: Instances.
 CIA Factbook: Data.
 Where to use:
 As a commonsense KB.

Dictionaries
 WordNet: Large Lexical Database
 Wikitionary: Dictionary
 Where to use:
 Query expansion.
 Semantic similarity.
 Semantic relatedness.
 Word sense disambiguation.
 Complementary tools:
 WordNet::Similarity

Parsers
 Stanford Parser: POS-Tagger, Syntactic &
Dependency Trees.
 Where to use:
 Question Analysis.

Named Entity Recognition/Resolution
 Stanford NER.
 DBpedia Spotlight
 Treo NER.
 Where to use:
 Question Analysis

Search Engines
 Lucene & Solr: Advanced search engine
framework.
 Treo -Space: Distributional semantic searchƬ
over structured data.
 Treo Entity Search: Semantic search engine for
retrieving individual entities and vocabulary
elements.

Distributional Semantics
 E2SA: High-performance Explicit Semantic
Analysis (ESA) framework (based on NoSQL).
 Semantic Vectors.
 Where to use:
 Semantic relatedness & similiarity.
 Word Sense Disambiguation.

Linked Data Extraction
 Fred: Represents the extracted data using
ontology patterns.
 Graphia: Extracts contextualized Linked Data
graphs.

QA over Linked Data: Roadm
171

Research Topics/Opportunities
 #1: Merge Linked Data extraction into QA4LD.

Motivation

Extraction Examples

Research Topics/Opportunities
 #1: Merge Linked Data extraction into QA4LD.
 #2: Explore complex dialog/context in QA4LD tasks.
 #3: Advance the use of distributional semantic models on
QA.
 #4: Provide the incentives for the advancement of open
robust software resources and high quality data.
(altimetrics.org).
 #5: Multilingual QA.
 #6: Creation of multi-sourced test collections.
 #7: Integration of reasoning (deductive, inductive,
counterfactual, abductive ...) on QA approaches and test
collections.
 #8: Development of QA approaches/tasks which use both
structured and unstructured data.

From QA to Semantic
Applications
(Deriving Patterns)

Semantic Application Patterns
 Derived from the experience developing Treo.
 Not restricted to QA over Linked Data.
 The following list is not intended to be complete.

 Pattern #1: Maximize the amount of knowledge in
your semantic application.
 Meaning interpretation depends on knowledge.
 Using LOD: DBpedia, Freebase, YAGO can give you
a very comprehensive set of instances and their
types.
 Wikipedia can provide you a comprehensive
distributional semantic model.

 Pattern #2: Allow your databases to grow.
 Dynamic schema.
 Entity-centric data integration.

 Pattern #3: Once the database grows in complexity
use semantic search instead of structured queries.
 Instances can be used as pivot entities to reduce
the search space
 They are easier to search.
 Higher specificity and lower vocabulary variation.

 Pattern #4: Use distributional semantics and
semantic relatedness for a robust semantic
matching.
 Distributional semantics allows your application to
digest (and make use of) large amounts of
unstructured information.
 Multilingual solution.
 Can be complemented with WordNet.

 Pattern #5: POS-Tags, Syntactic Parsing + Rules will
go a long way to help your application to interpret
natural language queries and sentences.
 Use them to explore the regularities in natural
language.
 Define a scope for natural language processing for
your application (restrict by domain, syntactic
complexity).
 These tools are easy to use and quite robust
(*English).

 Pattern #6: Provide a user dialog mechanism in the
application
 Improve the semantic model with user feedback.
 Record the user feedback.

Take-away Message
 Big Data/complex dataspaces demand new principled
semantic approaches to cope with the scale and
heterogeneity of data.
 Information systems in the future will depend on semantic
technologies. Be the first to develop.
 Part of the Semantic Web/AI vision can be addressed today
with a multi-disciplinary perspective:
 Linked Data, IR and NLP
 You can build your own IBM Watson-like application.
 Both data and tools are available and ready to use: the main
barrier is the mindset.
 Huge opportunity for new solutions.

References
[1] Eifrem, A NOSQL Overview And The Benefits Of Graph Database (2009)
[2] Idehen, Understanding Linked Data via EAV Model (2010).
[3] Kaufmann & Bernstein, How Useful are Natural Language Interfaces to the Semantic Web
for Casual End-users? (2007)
[4] Chin-Yew Lin, Question Answering.
[5] Farah Benamara, Question Answering Systems: State of the Art and Future Directions.
[6] Freitas et al., Querying Heterogeneous Datasets on the Linked Data Web: Challenges,
Approaches and Trends, 2012.
[7] Freitas et al, A Distributional Structured Semantic Space for Querying RDF Graph Data.,
2012.
[8] Freitas et al, A Distributional Approach for Terminological Semantic Search on the Linked
Data Web, 2012.
[9] Freitas et al, A Semantic Best-Effort Approach for Extracting Structured Discourse Graphs
from Wikipedia., 2012
[10]Freitas et al., Answering Natural Language Queries over Linked Data Graphs: A
Distributional Semantics Approach,, 2013.
[11] Freitas et al., Querying Heterogeneous Datasets on the Linked Data Web: Challenges,
Approaches and Trends, 2012.

References
[12] Cimiano et al., Towards portable natural language interfaces to knowledge bases, 2008.
[13] Lopez et al., PowerAqua: shing the semantic web, 2006.ﬁ
[14] Damljanovic et al., Natural Language Interfaces to Ontologies: Combining Syntactic Analysis
and Ontology-based Lookup through the User Interaction, 2010
[16] Unger et al. Template-based Question Answering over RDF Data, 2012.
[17] Cabrio et al., QAKiS: an Open Domain QA System based on Relational Patterns, 2012.
[18] How Useful Are Natural Language Interfaces to the Semantic Web for Casual End-Users?,
2007.
[19] Popescu et al.,Towards a theory of natural language interfaces to databases., 2003

192

Contact
http://andrefreitas.org
andre.freitas@deri.org

Introduction to question answering for linked data & big data

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to Introduction to question answering for linked data & big data (20)

More from Andre Freitas (20)

Recently uploaded (20)

Introduction to question answering for linked data & big data

Editor's Notes