Tutorial on Question Answering Systems

A Tutorial on
Question Answering Systems
Saeedeh Shekarpour
Postdoc researcher from EIS research group
31 August 2015EIS research group - Bonn University
1

Outline
 Introduction
 Associations for evaluation QA systems
 Preliminary Concepts
 Data Web
 Emerging Concepts
 Deeper view on SINA Project
2

Query
Answer
What is a Question Answering (QA)
system?
Systems that automatically answer questions posed by
humans in natural language query.
3
Natural language is the common
way for sharing knowledge

Question answering is a multi
disciplinary field
4
Semantic
Web
Linked Data

Difference to Information Retrieval
 Information Retrieval is a query driven approach for
accessing information.
 System returns a list of documents.
 It is responsibility of user to navigate on the retrieved
documents and find its own information need.
 Question Answering is an answer driven approach for
accessing information.
 User asks its question in natural language ( i.e. phrase-
based, full sentence or even keyword based) queries.
 System returns the list of short answers.
 More complex functionality.
5

Search engines are moving towards
QA
6

Search engines still lack the ability to
answer more complex queries
7

Natural language queries are
classified into different categories
 Factoid queries: WH questions like when, who, where.
 Yes/ No queries: Is Berlin capital of Germany?
 Definition queries: what is leukemia?
 Cause/consequence queries: How, Why, What. what are
the consequences of the Iraq war ?
8

Natural language queries are
classified into different categories
 Procedural queries: which are the steps for getting a
Master degree?
 Comparative queries: what are the differences between
the model A and B?
 Queries with examples: list of hard disks similar to hard
disk X.
 Queries about opinion: What is the opinion of the majority
of Americans about the Iraq war?
9

Corpus Type
 Structured data (relational data bases, RDF knowledge
bases).
 Semi-structured data (XML databases)
 Free text
 Multimodal data: image, voice, video
10

Types of QA systems
 Open-domain: domain independent QA systems can
answer any query from any corpus
+ covers wide range of queries
- low accuracy
 Closed-domain: domain specific QA systems are limited
to specific domains
+ High accuracy
- limited coverage over the possible queries
- Needs domain expert
11

Is QA system a need for user?
Search engines query log analysis shows that
Real informational queries in Google:
 Who first invented rock and roll music?
 When was the mobile phone invented?
 Where was the hamburger invented?
 How to lose weight?
12
Type of query Query log
analysis
Informational 48%
Navigational 20%
Transactional 30%

 Introduction
 Data Web
13

Text Retrieval Conference (Trec)
 In 1999, Trec initiated a QA track,
 From 1999-2002, participant systems were allowed to
return ranked answer snippets. These snippets of text
contain the actual answer.
 From 2002, participant systems were allowed to return
only the exact answer with confidence rate.
 The evaluation metric was mean reciprocal rank (MRR).
14

Typical pipeline for the participating
systems in Trec
15
Query
Expansion
IR
Approach
Passage
Retrieval
Answer
Selection
Predicting
the type
of answer
Answer
Query

QA at Clef
The Cross-Language Evaluation Forum (CLEF) initiative
provides
 Since 2003, Clef included a QA track.
 an infrastructure for the testing, tuning and evaluation of
information retrieval systems operating on European
languages in both monolingual and cross-language
contexts.
16

Outline
 Introduction
 Data Web
17

Core of a QA system
18
query

Syntactic Parsing: Part-of-speech
Tagging
I like sweet cakes
I/PRP like/VBP sweet/JJ cakes/NNS
19
like/VBP
I/PRP cakes/NNS
sweet/JJ

Type of Answer
Where was the hamburger invented?
White Castle traces the origin of the hamburger to
Hamburg, Germany with its invention by Otto Kuase.
20
Place

Named Entity Recognition on Query
where was Franklin Roosevelt born?
21
Named Entity: Person

Relation Extraction
Barack Hussein Obama is the 44th and current President of
the United States. Born in Honolulu, Hawaii, Obama is a
graduate of Columbia University and Harvard Law School.
22
Named Entity: Person
Named Entity: Place
Relation: President of

Watson Project
 Watson is a computer which is capable of answering
question issued in natural language.
 Questions come from quiz show called Jeopardy.
 The software of this project is called DeepQA project.
 In 2011, Watson won the former winners of quiz show
Jeopardy.
23

Watson Description
 Hardware: Watson system has 2,880 POWER processor
threads and has 16 terabytes of RAM.
 Data: encyclopedias, dictionaries, thesauri, newswire
articles, and literary works. Watson also used databases,
taxonomies, and ontologies such as DBPedia, WordNet, and
Yago were used.
 Software: DeepQA software and the Apache UIMA
framework. The system was written in various languages,
including Java, C++, and Prolog, and runs on the SUSE Linux
Enterprise Server 11 operating system using Apache Hadoop
framework to provide distributed computing.
24

The high level architecture of
DeepQA
25

Decomposition example
Query: Of the four countries in the world that the United
States does not have diplomatic relations with, the one
that’s farthest north.
Bhutan, Cuba, Iran, North Korea
North Korea
26

Outline
 Introduction
 Data Web
27

Evolution of Web
28

The growth of Linked Open Data
29
August 2014
570 Datasets
More than 74 billion triples
May 2007
12 Datasets

How to retrieve data from Linked
Data?
30

RDF Model
 RDF is an standard for describing Web resources.
 The RDF data model expresses statements about Web
resources in the form of subject-predicate-object (triple).
 The statement “Jack knows Alice” is represented as:
31
knowing

Outline
 Introduction
 Data Web
32

Semantic Indexing
33

Semantic Annotation
 Name Entity Recognition
Where is the capital of Germany?
 Semantic Annotation
Where is the capital of Germany?
34
Named Entity: Place
Entity:
http://dbpedia.org/resource/Germ
any

35
Berlin in the 1920s was the third largest municipality in the world.
After World War II, the city became divided into East Berlin -- the
capital of East Germany -- and West Berlin, a West German
exclave surrounded by the Berlin Wall from 1961–89.
Berlin in the 1920s was the third largest municipality in the world.
After World War II, the city became divided into East Berlin -- the
capital of East Germany -- and West Berlin, a West German
exclave surrounded by the Berlin Wall from 1961–89.

Relation extraction leveraging Data
Web: BOA Library
36

HAWK: Hybrid Question Answering
over Linked Data
37

TBSL: Template-based SPARQL
generator
38

Outline
 Introduction
 Data Web
39

SINAArchitecture
40
Client
QueryPreprocessing
QueryExpansion
ResourceRetrieval
Disambiguation
QueryConstruction
Representation
Server
UnderlyingInterlinked
KnowledgeBases
query result
keywords
valid segments
mapped resources
tuple of
resources
SPARQL
queries
OWL API
http client
Stanford
CoreNLP
SegmentValidation
Reformulated query

Comparison of search approaches
41
Data-semantic
unaware
Data-semantic
aware
Keyword-based
query

Objective: transformation from
textual query to formal query
42
Which televisions shows were created by Walt
Disney?
SELECT * WHERE
{ ?v0 a dbo:TelevisionShow.
?v0 dbo:creator dbr:Walt_Disney. }

The addressed challenges in SINA
43

Query Segmentation
44
 Definition: query segmentation is the process of identifying the right
segments of data items that occur in the keyword queries.

Resource Disambiguation
45
 Definition: resource disambiguation is the process of
recognizing the suitable resources in the underlying
knowledge base.
•Who produced films starring Natalie
Portman?
•dbpedia/ontology/film
•dbpedia/property/film
•What are the side effects of drugs used
for Tuberculosis?
•diseasome:Tuberculosis
•sider:Tuberculosis

Concurrent Approach
46
Query
Segmentation
Resource
Disambiguation

Modeling using hidden Markov
model
47
1
2
3
Unknown
Entity
4
5
6
7
8
9
Start
Keyword 1 Keyword 3Keyword 2 Keyword 4

Bootstrapping the model parameters
1. Emission probability is defined based on the similarity of the label of
each state with a segment, this similarity is computed based on
string-similarity and Jaccard-similarity.
2. Semantic relatedness is a base for transition probability and initial
probability. Intuitively, it is based on two values: distance and
connectivity degree. We transform these two values to hub and
authority values using weighted HITS algorithm.
3. HITS algorithm is a link analysis algorithm that was originally
developed for ranking Web pages. It assign a hub and authority
value to each web page.
4. Initial probability and transition probability are defined as a uniform
distribution over the hub and and authority values.
48

49
Sequence
of
keywords
(television show creat Walt Disney)
Paths 0.0023 dbo:TelevisionShow dbo:creator dbr:Walt_Disney
0.0014 dbo:TelevisionShow dbo:creator dbr:Category:Walt_Disney
0.000589 dbr:TelevisionShow dbo:creator dbr:Walt_Disney
0.000353 dbr:TelevisionShow dbo:creator dbr:Category:Walt_Disney
0.0000376 dbp:television dbp:show dbo:creator dbr:Category:Walt_Disney
Output of the model

Query Expansion
 Definition: query expansion is a way of reformulating the
input query in order to overcome the vocabulary
mismatch problem.
50
• Wife of Barak Obama
•Spouse of Barak Obama

Query Expansion
51

Linguistic features
 WordNet is a popular data source for expansion.
 Linguistic features extracted from WordNet are:
1. Synonyms: words having a similar meanings to the input
keyword.
2. Hyponyms: words representing a specialization of the input
keyword.
3. Hypernyms: words representing a generalization of the input
keyword.
52

Semantic features from Linked Data
53
1. SameAs: deriving resources using owl:sameAs.
2. SeeAlso: deriving resources using rdfs:seeAlso.
3. Equivalence class/property: deriving classes or properties
using owl:equivalentClass and owl:equivalentProperty.
4. Super class/property: deriving all super classes/properties of
by following the rdfs:subClassOf or rdfs:subPropertyOf
property.
5. Sub class/property: deriving resources by following the
rdfs:subClassOf or rdfs:subPropertyOf property paths ending
with the input resource.

Semantic features from Linked Data
5. Broader concepts: deriving using the SKOS vocabulary
properties skos:broader and skos:broadMatch.
6. Narrower concepts: deriving concepts using skos:narrower
and skos:narrowMatch.
7. Related concepts: deriving concepts using skos:closeMatch,
skos:mappingRelation and skos:exactMatch.
54

Exemplary expansion graph of the
word movie
55
movie
home
movie
production
film
motion
picture show
video
telefilm

Statistics over the number of the
derived words from WordNet and
Linked Data
56
Feature #derived words
synonym 503
hyponym 2703
hypernym 657
sameAs 2332
seeAlso 49
equivalence 2
super class/property 267
Sub class/property 2166

Automatic query expansion
57
Input query
External Data source
Data extraction and
preparation
Heuristic method
Reformulated query

Expansion set for each segment
58

Reformulating query using hidden
Markov model
59
Barak
Barak
Obama
spouse
Obama
wife
first
lady
woman
Barak Obama wife
Barak
Obama
Start
Obama
wife
Barak
Obama
wife
Input query: wife of Barak Obama

Formal Query Construction
 Definition: Once the resources are detected, a
connected subgraph of the knowledge base graph,
called the query graph, has to be determined which fully
covers the set of mapped resources.
60
Disambiguated
resources
sider:sideEffect
diseasome:possibleDrug
diseasome:1154
SPARQL query SELECT ?v3 WHERE {
diseasome:115 diseasome:possibleDrug ?v1 .
?v1 owl:sameAs ?v2 .
?v2 sider:sideEffect ?v3 .}

Data Fusion on Linked Data
 Answer of a question may be spread among different
datasets employing heterogeneous schemas.
 Constructing a federated query from needs to exploit
links between the different datasets on the schema and
instance levels.
61

Two different approaches
Formal query construction based on
 Template-based query construction
 Forward chaining based query construction
62

Federated query construction using
forward chaining
63
Query What is the side effects of drugs used for Tuberculosis?
resources diseasome:1154 (type instance)
diseasome:possibleDrug (type property)
sider:sideEffect (type property)
1154 ?v0
possibleDrug
Graph 1
?v1 ?v2
sideEffect
Graph 2
115
4
?v0
possibleDrug
Template 1
?v1 ?v2
sideEffect
Template 2
115
4
?v0
possibleDrug
?v1 ?v2
sideEffect

Any question?
64

Tutorial on Question Answering Systems

More Related Content

What's hot (20)

Viewers also liked (6)

Similar to Tutorial on Question Answering Systems (20)

More from Saeedeh Shekarpour (6)

Recently uploaded (20)

Tutorial on Question Answering Systems

Editor's Notes