Noun Sense Induction and Disambiguation using Graph-Based Distributional Semantics

Seminar at University of Leipzig
8 September, 2016, Leipzig, Germany
Noun Sense Induction and Disambiguation using
Graph-Based Distributional Semantics
Alexander Panchenko, Johannes Simon, Martin Riedl and Chris Biemann
Technische Universität Darmstadt, LT Group, Computer Science Department, Germany
September 7, 2016 | 1

Summary
▶ Panchenko A., Simon J., Riedl M., Biemann C. "Noun Sense Induction and
Disambiguation using Graph-Based Distributional Semantics". In
Proceedings of the KONVENS 2016, Bochum, Germany
▶ An approach to word sense induction and disambiguation.
▶ The method is unsupervised and knowledge-free.
▶ Sense induction by clustering of word similarity networks
▶ Feature aggregation w.r.t. the induced inventory.
▶ Comparable to the state-of-the-art unsupervised WSD (SemEval’13
participants and various sense embeddings).
▶ Open source implementation: github.com/tudarmstadt-lt/JoSimText

Motivation for Unsupervised Knowledge-Free
Word Sense Disambiguation
▶ A word sense disambiguation (WSD) system:
▶ Input: word and its context.
▶ Output: a sense of this word.
▶ Surveys: Agirre and Edmonds (2007) and Navigli (2009).
▶ Knowledge-based approaches that rely on hand-crafted resources, such as
WordNet.
▶ Supervised approaches learn from hand-labeled training data, such as SemCor.
▶ Problem 1: hand-crafted lexical resources and training data are expensive to
create, often inconsistent and domain-dependent.
▶ Problem 2: These methods assume a ﬁxed sense inventory:
▶ senses emerge and disappear over time.
▶ different applications require different granularities the sense inventory.
▶ An alternative route is the unsupervised knowledge-free approach.
▶ learn an interpretable sense inventory
▶ learn a disambiguation model

Contribution
▶ The contribution is a framework that relies on induced inventories as a pivot
for learning contextual feature representations and disambiguation.
▶ We rely on the JoBimText framework and distributional semantics (Biemann
and Riedl, 2013) adding a word sense disambiguation functionality on top of it.
▶ The advantage of our method, compared to prior art, is that it can integrate
several types of context features in an unsupervised way.
▶ The method achieves state-of-the-art results in unsupervised WSD.

Method: Data-Driven Noun Sense Modelling
1. Computation of a distributional thesaurus
▶ using distributional semantics
2. Word sense induction
▶ using ego-network clustering of related words
3. Building a disambiguation model of the induced senses
▶ by feature aggregation w.r.t. the induced sense inventory

Method: Distributional Thesaurus of Nouns us-
ing the JoBimText framework
▶ A distributional thesaurus (DT) is a graph of word similarities, such as
“(Python, Java, 0.781)”.
▶ We used the JoBimText framework (Biemann and Riedl, 2013):
▶ efﬁcient computation of nearest neighbours for all words
▶ providing state-of-the-art performance (Riedl, 2016)
▶ For each noun in the corpus get 200 most similar nouns

Method: Distributional Thesaurus of Nouns us-
ing the JoBimText framework (cont.)
▶ For each noun in the corpus get l = 200 most similar nouns:
1. Extract word, feature and word-feature frequencies.
▶ Using dependency-based features, such as amod(•, grilled) or prep_for(•,
dinner) using the Malt parser (Nivre et al., 2007)
▶ Collapsing of dependencies in the same way as the Stanford dependencies.
2. Discard rare words, features and word-features (t < 3).
3. Normalize word-feature scores using the Local Mutual Information (LMI):
LMI(i, j) = fij · PMI(i, j) = fij · log
fij
∑
i,j fij
fi∗ · f∗j
4. Ranking word features by LMI.
5. Prune all, but p = 1000 most signiﬁcant features per word.
6. Word similarities are computed as a number of common features for two words:
sim(ti , tj ) = |k : fik > 0 ∧ fjk > 0|
7. Return l = 200 most related words per word.

Method: Noun Sense Induction via Ego-Network
Clustering
▶ The "furniture" and the "data" sense clusters of the word "table".
▶ Graph clustering using the Chinese Whispers algorithm (Biemann, 2006).

Method: Noun Sense Induction via Ego-Network
Clustering (cont.)
▶ Process one word per iteration
▶ Construct an ego-network of the word:
▶ use dependency-based distributional word similarities
▶ the ego-network size (N): the number of related words
▶ the ego-network connectivity (n): how strongly the neighbours are related; this
parameter controls granularity of sense inventory.
▶ Graph clustering using the Chinese Whispers algorithm.

Method: Disambiguation of Induced Noun
Senses
▶ Learning a disambiguation model P(si |C) for each of the induced senses
si ∈ S of the target word w in context C = {c1, ..., cm}.
▶ We use the Naïve Bayes model:
P(si |C) =
P(si )
∏|C|
j=1 P(cj |si )
P(c1, ..., cm)
,
▶ The best sense given the context C:
s∗
i = arg max
si ∈S
P(si )
|C|
∏
j=1
P(cj |si ).

Senses (cont.)
▶ The prior probability of each sense is computed based on the largest cluster
heuristic:
P(si ) =
|si |
∑
si ∈S |si |
.
▶ Extract sense representations by aggregation of features from all words of the
cluster si .
▶ Probability of the feature cj given the sense si :
P(cj |si ) =
1 − α
Λi
|si |
∑
k
λk
f(wk , cj )
f(wk )
+ α,
▶ To normalize the score we divide it by the sum of all the weights Λi =
∑|si |
k λk :
▶ α is a small number, e.g. 10−5
, added for smoothing.

Senses (cont.)
▶ To calculate a WSD model we need to extract from corpus:
1. the distributional thesaurus;
2. sense clusters;
3. word-feature frequencies.
▶ Sense representations are obtained by “averaging” of feature representations
of words in the sense clusters.

Feature Extraction: Single Models
▶ The method requires sparse word-feature counts f(wk , cj ).
▶ We demonstrate the approach on the four following types of features:
1. Features based on sense clusters: Cluster
▶ Features: words from the induced sense clusters;
▶ Weights: similarity scores.
2. Dependency features: Deptarget, Depall
▶ Features: syntactic dependencies attached to the word, e.g. “subj(•,type)” or
“amod(digital,•)”
▶ Weights: LMI scores of the scores.
3. Dependency word features: Depword
▶ Features: words extracted from all syntactic dependencies attached to a target word.
For instance, the feature “subj(•,write)” would result in the feature “write”.
▶ Weights: LMI scores.
4. Trigram features: Trigramtarget, Trigramall
▶ Features: pairs of left and right words around the target word, e.g. “typing_•_or” and
“digital_•_.”.
▶ Weights: LMI scores.

Feature Combination: Combined Models
▶ Feature-level Combination of Features
▶ Union context features of different types, such as dependencies and trigrams.
▶ “Stack” feature spaces.
▶ Meta-level Combination of Features
1. Independent sense classifications by single models
2. Aggregation of predictions with:
▶ Majority selects the sense si selected by the largest number of single models.
▶ Ranks. First, results of single model classification are ranked by their confidence
ˆP(si |C): the most suitable sense to the context obtains rank one and so on. Finally,
we assign the sense with the least sum of ranks.
▶ Sum. This strategy assigns the sense with the largest sum of classification
confidences i.e.,
∑
i
ˆP(si |Ci
k ), where i is the number of the single model.

Corpora used for experiments
# Tokens Size Text Type
Wikipedia 1.863 · 109
11.79 Gb encyclopaedic
ukWaC 1.980 · 109
12.05 Gb Web pages
Table: Corpora used for training our models.

Results: Evaluation on the “Python-Ruby-
Jaguar” (PRJ) dataset: 3 words, 60 contexts, 2
senses per word
▶ A simple dataset: 60 contexts, 2 homonyms per word.
▶ The models based on the meta-combinations are not shown for brevity as they
did not improve performance of the presented models in terms of F-score.

Results: Evaluation on the TWSI dataset: 1012
nouns, 145140 contexts, 2.33 senses per word

Results: the TWSI dataset: effect of the corpus
choice on the WSD performance
▶ 10 best models according to the F-score on the TWSI dataset
▶ Trained on Wikipedia and ukWaC corpora

Results: Evaluation on the SemEval 2013 Task
13 dataset: 20 nouns, 1848 contexts

Thank you!

Word Embeddings for WSD using Graph-Based
Distributional Semantics
▶ Pelevina M., Areﬁev N., Biemann C., Panchenko A. "Making Sense of Word
Embeddings". In Proceedings of the 1st Workshop on Representation
Learning for NLP. ACL 2016, Berlin, Germany. Best Paper Award
▶ An approach to learn word sense embeddings.
▶ The same approach as presented above, but using word2vec instead of
JoBimText: dense vs sparse feature representations.

Overview of the contribution
Prior methods:
▶ Induce inventory by clustering of word instances (Li and Jurafsky, 2015)
▶ Use existing inventories (Rothe and Schütze, 2015)
Our method:
▶ Input: word embeddings
▶ Output: word sense embeddings
▶ Word sense induction by clustering of word ego-networks
▶ Word sense disambiguation based on the induced sense representations

Learning Word Sense Embeddings

Word Sense Induction: Ego-Network Clustering
▶ The "furniture" and the "data" sense clusters of the word "table".
▶ Graph clustering using the Chinese Whispers algorithm (Biemann, 2006).

Neighbours of Word and Sense Vectors
Vector Nearest Neighbours
table
tray, bottom, diagram, bucket, brackets, stack, bas-
ket, list, parenthesis, cup, trays, pile, playﬁeld,
bracket, pot, drop-down, cue, plate
table#0
leftmost#0, column#1, randomly#0, tableau#1, top-
left0, indent#1, bracket#3, pointer#0, footer#1, cur-
sor#1, diagram#0, grid#0
table#1
pile#1, stool#1, tray#0, basket#0, bowl#1, bucket#0,
box#0, cage#0, saucer#3, mirror#1, birdcage#0,
hole#0, pan#1, lid#0
▶ Neighbours of the word “table" and its senses produced by our method.
▶ The neighbours of the initial vector belong to both senses.
▶ The neighbours of the sense vectors are sense-speciﬁc.

Word Sense Disambiguation
1. Context Extraction
▶ use context words around the target word
2. Context Filtering
▶ based on context word’s relevance for disambiguation
3. Sense Choice
▶ maximize similarity between context vector and sense vector

Word Sense Disambiguation: Example

Evaluation on SemEval 2013 Task 13 dataset:
comparison to the state-of-the-art
Model Jacc. Tau WNDCG F.NMI F.B-Cubed
AI-KU (add1000) 0.176 0.609 0.205 0.033 0.317
AI-KU 0.176 0.619 0.393 0.066 0.382
AI-KU (remove5-add1000) 0.228 0.654 0.330 0.040 0.463
Unimelb (5p) 0.198 0.623 0.374 0.056 0.475
Unimelb (50k) 0.198 0.633 0.384 0.060 0.494
UoS (#WN senses) 0.171 0.600 0.298 0.046 0.186
UoS (top-3) 0.220 0.637 0.370 0.044 0.451
La Sapienza (1) 0.131 0.544 0.332 – –
La Sapienza (2) 0.131 0.535 0.394 – –
AdaGram, α = 0.05, 100 dim 0.274 0.644 0.318 0.058 0.470
w2v 0.197 0.615 0.291 0.011 0.615
w2v (nouns) 0.179 0.626 0.304 0.011 0.623
JBT 0.205 0.624 0.291 0.017 0.598
JBT (nouns) 0.198 0.643 0.310 0.031 0.595
TWSI (nouns) 0.215 0.651 0.318 0.030 0.573

Conclusion
▶ Novel approach for learning word sense embeddings.
▶ Can use existing word embeddings as input.
▶ WSD performance comparable to the state-of-the-art systems.
▶ Source code and pre-trained models:
https://github.com/tudarmstadt-lt/SenseGram

Evaluation based on the TWSI dataset: a large-
scale dataset for development

Thank you!

Noun Sense Induction and Disambiguation using Graph-Based Distributional Semantics

More Related Content

Similar to Noun Sense Induction and Disambiguation using Graph-Based Distributional Semantics (20)

More from Alexander Panchenko (18)

Recently uploaded (20)

Noun Sense Induction and Disambiguation using Graph-Based Distributional Semantics