![Neural Language Model - Word2Vec
Word embedding stands for a continuous vector representation
able to capture syntactic and semantic information of a word.
vec(“Gilmour”) = [0.1, 0.3, 0.2, ...]
vec(“Barrett”) = [0.3, 0.1, 0.6, ...]
...
Figure: Continuous bag-of-words and Skip-gram [Mikolov et al., 2013]
vec(Barrett) − vec(singer) + vec(guitarist) ≈ vec(Gilmour)
Gaetano Rossiello, Pierpaolo Basile, Giovanni Semeraro Centroid-based Text Summarization using Word Embeddings](https://image.slidesharecdn.com/multiling17slide-180221143146/85/Centroid-based-Text-Summarization-through-Compositionality-of-Word-Embeddings-6-320.jpg)
![Centroid-based Text Summarization: Overview
Centroid-based Extractive Text Summarization [Radev et al., 2004]
The centroid represents a pseudo-document which condenses
the meaningful information of a document (tfidf (w) > t)
The main idea is to project in the vector space the vector
representations of both the centroid and each sentence of a
document
The sentences closer to the centroid are selected
Gaetano Rossiello, Pierpaolo Basile, Giovanni Semeraro Centroid-based Text Summarization using Word Embeddings](https://image.slidesharecdn.com/multiling17slide-180221143146/85/Centroid-based-Text-Summarization-through-Compositionality-of-Word-Embeddings-7-320.jpg)
![Sentence Scoring using Word Embeddings
Word Embeddings Lookup Table
Given a corpus of documents [D1, D2, . . . ]a and its vocabulary V
with size N = |V |, we define a matrix E ∈ RN,k, so-called lookup
table, where the i-th row is a word embedding of size k, k << N,
of the i-th word in V .
a
The model can be trained on the collection of documents to be summarized
or on a larger corpus.
Given a document D to be summarized:
Preprocessing: split into sentences, remove stopwords, no stemming
Centroid Embedding: C = w∈D,tfidf (w)>t E[idx(w)]
Sentence Embedding: Sj = w∈Sj
E[idx(w)]
Sentence Score: sim(C, Sj ) =
CT •Sj
||C||·||Sj ||
Gaetano Rossiello, Pierpaolo Basile, Giovanni Semeraro Centroid-based Text Summarization using Word Embeddings](https://image.slidesharecdn.com/multiling17slide-180221143146/85/Centroid-based-Text-Summarization-through-Compositionality-of-Word-Embeddings-8-320.jpg)
![Sentence Selection
Input: S, Scores, st, limit
Output: Summary
1: S ← sortDesc(S,Scores)
2: k ← 1
3: for i ← 1 to m do
4: length ← length(Summary)
5: if length > limit then return Summary
6: SV ← sumVectors(S[i])
7: include ← True
8: for j ← 1 to k do
9: SV 2 ← sumVectors(Summary[j])
10: sim ← similarity(SV ,SV 2)
11: if sim > st then
12: include ← False
13: if include then
14: Summary[k] ← S[i]
15: k ← k + 1
Gaetano Rossiello, Pierpaolo Basile, Giovanni Semeraro Centroid-based Text Summarization using Word Embeddings](https://image.slidesharecdn.com/multiling17slide-180221143146/85/Centroid-based-Text-Summarization-through-Compositionality-of-Word-Embeddings-9-320.jpg)
![Text Summarization using Word Embeddings: Example
Figure: Embeddings visualization using t-SNE [van der Maaten et al., 2008]
Gaetano Rossiello, Pierpaolo Basile, Giovanni Semeraro Centroid-based Text Summarization using Word Embeddings](https://image.slidesharecdn.com/multiling17slide-180221143146/85/Centroid-based-Text-Summarization-through-Compositionality-of-Word-Embeddings-10-320.jpg)
Centroid-based Text Summarization through Compositionality of Word Embeddings
- 1. Centroid-based Text Summarization through Compositionality of Word Embeddings Gaetano Rossiello, Pierpaolo Basile, Giovanni Semeraro gaetano.rossiello@uniba.it Department of Computer Science University of Bari - Aldo Moro, Italy MultiLing 2017 Workshop in EACL 2017 Summarization and summary evaluation across source types and genres 3 April 2017 - Valencia, Spain
- 2. Introduction Extractive Text Summarization The generated summary is a selection of relevant sentences from the source text in a copy-paste fashion. A good extractive summarization method must satisfy: Coverage: the selected sentences should cover a sufficient amount of topics from the original source text Diversity: avoid the redundancy of information in the summary Gaetano Rossiello, Pierpaolo Basile, Giovanni Semeraro Centroid-based Text Summarization using Word Embeddings
- 3. Extractive Text Summarization An extractive text summarization method should define: Representation model: a paradigm to represent the sentences Scoring method: a technique for assigning a score to each sentence Ranking module: a method to properly select the relevant sentences Bag-of-Words Several summarization methods proposed in the literature use the bag-of-words as representation model for the sentence scoring and selection modules. Gaetano Rossiello, Pierpaolo Basile, Giovanni Semeraro Centroid-based Text Summarization using Word Embeddings
- 4. Limit of the Bag-of-Words The textual similarity is a crucial aspect for many extractive text summarization methods. For example, taking into account two semantically related phrases: S1 Syd leaves Pink Floyd S2 Barrett abandons the band abandons band Barrett Floyd leaves Pink Syd the S1 0 0 0 1 1 1 1 0 S2 1 1 1 0 0 0 0 1 S1 ⊥ S2 =⇒ cosine(S1, S2) = 0 In the BOW model their vector (sparse) representation result orthogonal since they have no words in common. Gaetano Rossiello, Pierpaolo Basile, Giovanni Semeraro Centroid-based Text Summarization using Word Embeddings
- 5. The Idea Humans use a wide background knowledge when writing a summary. What Representation Learning How Distributional Semantics Why Transfer Learning Gaetano Rossiello, Pierpaolo Basile, Giovanni Semeraro Centroid-based Text Summarization using Word Embeddings
- 6. Neural Language Model - Word2Vec Word embedding stands for a continuous vector representation able to capture syntactic and semantic information of a word. vec(“Gilmour”) = [0.1, 0.3, 0.2, ...] vec(“Barrett”) = [0.3, 0.1, 0.6, ...] ... Figure: Continuous bag-of-words and Skip-gram [Mikolov et al., 2013] vec(Barrett) − vec(singer) + vec(guitarist) ≈ vec(Gilmour) Gaetano Rossiello, Pierpaolo Basile, Giovanni Semeraro Centroid-based Text Summarization using Word Embeddings
- 7. Centroid-based Text Summarization: Overview Centroid-based Extractive Text Summarization [Radev et al., 2004] The centroid represents a pseudo-document which condenses the meaningful information of a document (tfidf (w) > t) The main idea is to project in the vector space the vector representations of both the centroid and each sentence of a document The sentences closer to the centroid are selected Gaetano Rossiello, Pierpaolo Basile, Giovanni Semeraro Centroid-based Text Summarization using Word Embeddings
- 8. Sentence Scoring using Word Embeddings Word Embeddings Lookup Table Given a corpus of documents [D1, D2, . . . ]a and its vocabulary V with size N = |V |, we define a matrix E ∈ RN,k, so-called lookup table, where the i-th row is a word embedding of size k, k << N, of the i-th word in V . a The model can be trained on the collection of documents to be summarized or on a larger corpus. Given a document D to be summarized: Preprocessing: split into sentences, remove stopwords, no stemming Centroid Embedding: C = w∈D,tfidf (w)>t E[idx(w)] Sentence Embedding: Sj = w∈Sj E[idx(w)] Sentence Score: sim(C, Sj ) = CT •Sj ||C||·||Sj || Gaetano Rossiello, Pierpaolo Basile, Giovanni Semeraro Centroid-based Text Summarization using Word Embeddings
- 9. Sentence Selection Input: S, Scores, st, limit Output: Summary 1: S ← sortDesc(S,Scores) 2: k ← 1 3: for i ← 1 to m do 4: length ← length(Summary) 5: if length > limit then return Summary 6: SV ← sumVectors(S[i]) 7: include ← True 8: for j ← 1 to k do 9: SV 2 ← sumVectors(Summary[j]) 10: sim ← similarity(SV ,SV 2) 11: if sim > st then 12: include ← False 13: if include then 14: Summary[k] ← S[i] 15: k ← k + 1 Gaetano Rossiello, Pierpaolo Basile, Giovanni Semeraro Centroid-based Text Summarization using Word Embeddings
- 10. Text Summarization using Word Embeddings: Example Figure: Embeddings visualization using t-SNE [van der Maaten et al., 2008] Gaetano Rossiello, Pierpaolo Basile, Giovanni Semeraro Centroid-based Text Summarization using Word Embeddings
- 11. Text Summarization using Word Embeddings: Example arcade donkey kong game nintendo coleco centroid embedding arcades goat hong gameplay mario intellivision nes pac-man pig macao multiplayer wii atari gamecube console monkey fung videogame console nes konami famicom horse taiwan rpg nes msx wii sega cow wong gamespot gamecube 3do famicom Table: Centroid words of the Donkey Kong (video game) article having the tf-idf values greater than a topic threshold Sent ID Sentence Score 136 The original arcade version of the game appears in the Nintendo 64 game Donkey Kong 64. 0.9533 131 The game was ported to Nintendo’s Family Computer (Famicom) console in 1983 as one of the system’s three launch titles; the same version was a launch title for the Famicom’s North American version, the Nintendo Entertainment System (NES). 0.9375 186 In 2004, Nintendo released Mario vs. Donkey Kong, a sequel to the Game Boy title. 0.9366 192 In 2007, Donkey Kong Barrel Blast was released for the Nintendo Wii. 0.9362 135 The NES version was re-released as an unlockable game in Animal Crossing for the GameCube and as an item for purchase on the Wii’s Virtual Console. 0.9308 Table: The most relevant sentences Gaetano Rossiello, Pierpaolo Basile, Giovanni Semeraro Centroid-based Text Summarization using Word Embeddings
- 12. Experiments Research Question Can word embeddings improve the effectiveness of the centroid-based text summarization method? Python implementation on GitHub: github.com/gaetangate/text-summarizer/ DUC-2004 Multi-document Summarization task 2 Tuning Grid search on DUC-2003 dataset Word2Vec CBOW, Skip-gram on DUC-03/04 - Pre-trained on Google News MSS 2015 Multilingual Single Document task 2015 Tuning Grid search on MSS 2015 training set Word2Vec Skip-gram on Wikipedia (en, it, de, es, fr) MSS 2017 Multilingual Single Document task 2017 Tuning Grid search on MSS 2017 training set Word2Vec Skip-gram on Wikipedia (en, it, de, es, fr) Gaetano Rossiello, Pierpaolo Basile, Giovanni Semeraro Centroid-based Text Summarization using Word Embeddings
- 13. DUC-2004 task 2: Multi-document Summarization ROUGE-1 ROUGE-2 tt st size LEAD 32.42 6.42 SumBasic 37.27 8.58 Peer65 38.22 9.18 NMF 31.60 6.31 LexRank 37.58 8.78 RNN 38.78 9.86 C BOW 37.76 8.08 0.1 0.6 C GNEWS 37.91 8.45 0.2 0.9 300 C CBOW 38.68 8.93 0.3 0.93 200 C SKIP 38.81 9.97 0.3 0.94 400 Table: ROUGE scores (%) on DUC-2004 dataset. tt and st are the topic and similarity thresholds respectively. size is the dimension of embeddings Gaetano Rossiello, Pierpaolo Basile, Giovanni Semeraro Centroid-based Text Summarization using Word Embeddings
- 14. DUC-2004 task 2: Multi-document Summarization Although the different methods achieve similar ROUGE scores, they not necessarily generate similar summaries. GNEWS CBOW SKIP BOW GNEWS 1 0.109 0.171 0.075 CBOW 1 0.460 0.072 SKIP 1 0.105 BOW 1 Table: Sentence overlap using the Jaccard coefficient Gaetano Rossiello, Pierpaolo Basile, Giovanni Semeraro Centroid-based Text Summarization using Word Embeddings
- 15. MSS 2015: Multilingual Single Document Summarization English Italian German Spanish French R1 R2 R1 R2 R1 R2 R1 R2 R1 R2 LEAD 44.33 11.68 30.46 4.38 29.13 3.21 43.02 9.17 42.73 8.07 WORST 37.17 9.93 39.68 10.01 33.02 4.88 45.20 13.04 46.68 12.96 BEST 50.38 15.10 43.87 12.50 40.58 8.80 53.23 17.86 51.39 15.38 C BOW 49.06 13.43 33.44 4.82 35.28 4.93 48.38 12.88 46.13 10.45 C W2V 50.43‡ 13.34† 35.12 6.81 35.38† 5.39† 49.25† 12.99 47.82† 12.15 ORACLE 61.91 22.42 53.31 17.51 54.34 13.32 62.55 22.36 58.68 17.18 Table: ROUGE-1, -2 scores (%) on MultiLing MSS 2015 dataset for five different languages Gaetano Rossiello, Pierpaolo Basile, Giovanni Semeraro Centroid-based Text Summarization using Word Embeddings
- 16. Improvements for MSS 2017 SWAP system in MSS 2017 task: Always retain the first sentence in the summary Subtract to each word embedding the centroid vector of the whole embedding space Combination of four scores: sc1 word2vec centroid similarity sc2 bag-of-words centroid similarity sc3 normalized sentence length sc4 normalized sentence position score(Sj ) = λ1 ∗ sc1 + λ2 ∗ sc2 + λ3 ∗ sc3 + λ4 ∗ sc4 Gaetano Rossiello, Pierpaolo Basile, Giovanni Semeraro Centroid-based Text Summarization using Word Embeddings
- 17. MSS 2017: Multilingual Single Document Summarization English Italian German Spanish French R1 MM R1 MM R1 MM R1 MM R1 MM CIST 45.06 16.83 30.07 20.22 32.32 16.52 45.31 16.94 41.67 17.66 TeamMD 43.08 16.35 30.22 20.79 32.91 15.76 44.95 16.54 42.81 17.18 SWAP 45.62 17.05 32.66 18.45 35.15 18.27 46.67 20.58 43.68 20.06 ORACLE 55.52 20.98 41.25 26.89 41.58 23.75 52.20 23.16 51.41 25.44 Table: ROUGE-1, MeMog scores (%) on MultiLing MSS 2017 dataset for five different languages Gaetano Rossiello, Pierpaolo Basile, Giovanni Semeraro Centroid-based Text Summarization using Word Embeddings
- 18. Future Works Try word embeddings with other summarization methods Sentence embeddings using Deep Learning Doc2Vec Autoencoders Convolutional Neural Networks Recurrent Neural Networks (LSTM, GRU) Attention and Memory Networks Joint Learning of Distributional and Relational Semantics WordNet DBpedia Wikidata ... Infuse prior knowledge in sentence embeddings Compositionality of Concepts Transfer Learning Gaetano Rossiello, Pierpaolo Basile, Giovanni Semeraro Centroid-based Text Summarization using Word Embeddings
- 19. Moral of the Story Math > Magic Gaetano Rossiello, Pierpaolo Basile, Giovanni Semeraro Centroid-based Text Summarization using Word Embeddings