The Magic of Text Summarization using Deep Networks

The Magic of Text Summariza3on
using Deep Networks
SK Reddy
skreddy99@gmail.com
12 Sep 2017

hGps://www.meetup.com/Silicon-Valley-Big-Data-Science/events/242784774/?rv=co1

2
Ref: Text summariza3on with TensorFlow; hGps://research.googleblog.com/2016/08/text-summariza3on-with-tensorﬂow.html

Building blocks
•  Word embeddings
•  RNNs, LSTMs
•  Seq2Seq
•  Encoder – Decoder
•  AGen3on
•  Beam Search
•  ROUGE
3

4
An example of the t-SNE word embedding plot

5
Wiki Ar'cle Embeddings
Ref: hGps://colah.github.io/posts/2015-01-Visualizing-Representa3ons/

Ref: hGp://colah.github.io/posts/2015-08-Understanding-LSTMs/
LSTM
ft it
ot
6

Seq2Seq
Ref: Sequence to Sequence Learning with Neural Networks; hGps://arxiv.org/pdf/1409.3215.pdf
where is input sequence
is corresponding output sequence
Note: LSTMs learn much better when the source sentences are reversed

Encoder-Decoder framework
Hidden state
Output vector
inputs
Encoder
Decoder
reads the inputs
predicts the next word
Ref: NEURAL MACHINE TRANSLATION BY JOINTLY LEARNING TO ALIGN AND TRANSLATE; hGps://arxiv.org/pdf/1409.0473.pdf 8

AGen3on
9 Ref: NEURAL MACHINE TRANSLATION BY JOINTLY LEARNING TO ALIGN AND TRANSLATE; hGps://arxiv.org/pdf/1409.0473.pdf

Ref: A Tutorial on Deep Learning by Quoc V. Le ; hGps://cs.stanford.edu/~quocle/tutorial2.pdf
yi is the ground truth during training and
is the previous output during predic3on
Greedy search: Best predic3on of the previous 3me-step is the input for the current 3me-step
Full search: Joint probability of the output sequence is highest
Beam search: Top-k joint probabili3es
Greedy search vs. Full search vs. Beam search during predic3on
How to have an RNN take x as input and
arrive at a sequence y1y2y3?

Approaches to summariza3on
•  Seq2Seq
•  BiDirec3onal Seq2Seq
•  Encoder-Decoder
•  Encoder AGen3on
•  Decoder AGen3on
•  Re-feed Decoder
•  Copy from input
•  Re-read inputs
•  Reinforcement Learning
11

Read again in encoder
Copy words in decoder
Ref: EFFICIENT SUMMARIZATION WITH READ-AGAIN AND COPY MECHANISM, Zeng et al; hGps://arxiv.org/pdf/1611.03382.pdf 12
Encoder
Decoder
Read again & copy model
(Nov ‘16)

Read again encoder (Nov ‘16)
Hierarchical Read again encoder
(for mul'ple sentences)
Ref: EFFICIENT SUMMARIZATION WITH READ-AGAIN AND COPY MECHANISM, Zeng
et al; hGps://arxiv.org/pdf/1611.03382.pdf
13

14 Ref: Get To The Point: Summariza3on with Pointer-Generator Networks; hGps://arxiv.org/pdf/1704.04368.pdf
Pointer-Generator model
(Apr ‘17)

Ref: A Deep Reinforced Model for Abstrac3ve Summariza3on; R Paulus, C Xiong, R Socher; hGps://arxiv.org/pdf/1705.04304.pdf
Neural Intra-AKen'on Model (May ’17)
15
To avoid repe33on, set
for a tri-gram
Reinforcement
Learning

Latest research (Sep 2017)
•  Auto summariza3on of online debates and topic iden3ﬁca3on
•  Text summariza3on of video discussions using sub3tles
•  Coarse to ﬁne tuned headline crea3on (hGp://sta3c.ijcai.org/proceedings-2017/0574.pdf)
16

The Magic of Text Summarization using Deep Networks

More Related Content

What's hot (8)

Similar to The Magic of Text Summarization using Deep Networks (20)

More from SK Reddy (17)

Recently uploaded (20)

The Magic of Text Summarization using Deep Networks