Quoc Le, Software Engineer, Google at MLconf SF
Download as PPTX, PDF3 likes3,855 views
The document discusses sequence learning for language understanding, focusing on machine translation and the use of recurrent neural networks (RNNs). It highlights the advantages of encoding input sequences into vectors and decoding them back into output sequences, introducing techniques like beam search for improved output. A notable experiment achieved state-of-the-art results in machine translation using a new representation method that could benefit various language understanding tasks.
Quoc Le, Software Engineer, Google at MLconf SF
- 1. Sequence Learning for Language Understanding Presenter: Quoc V. Le Google Thanks: Andrew Dai, Jeff Dean, Matthieu Devin, Geoff Hinton, Thang Luong, Rajat Monga, Ilya Sutskever, Oriol Vinyals
- 2. Sequence Learning Typical success of Machine Learning: Mapping fixed length input to a scalar value: - Image recognition (Pixels -> “cat”) - Speech recognition (Waveforms -> the utterance of “cat”) Many language understanding problems require mapping from sequences to sequences: - Machine Translation (“I love music” -> “Je aime la musique”) Quoc V. Le
- 3. Sequence Learning Typical success of Machine Learning: Mapping fixed length input to a scalar value: - Image recognition (Pixels -> “cat”) - Speech recognition (Waveforms -> the utterance of “cat”) Many language understanding problems require mapping from sequences to sequences: - Machine Translation (“I love music” -> “Je aime la musique”) Quoc V. Le
- 4. How does Machine Translation work? Use a dictionary to translate one word at a time Use a model put reorder the words so that the sentence looks reasonable. Lots of rules: - Phrases instead of words (“New York” should not be translated as “New” + “York”) - Meaning of words depend on contexts Quoc V. Le
- 5. Ideas: Sequence Learning - Use a Recurrent Neural Net encoder to map an input sequence to a vector - Use a Recurrent Neural Net decoder to map the vector to another sequence Quoc V. Le
- 6. Sequence Learning W X Y Z <EOS> Quoc V. Le Example network that maps ABC -> WXYZ A B C <EOS> W X Y Z At test time, feed the output back into the decoder as the input For better output sequence, generate many candidates, feed each candidate to the decoder to have a beam of possible sequences Use “beam search” to find the top sequences
- 7. Sequence Learning W X Y Z <EOS> Quoc V. Le Example network that maps ABC -> WXYZ A B C <EOS> W X Y Z At test time, feed the output back into the decoder as the input For better output sequence, generate many candidates, feed each candidate to the decoder to have a beam of possible sequences Use “beam search” to find the top sequences
- 8. A machine translation experiment WMT’2014 (small in comparison to Google’s data): - State-of-art (a combination of many methods, took 20 years to develop): 37 - Our method (took 3 person year): 37 Important achievement because it’s a new way to represent input texts and output texts. Potential breakthrough in many other areas of language understanding. Quoc V. Le
- 9. Sequence Learning W X Y Z <EOS> A B C <EOS> W X Y Z Quoc V. Le
- 12. Contact: Quoc V. Le (qvl@google.com), Ilya Sutskever (ilyasu@google.com), Oriol Vinyals (vinyals@google.com) Minh-Thang Luong (lmthang@cs.stanford.edu) Paper: Sequence to Sequence Learning with Neural Networks Addressing the Rare Word Problem in Neural Machine Translation Upcoming NIPS paper Quoc V. Le