Deep Learning and Modern Natural Language Processing (AnacondaCon2019)
Download as PPTX, PDF5 likes224 views
The document discusses the evolution of natural language processing (NLP) toward neural methods, highlighting their advantages over traditional approaches. It covers the different shapes of NLP tasks, such as binary and multi-class classification, and emphasizes the role of contextual representations and sequential models in improving language understanding. Neural networks, particularly recurrent neural networks, are presented as essential tools in addressing the complexities of language modeling and sentiment classification.
Deep Learning and Modern Natural Language Processing (AnacondaCon2019)
- 1. Deep Learning and Modern Natural Language Processing Zachary Brown, Lead Data Scientist, S&P Global
- 2. Outline • Neural Methods for Natural Language Processing • Shapes of Natural Language Processing Tasks • Perceptron Text Classification • Local vs. Global Text Representations • Contextual Representations and Sequence Modeling 2
- 3. Neural Methods for Natural Language Processing
- 4. Neural Methods for Natural Language Processing • Natural Language Processing (NLP) has moved largely to neural methods in recent years 4
- 5. Neural Methods for Natural Language Processing • Natural Language Processing (NLP) has moved largely to neural methods in recent years 5 • Traditional NLP builds on years of research into language representation • Theoretical foundations can lead to model rigidity • Tasks often rely on manually generated and curated dictionaries and thesauruses • Built upon local word representations
- 6. Neural Methods for Natural Language Processing • Natural Language Processing (NLP) has moved largely to neural methods in recent years 6 • Few to no assumptions need to be made • Active area of research; most open-source • Ability to learn global and contextualized word representations • Purpose-built model architectures
- 7. Neural Methods for Natural Language Processing • Natural Language Processing (NLP) has moved largely to neural methods in recent years 7 • Few to no assumptions need to be made • Active area of research; most open-source • Ability to learn global and contextualized word representations • Purpose-built model architectures
- 8. Shapes of Natural Language Processing Tasks
- 9. Shapes of Natural Language Processing Tasks • A general task in natural language processing often takes the form: 9
- 10. Shapes of Natural Language Processing Tasks • For binary classification processes (relevance), our target is a single number, often interpreted as a probability 10
- 11. Shapes of Natural Language Processing Tasks • For multi-class classification processes (type of text), our target is a set of probabilities, one for each of the output classes 11
- 12. Shapes of Natural Language Processing Tasks • For sequential classification (e.g. LM, NER, POS) the target is a probability of each class for each element in the input 12
- 13. Shapes of Natural Language Processing Tasks • In a traditional machine learning pipeline, vectorization (feature engineering) process is often a (very) time consuming process 13 80-90%
- 14. Shapes of Natural Language Processing Tasks • A relatively small proportion of time is spent on the actual modeling 14 10-20%
- 15. Shapes of Natural Language Processing Tasks • Neural networks allow us to develop purpose-built architectures to solve tasks, that learn the appropriate vectorization for a task 15 100%
- 17. Perceptron Text Classification • To introduce the shape of information as it flows through a neural network, we'll first look at a network that only handles classification 17
- 18. Perceptron Text Classification • For the vectorization, we'll assume that we've converted our text into a vector using a count-based method like tf-idf 18 tf-idf
- 19. Perceptron Text Classification • A perceptron is one of the simplest neural network architectures, and is a good fit for this task 19
- 20. Perceptron Text Classification • A perceptron is one of the simplest neural network architectures, and is a good fit for this task 20 input hidden (linear) activation output
- 21. Perceptron Text Classification • The hidden layer represents the weights that will be optimized by the deep learning framework. 21 input hidden (linear) activation output weights
- 22. Perceptron Text Classification • If we want to change our task to multiclass classification, we can simply change the size of our hidden layer (+ minor mods) 22
- 23. Perceptron Text Classification • The result of this is that we now have a matrix of weights to optimize 23 weights
- 24. Local vs. Global Text Representations
- 25. Local vs. Global Text Representations • Let's look back to the problem of creating a vector representation for our text 25 tf-idf
- 26. Local vs. Global Text Representations • Further, let's only consider the task of how we'd represent single words or tokens as vectors 26 dog
- 27. Local vs. Global Text Representations • Traditional approaches to word representations treat each word as a unique entity 27
- 28. Local vs. Global Text Representations • Traditional approaches to word representations treat each word as a unique entity 28
- 29. Local vs. Global Text Representations • Traditional approaches to word representations treat each word as a unique entity 29
- 30. Local vs. Global Text Representations • Traditional approaches to word representations treat each word as a unique entity 30
- 31. Local vs. Global Text Representations • Modern approaches move to a fixed dimensional vector size, with dense vectors 31
- 32. Local vs. Global Text Representations • Modern approaches move to a fixed dimensional vector size, with dense vectors 32
- 33. Local vs. Global Text Representations • Modern approaches move to a fixed dimensional vector size, with dense vectors 33
- 34. Local vs. Global Text Representations • There are a variety of frameworks available that allow for computing these vectors in an unsupervised way 34
- 35. Contextual Representations and Sequence Modeling
- 36. Contextual Representations and Sequence Modeling • Global word representations are a fantastic starting point for many problems in NLP, but consider the following sentence 36 I'm going to book our vacation then relax and read a good book
- 37. Contextual Representations and Sequence Modeling • Global word representations are a fantastic starting point for many problems in NLP, but consider the following sentence 37 I'm going to book our vacation then relax and read a good book
- 38. I don't really hate horror movies, but I hate comedies Contextual Representations and Sequence Modeling • Global word representations are a fantastic starting point for many problems in NLP, but consider the following sentence 38
- 39. I don't really hate horror movies, but I hate comedies Contextual Representations and Sequence Modeling • Global word representations are a fantastic starting point for many problems in NLP, but consider the following sentence 39 Context Matters
- 40. Contextual Representations and Sequence Modeling • For modeling tasks where word ordering and context matter, sequential models are often used. These tasks often take the following shape: 40
- 41. Contextual Representations and Sequence Modeling • Recurrent neural networks are a type of neural network architecture that naturally handles modeling sequential data 41
- 42. Contextual Representations and Sequence Modeling • This type of network generates a new output vector for each input in a sequence, and also feeds that same information forward 42
- 43. Contextual Representations and Sequence Modeling • By feeding the information forward, each subsequent output vector has contextual information encoded from the preceding words 43
- 44. Contextual Representations and Sequence Modeling • This type of architecture can be used to build language models, where the task is to predict the next word in the sequence 44
- 45. Contextual Representations and Sequence Modeling • It can also be used for problems like named entity recognition 45 animalo o o o animal
- 46. Contextual Representations and Sequence Modeling • By taking the final vector in the sequence, you can perform tasks like sentiment classification 46 positive
- 47. Contextual Representations and Sequence Modeling • For all of these different types of tasks, a network similar to the perceptron can be placed at the end to carry out the final classification of each word 47
- 48. Contextual Representations and Sequence Modeling • For all of these different types of tasks, a network similar to the perceptron can be placed at the end to carry out the final classification of each word, or the classification of the whole sequence 48
- 49. Contextual Representations and Sequence Modeling • In a similar manner, these individual elements can be combined in a variety of ways to tackle very complex tasks 49
- 50. Contextual Representations and Sequence Modeling • In a similar manner, these individual elements can be combined in a variety of ways to tackle very complex tasks 50
- 51. Contextual Representations and Sequence Modeling • In a similar manner, these individual elements can be combined in a variety of ways to tackle very complex tasks 51
- 52. Contextual Representations and Sequence Modeling • In a similar manner, these individual elements can be combined in a variety of ways to tackle very complex tasks 52
- 53. Thank you.