Deep learning for NLP
- 1. Deep Learning for Natural Language Processing Sihem Romdhani
- 2. Agenda What is deep learning? Why is this important? How to apply deep learning to natural language processing?
- 3. About Me Sihem Romdhani Software engineer at Veeva MEng in Computer Engineering MSc in ECE at University of Waterloo - Deep learning - Speech/image recognition
- 4. What is machine learning? Training Text DOCUMENTS, IMAGES, SOUNDS, ... Labels Features Vectors Machine Learning Algorithm
- 5. What is machine learning? Training Text DOCUMENTS, IMAGES, SOUNDS, ... Labels Features Vectors Features Vector New Text DOCUMENTS, IMAGES, SOUNDS, ... Predictive Model Machine Learning Algorithm Expected Label
- 6. What is deep learning? image (pixel) edges object parts object models Input Layer Output Layer SARA layers of abstraction DEEP NEURAL NETWORK (DNN)
- 7. Speech technology DeepMind beats the best in GO Nvidia to drive car with Drive PX 2 super computer Apple’s Siri Amazon’s Alexa
- 8. Quiz Time!
- 9. Which is human? Which is machine? I’m in a love affair I can’t share it ain’t fair Haha I’m just playin’ ladies you know I love you. I know my love is true and I know you love me too Girl I’m down for whatever cause my love is true SONG A For a chance at romance I would love to enhance But everything I love has turned to a tedious task One day we gonna have to leave our love in the past I love my fans but no one ever puts a grasp SONG B
- 10. DeepBeat.org Online tool for rap lyrics generator
- 11. Natural Language Processing (NLP) understanding and generating human language
- 12. Human language is difficult! variability kitty vs. cat ambiguity blue jays vs. Blue Jays vague and latent “Eating babies can be messy.”
- 13. How can we represent words? AUDIO SPECTROGRAM IMAGE PIXELS VECTOR OF REAL NUMBERS Audio Image Word
- 14. ‘CAT’ 0 0.5 0 0.2 0.1 0 0.1 0 0.3 0 0.4 0.1 ‘KITTY’ Similarity? Embedding Embedding
- 15. The cat/kitty purrs Context defines similarity Infer the meaning of words from the company they keep The cat/kitty hunts mice
- 16. Word to Vector The little cat hunts mice Embedding (‘cat’) Context Window Predict
- 17. Word2Vec (Google) for learning embeddings 1. Collect large corpus of text 2. Randomly initialize embeddings Each word is mapped into a fixed-size embedding with random values 3. Set up prediction task For each word, pick a random word from the context and consider it as the target (label) INPUT WORD → TARGET WORD ‘Cat’ → ‘Hunt’ 4. Learn the embeddings Train a model (logistic classifier) to predict the target (neighbours of each word) Update the embedding
- 19. ‘cat’ Embedding Predicted target PREDICTION Linear model wVcat + b Word2Vec Process
- 20. 0 0 0 0 0 1 0 0 0 0 ‘cat’ Embedding Predicted target Real target ‘hunt’ PREDICTION COMPARE Linear model wVcat + b Volcabulary Word2Vec Process
- 21. Similar words are nearby Word Embedding ‘cat’ ‘tiger’ ‘home’ ‘kitten’ ‘dog’
- 22. Let’s discuss a real problem.
- 23. Pharma Company’s Market Expansion New product: pancreatic cancer treatment! Who are the oncologists to inform about it?
- 24. Web scraping DIFFERENT DATA-EXTRACTION SCRIPTS FOR DIFFERENT WEB PAGES
- 25. Consolidating into one record
- 26. Extracting addresses from webpages abbreviations Avenue - Ave - Ave. ambiguity in names main vs. Main Street missing elements mispellings misorder
- 27. Demo
- 28. Solution architecture WEB PAGE (HTML) TOKENIZATION WORD REPRESENTATION/ WORD EMBEDDING DEEP NEURAL NETWORK CLASSIFIER POST PROCESSING OUTPUT EXTRACTED ADDRESS
- 29. Solution architecture WEB PAGE (HTML) TOKENIZATION WORD REPRESENTATION/ WORD EMBEDDING DEEP NEURAL NETWORK CLASSIFIER POST PROCESSING OUTPUT EXTRACTED ADDRESS
- 30. Solution architecture WEB PAGE (HTML) TOKENIZATION WORD REPRESENTATION/ WORD EMBEDDING DEEP NEURAL NETWORK CLASSIFIER POST PROCESSING OUTPUT EXTRACTED ADDRESS
- 31. Solution architecture WEB PAGE (HTML) TOKENIZATION WORD REPRESENTATION/ WORD EMBEDDING DEEP NEURAL NETWORK CLASSIFIER POST PROCESSING OUTPUT EXTRACTED ADDRESS
- 32. Solution architecture WEB PAGE (HTML) TOKENIZATION WORD REPRESENTATION/ WORD EMBEDDING DEEP NEURAL NETWORK CLASSIFIER POST PROCESSING OUTPUT EXTRACTED ADDRESS
- 33. TensorFlow for word embedding
- 34. TensorFlow for word embedding
- 35. TensorFlow for word embedding
- 36. TensorFlow for word embedding
- 40. Training a DNN classifier 123 Main Street, Florida 32209 123 Street Florida 32209Main Context Words AO A: Address O: Other (non-address) Hidden Layer 3 Hidden Layer 2 Hidden Layer 1
- 41. Address extraction algorithm 1. Load the trained DNN model 2. for each new web page do 3. The web page is pre-processed to remove the HTML tags, then tokenized 4. Each token (word) is replaced by its embedding vector 5. Create the DNN input by concatenating the embedding of the context 6. The DNN labels the central word of the context as A (ADDRESS) or O (OTHER) 7.Extract the sequence of tokens with A label 8.if the total number of tokens is within range then 9. Output the token block as extracted address 10.End if 11.End For Input: Trained DNN model and unlabeled web page Output: Extracted address
- 42. Applications of NLP Information extraction 文 A Social media analytics Question & Answer Natural language generation Speech recognition
- 43. Useful Tools
- 44. Questions?
- 45. Slides available on Slack channel #veeva Stop by our booth for questions and cool swag!