Practical Natural language processing
Download as PPTX, PDF1 like222 views
The document provides a comprehensive overview of natural language processing (NLP), including tasks, tools, and use cases such as social media monitoring and job classification. It emphasizes the challenges faced in NLP, such as handling ambiguous terms and the need for efficient data processing. Additionally, it outlines a specific use case involving job market analysis, highlighting the importance of stakeholder involvement and iterative development in achieving project success.
Practical Natural language processing
- 1. Practical Natural language processing (NLP)
- 2. Overview 1. Intro & overview 2. NLP task & tools 3. NLP Use case sharing 4. Impact， Lesson learned
- 3. What is Natural Language Processing? Why we need it?
- 4. Explosive growth of unstructured data
- 5. Better understanding on user interaction & behavior
- 6. Example NLP use case
- 9. NLP in finance & credit scoring
- 10. Other use case
- 13. Buffalo…. this is a correct sentence
- 14. Two interpretation of same sentence I saw a girl with telescope
- 18. Natural language processing (NLP) task stemming stopwords Word segment Part-of-speech Name entity recognition Abbrevation Ambigious term Word similarity Translation Python package: spacy, NLTK
- 19. 1. Stemmer Simplify words to root word • Affect / Affection / Affections /Affected /Affecting Affect
- 20. 2. Remove Stopwords Remove common words and word with little meaning • I, am, it, she ,he, want, do…..
- 21. 3. Handle ambigious term word with potential multiple meanings ● “I love Blackberry” Fruit or mobile phone? ● Java Programming language or Indonesian island?
- 22. 4. Handle abbreviation a shortened form of a word or phrase ● HDB/ MIC / NTUC / PM/ CV
- 23. 5. Name entity extraction (NER) Python package: spacy, NLTK
- 24. 6. word2vec ( word embedding) e.g. semantic similarity, what are the words that have similar meaning of given word/phrase Python package: genism, tensorflow Search: Java developer ( what is the word semantic similar to?)
- 25. 7. Wordcloud
- 26. Other challenge in text processing ● Language library Malay & Tamils language are not supported yet by most service provider ● Spelling mistakes ● Language translation and mapping ● Contextual meaning ● Informal language handling ● Sarcassim
- 27. Use case: Build a job classification engine using NLP (with human-in-loop design)
- 28. 1a) Business objective To understand job market demand & supply, location, skillset needed for each profession & role
- 29. 1b) Overview 2) Key information extraction 1) Classification Classifier 1 – MASCO job category Classifier 2 – MSIC industry Classifier 3 – NEC field of study Classifier 4 - SKILL library
- 30. • To classify a job post into MASCO job category (6000+ categories !!) • To classify a job post into MSIC industry category ( 300+ categories ) • To classify a job post into NEC field of study category (100+ categories) • To extract relevant skill that match to SKILL library ( 2000+ categories)
- 32. 1c) Problem framing and solution approach 1. Business objective: To classify a job post into MASCO job category (classification problem, text & language) 2. Input data: Job title + Job description ( very sparse & text-based data) 3. Output category: 6000+ MASCO category ( even google NLP API only able to categorize 300+) 4. Selected algorithm/ model: word2vec semantic similarity 5. Other alternative model: custom deep learning model like BERT, LSTM (but wait… it take 3~4 months to finetune and pray hard for the accuracy!!!) 6. Other challenge: Limited time ( < 2 months) and resource ( 1 Data scientist , 1 Data engineer)
- 33. Let’s test on Google world leading NLP API
- 35. 2a) Data preparation & standardization
- 36. 3a) Technology architecture overview
- 37. 3b) Data engineering & workflow
- 40. 4c) Train word2vec model
- 41. 4d) tensorboard to visualize word2vec model
- 42. 5a) Classification ( exact-match) MASCO job category NEC field of study
- 43. 5b) Classification ( semantic-match) MASCO job category
- 45. 7) Impact & Benefit 1) Faster discovery of job market insight & trend Improve time data-to-decision from months to day 2) Automate 90% of manual work 50,000+ job post auto-classified per month 300+ man-hours saved per month ( 2 two head-count)
- 46. 8a) Business challenge & lesson learned ● Get buy-in early - Identify all your stakeholders and involve them since project initiation ( Avoid i build first, they will come later) ● Goal & impact oriented - Understand what matter to the organization & identify high impact and low-hanging fruit use case ● Be Agile – Start with simple model, build first prototype, get feedback and run iteration
- 47. 8b) Technical challenge & lesson learned ● Be realistic – Data is not always came in “perfect” structures as per your wish list! ● Technology/Technical gap - in your organization i.e. legacy systems & integration ● Performance - Architecture & pipelines to solve performance i.e. increment of concurrent users ● End-to-end solution mindset - You need knowledge in software infrastructures, development pipeline & deployment!
- 48. Questions?
- 51. NLP & NLU