Sentiment analysis-by-nltk
35 likes13,205 views
This document discusses sentiment analysis using NLTK (Natural Language Toolkit) in Python. It begins with an overview of sentiment analysis and examples of determining sentiment from texts. Then it demonstrates the basics of using a sentiment dictionary to analyze sentences. It discusses challenges with real texts, like handling punctuation and splitting into sentences. NLTK tools for tokenization, sentence splitting, and counting positive and negative words are presented. Finally, it briefly introduces machine learning approaches to sentiment analysis using training data to build a model that can predict sentiment for new texts.
Sentiment analysis-by-nltk
- 1. Sentiment Analysis by NLTK Wei-Ting Kuo PyconApac2015
- 3. Sentiment Analysis? Aim to determine the attitude of a speaker/writer with respect to some text.
- 6. Positive or Negative? This is a good book! This is a good book! I like it! Positive The first chapter is good, but the rest is terrible Negative (more) Positive This is a bad book! Negative
- 7. How to compute it?
- 8. This is a good book! From scouter: This is a postive review
- 9. Why is sentiment analysis useful?
- 10. • This is a terrible book. Because it’s important, so I mention three times, terrible, terrible, terrible!!!
- 11. Let’s begin with the easiest way!
- 12. Sentiment Dictionary like 1 good 2 bad -2 terrible -3
- 13. This is a good book! Dictionary like 1 good 2 bad -2 terrible -3 2 This is a good book! I like it! Positive 3 The first chapter is good, but the rest is terrible -1 Negative (more) Positive This is a bad book! -2 Negative
- 14. AFINN-111 • http://www2.imm.dtu.dk/pubdb/ views/publication_details.php? id=6010 • A list of words rated between -5 (neg) to 5 (pos)
- 15. Let’s build the dictionary in Python
- 16. Let’s split the sentence first
- 17. And compute the score •
- 18. Recap
- 19. What if the text is long? And have many punctuation? • Nice book! Though it is lack of advanced topics. It’s still good for beginners.
- 20. Doesn’t work!
- 22. NLTK to the rescue • Natural Language ToolKit • Works with Python3!
- 23. Tokenization the process of breaking a stream of text up into words, phrases, symbols, or other meaningful elements called tokens
- 24. Word tokenize
- 25. It seems we lose some information • Nice book! Though it is lack of advanced topics. It’s still good for beginners. Positive: 3
- 26. Separate to multiple sentences first • Nice book! • Though it is lack of advanced topics. • It’s still good for beginners. Positive Negative Positive
- 27. How to split the text to sentences?
- 29. Compute score for each sentence
- 30. But we still miss some information in another case It’s a bad idea to buy this book. Negative Negative The first chapter is good, but the rest is terrible and confusing At least the customer mentioned something good, but it’s not recorded
- 31. Let’s count Pos & Neg separately The first chapter is good, but the rest is terrible and confusing It’s a bad idea to buy this book. Neg: -5 Neg: -3 Pos:3
- 32. In Python
- 33. how about new words? how about domain specific term?
- 35. Traing Data (with Labels) Model Trained ModelReal Data Prediction
- 36. Training Data • This is a good book! Postive • This is a awesome book! Postive • This is a bad book! Negative • This is a terrible book! Negative
- 37. • This is a good book! Postive • This is a awesome book! Postive • This is a bad book! Negative • This is a terrible book Negative Model This is a good article This is a bad article Trained Model Postive Negative
- 38. The format NLTK use
- 39. Prepare the Training Set
- 40. Build the model, and train it!
- 41. Real Case • Movie Review Data http://www.cs.cornell.edu/people/pabo/movie-review-data/ • 5331 positive reviews & 5331 negative reviews labelled by human.
- 42. Positive Reviews
- 43. Negative Reviews
- 44. Read our data
- 45. Separate the data Training Data Testing Data Training data to train the model Testing data to compute the accuracy
- 47. Train the data
- 48. Compute the accuracy accuracy = number of correct / total
- 49. How to enhance? • Use the most frequent 1000 words only • Use different model, maybe SVC • Read more paper about the latest research
- 50. Q & A