Intro to Sentiment Analysis
2 likes1,611 views
The document outlines the steps for sentiment analysis, including acquiring data, pre-processing (like handling case sensitivity and stop words), exploring the data, and modeling using techniques like naive Bayes and maximum entropy. It discusses the methods for both supervised and unsupervised learning, providing examples of sentiment scoring. Finally, it touches on challenges faced during sentiment analysis, such as spelling errors, ambiguity, and irony in the text.
Intro to Sentiment Analysis
- 2. Steps • Acquire • Pre-Process • Explore • Model • Test
- 3. Acquire Sources: database, API, scraping... Storage: memory, database, flat file... Format: language specific classes that help with processing (corpus)
- 5. Pre-Process Letter Case "Hello World" -> "hello world" Stop Words ! Stemming
- 6. Pre-Process Letter Case "Hello World" -> "hello world" Stop Words "The iPhone is fantastic" -> "iPhone fantastic" Stemming
- 7. Pre-Process Letter Case "Hello World" -> "hello world" Stop Words "The iPhone is fantastic" -> "iPhone fantastic" Stemming "run, runs, running, ran" -> ("run," 4)
- 8. Explore
- 9. Model • Pointwise Mutual Information (PMI) • Discrete PMI • Naive Bayes • Maximum Entropy
- 10. Unsupervised • Pointwise Mutual Information (PMI) • Discrete PMI • Naive Bayes • Maximum Entropy
- 11. Pointwise Mutual Information Original: I am loving my new iPhone!! Pre-Processed: love new iphone TERM VALUE love +1 new +1 iphone 0 Sentiment Score: +2
- 12. Discrete PMI Original: I am loving my new iPhone!! Pre-Processed: love new iphone TERM VALUE love +4 new +1 iphone 0 Sentiment Score: +5
- 13. Supervised • Pointwise Mutual Information (PMI) • Discrete PMI • Naive Bayes • Maximum Entropy
- 14. Naive Bayes Pre-Processed: love new iphone POSITIVE Who doesn't love the new iPhone? I love tacos, especially from Taco Mayo. New shoes are the best!!! ! NEGATIVE My friend Chuck is an idiot. The new iPhone sucks, DO NOT BUY! I got dumped today, feeling super sad. love ! new ! iphone
- 15. Naive Bayes Pre-Processed: love new iphone POSITIVE Who doesn't love the new iPhone? I love tacos, especially from Taco Mayo. New shoes are the best!!! ! NEGATIVE My friend Chuck is an idiot. The new iPhone sucks, DO NOT BUY! I got dumped today, feeling super sad. love ! new ! iphone 1.0, 0.0 ! ! !
- 16. Naive Bayes Pre-Processed: love new iphone POSITIVE Who doesn't love the new iPhone? I love tacos, especially from Taco Mayo. New shoes are the best!!! ! NEGATIVE My friend Chuck is an idiot. The new iPhone sucks, DO NOT BUY! I got dumped today, feeling super sad. love ! new ! iphone 1.0, 0.0 ! 0.67, 0.33 !
- 17. Naive Bayes Pre-Processed: love new iphone POSITIVE Who doesn't love the new iPhone? I love tacos, especially from Taco Mayo. New shoes are the best!!! ! NEGATIVE My friend Chuck is an idiot. The new iPhone sucks, DO NOT BUY! I got dumped today, feeling super sad. love ! new ! iphone 1.0, 0.0 ! 0.67, 0.33 ! 0.5, 0.5
- 18. Naive Bayes Pre-Processed: love new iphone POSITIVE Who doesn't love the new iPhone? I love tacos, especially from Taco Mayo. New shoes are the best!!! ! NEGATIVE My friend Chuck is an idiot. The new iPhone sucks, DO NOT BUY! I got dumped today, feeling super sad. love ! new ! iphone 1.0, 0.0 ! 0.67, 0.33 ! 0.5, 0.5 0.72, 0.28 Evidence points to a 72% likelihood that this document is positive.
- 19. Maximum Entropy
- 20. Maximum Entropy
- 21. Maximum Entropy Really good at large document sets. Black box of magic VS.
- 22. Test Human Validation: Randomly select n documents to be manually tagged and compare accuracy.
- 23. Test Cross Validation: Hold n% out of already classified samples to see how good our model generalizes to new data.
- 24. Challenges Spelling: the new iPohne is grate! Ambiguous: The iPhone is fantastic at sucking Figurative: this new iPhone is sick. Irony: screen broke, thanks Apple Context: Angry Birds looks awesome on my iPhone