Text analysis using python
1 like1,643 views
This document provides an overview of text analysis with Python. It discusses common text mining tasks like entity recognition, sentiment analysis, and categorization. It promotes Python for short, concise text processing using tools like NLTK, Scipy, NumPy, and Scikit-learn. Key pre-processing steps covered include lowercasing, tokenization, handling entities, and removing stopwords. Supervised classification techniques like Naive Bayes classifiers and maximum entropy models are explained. Examples presented include a gender classifier and a system for recognizing questions in tweets.
![Pre-processing
Lower casing, stripping extra characters
”Realyyyyyyyyyy!!!!!”
Tokenisation (sentences, words)
>>>pktst = nltk.data.load('tokenizers/punkt/english.pickle')
>>>sentences = pktst.tokenize(tweet)
>>>words = nltk.word_tokenize(sent)
Handling Entities
>>> re.sub(r'(^| )@[^ ]+','',tweet).strip()
Removing stopwords
>>> stopwords = set([”a”, ”an”, ”the”, ”by”])
>>> ' '.join([w for w in words if w not in stopwords])](https://image.slidesharecdn.com/text-analysis-with-py-150701102157-lva1-app6891/85/Text-analysis-using-python-6-320.jpg)
![Training
”Teach” the classifier how to classify, using
training data
>>> from nltk import NaiveBayesClassifier as nbc
>>> trg = generate_features(training_samples)
>>> random.shuffle(trg)
>>> train, test = trg[:int(0.9*len(trg)], trg[int(0.9*len(trg):]
>>> clf = nbc.train(train)
>>> nltk.classify.accuracy(clf, test)](https://image.slidesharecdn.com/text-analysis-with-py-150701102157-lva1-app6891/85/Text-analysis-using-python-11-320.jpg)
![A Gender Classifier
def gender_features(word):
return {'last_letter': word[-1]}
names = ([(name, 'male') for name in names.words('male.txt')] +
[(name, 'female') for name in names.words('female.txt')])
featuresets = [(gender_features(n), g) for (n,g) in names]
train_set, test_set = featuresets[500:], featuresets[:500]
classifier = nltk.NaiveBayesClassifier.train(train_set)
nltk.classify.accuracy(classifier, test_set)](https://image.slidesharecdn.com/text-analysis-with-py-150701102157-lva1-app6891/85/Text-analysis-using-python-13-320.jpg)
Text analysis using python
- 1. Text Analysis with Python Vijay Ramachandran
- 2. Fools Rush In? ”The fact is, that to do anything in the world worth doing, we must not stand back shivering and thinking of the cold and danger, but jump in and scramble through as well as we can.” - Robert Cushing
- 3. Motivation Machine Learning everywhere Users expectations of ”standard experience” Many Resources!
- 4. Text Mining Extract high quality information from text Typically, trends and patterns are analysed using statistical methods – Machine Learning Common Tasks – entity recognition, sentiment analysis, categorization, clustering
- 5. Why Python? Short, concise text processing NLTK Scipy, numpy, scikit.learn Integration with other languages! Because when you start your company, YOU get to decide!
- 6. Pre-processing Lower casing, stripping extra characters ”Realyyyyyyyyyy!!!!!” Tokenisation (sentences, words) >>>pktst = nltk.data.load('tokenizers/punkt/english.pickle') >>>sentences = pktst.tokenize(tweet) >>>words = nltk.word_tokenize(sent) Handling Entities >>> re.sub(r'(^| )@[^ ]+','',tweet).strip() Removing stopwords >>> stopwords = set([”a”, ”an”, ”the”, ”by”]) >>> ' '.join([w for w in words if w not in stopwords])
- 7. Leveraging a Corpus Simple techniques to analyze a domain Term Frequency to find important entities ”low light photography”, ”travel photography” tf/idf to find representative terms across domains ”gaming” for TVs GND to find aliases e.g., ”e700” and ”Samsung e700” vs ”e310” and ”Samsung e310” Yahoo BOSS is great!
- 9. Bayes Theorem Conditional Probability Bayesian classifiers - given features, find Probability of Class PC∣Fi ,...,Fn=PC∏i PFi∣C
- 10. Features Characteristics of to-be-classified object For text, typically unigrams, ”n”-grams, POS, CHUNK, presence in gazeteer Can be numerical or boolean Crucial to performance of classifier! In NLTK, a dict of feature name to value >>> {”word1” : 2, ”word2” : 1, ”word1-word2” : 1, ”word2-word3” : 3, ”word1-in-monuments” : False}
- 11. Training ”Teach” the classifier how to classify, using training data >>> from nltk import NaiveBayesClassifier as nbc >>> trg = generate_features(training_samples) >>> random.shuffle(trg) >>> train, test = trg[:int(0.9*len(trg)], trg[int(0.9*len(trg):] >>> clf = nbc.train(train) >>> nltk.classify.accuracy(clf, test)
- 12. Training, part 2 Measure, tune, iterate Cross Validation Aim is to find a balance between Precision and Recall
- 13. A Gender Classifier def gender_features(word): return {'last_letter': word[-1]} names = ([(name, 'male') for name in names.words('male.txt')] + [(name, 'female') for name in names.words('female.txt')]) featuresets = [(gender_features(n), g) for (n,g) in names] train_set, test_set = featuresets[500:], featuresets[:500] classifier = nltk.NaiveBayesClassifier.train(train_set) nltk.classify.accuracy(classifier, test_set)
- 14. Advice on Training Its TEDIOUS! Cut the cognitive load e.g.: ”I love this camera!” BAD: 0 I PRP B-NP - 1 love VBP B-VP - 2 this DT B-NP - 3 camera NN I-NP 1 Good: (I, love) – NO, (love, this camera) – YES Dealing with ambiguity Mechanical Turk, jsonwidget
- 15. Other Classifiers Maximum Entropy Support Vector Machines Conditional Random Fields All follow the same workflow!
- 16. More Examples Find questions in Tweets unigrams, bigrams, trigrams, parse distance Recognize ”contextual questions” in discussions ”reco” words, ”thanks” words ”Use-type” recognizer POS, CHUNK, special words, verbs within 3 words of phrase e.g., ”I want to compose the perfect landscape shot”
- 17. Eats, shoots and leaves Common basic step! POS, chunk, parse tree Hairy theory FSA, Morphology, Phonology n-Grams, Probabilistic models, CFGs Don't Worry, Be Happy!
- 18. No License Required! What to use? NLTK, or ? Docking with the Evil MotherShip Jepp, JPype? ► use RPC >>> from .stanford_corenlp import jsonrpc >>> server = jsonrpc.TransportTcpIp(...) >>> result = loads(server.parse(paragraph)) POS, Parse tree, and more (but no chunks?)
- 19. Wordnet® ”a large lexical database of English” synsets, hypernyms, hyponyms, gloss synonyms, antonyms Sentiwordnet Start with candidate ”good” and ”bad” words Expand by recursively following edges Classify using definition e.g., ”good” → ”better”, ”good” → ”impressive”
- 20. Love or Hate? ”I love the screen, but the battery life is poor” Shallow? 3 class classifier Or Deep? Relationship classifier Extracting candidate subjects Lots of unsolved problems – co-reference, multiple subjects, negation, etc. Summary ratings for ”Executive Summaries”
- 21. Gender, revisited solr - search semi-structured text Out of the box text processing utilities – stemming, tokenising Highly configurable relevancy fields, weights Sorting! ”sort(term, field, edit) desc” for Levenshtein edit distance
- 22. Gender, revisited Schema <field name="name" type="string" /> <field name="name_phoneme" type="phonetic" /> Search: add ”sort=strdist(unknown_name, name, edit) desc) Python: for namerec in results: If namerec.gender == 'Male': male_score += namerec.match_score else: female_score += namerec.match_score Correctly guesses ”Sheena” and ”Ashish”!! 80/80 precision/recall
- 23. Miles to go before I sleep! Machine Learning on coursera
- 24. Thank You!! Vijay Ramachandran: vijay750@gmail.com