Inteligencia artificial 12
2 likes512 views
This document discusses several natural language processing techniques in Python including: 1. Sentence tokenization using the NLTK tokenizer to split text into sentences. 2. Stemming using the Snowball stemmer to reduce words to their root form. 3. Lemmatization to reduce words to their canonical form. 4. Part-of-speech tagging to label tokens with their part of speech like noun, verb, adjective. 5. Frequency distribution and rare word filtering to analyze word frequencies in a corpus.
![>>>inputstring = ' This is an example sent. The sentence
splitter will split on sent markers. Ohh really !!'
>>>from nltk.tokenize import sent_tokenize
>>>all_sent = sent_tokenize(inputstring)
>>>print all_sent
[' This is an example sent', 'The sentence splitter will split
on markers.','Ohh really !!']](https://image.slidesharecdn.com/inteligenciaartificial-12-170515210902/85/Inteligencia-artificial-12-5-320.jpg)
![>>>from nltk.corpus import stopwords
>>>stoplist = stopwords.words('english') # config the
language name
# NLTK supports 22 languages for removing the stop
words
>>>text = "This is just a test"
>>>cleanwordlist = [word for word in text.split() if word not
in stoplist]
# apart from just and test others are stopwords
['test']
StopWords](https://image.slidesharecdn.com/inteligenciaartificial-12-170515210902/85/Inteligencia-artificial-12-18-320.jpg)
![>>># tokens is a list of all tokens in corpus
>>>freq_dist = nltk.FreqDist(token)
>>>rarewords = freq_dist.keys()[-50:]
>>>after_rare_words = [ word for word in token not in
rarewords]
Rarewords](https://image.slidesharecdn.com/inteligenciaartificial-12-170515210902/85/Inteligencia-artificial-12-19-320.jpg)
![>>>import nltk
>>>from nltk import word_tokenize
>>>s = "I was watching TV"
>>>print nltk.pos_tag(word_tokenize(s))
[('I', 'PRP'), ('was', 'VBD'), ('watching', 'VBG'), ('TV', 'NN')]
What is Part of speech tagging](https://image.slidesharecdn.com/inteligenciaartificial-12-170515210902/85/Inteligencia-artificial-12-21-320.jpg)
![>>>from nltk.corpus import brown
>>>import nltk
>>>tags = [tag for (word, tag) in
brown.tagged_words(categories='news')]
>>>print nltk.FreqDist(tags)
<FreqDist: 'NN': 13162, 'IN': 10616, 'AT': 8893, 'NP': 6866, ',':
5133, 'NNS': 5066, '.': 4452, 'JJ': 4392 >](https://image.slidesharecdn.com/inteligenciaartificial-12-170515210902/85/Inteligencia-artificial-12-23-320.jpg)
![clusters <- hclust(dist(iris[, 3:4]))
plot(clusters)](https://image.slidesharecdn.com/inteligenciaartificial-12-170515210902/85/Inteligencia-artificial-12-50-320.jpg)
![clusters <- hclust(dist(iris[, 3:4]),
method = 'average')
plot(clusters)](https://image.slidesharecdn.com/inteligenciaartificial-12-170515210902/85/Inteligencia-artificial-12-52-320.jpg)
Inteligencia artificial 12
- 2. Python para processamento de Texto
- 5. >>>inputstring = ' This is an example sent. The sentence splitter will split on sent markers. Ohh really !!' >>>from nltk.tokenize import sent_tokenize >>>all_sent = sent_tokenize(inputstring) >>>print all_sent [' This is an example sent', 'The sentence splitter will split on markers.','Ohh really !!']
- 9. Stemming
- 13. Snowball stemmers that can be used for Dutch, English, French, German, Italian, Portuguese, Romanian, Russian, and so on
- 14. For Snowball Stemmer, which is based on Snowball Stemming Algorithm, can be used in NLTK like this: >>> from nltk.stem import SnowballStemmer >>> snowball_stemmer = SnowballStemmer(“english”) >>> snowball_stemmer.stem(‘maximum’) u’maximum’ >>> snowball_stemmer.stem(‘presumably’) u’presum’ >>> snowball_stemmer.stem(‘multiply’) u’multipli’ >>> snowball_stemmer.stem(‘provision’) u’provis’ >>> snowball_stemmer.stem(‘owed’) u’owe’ >>> snowball_stemmer.stem(‘ear’) u’ear’
- 15. Lemmatization
- 18. >>>from nltk.corpus import stopwords >>>stoplist = stopwords.words('english') # config the language name # NLTK supports 22 languages for removing the stop words >>>text = "This is just a test" >>>cleanwordlist = [word for word in text.split() if word not in stoplist] # apart from just and test others are stopwords ['test'] StopWords
- 19. >>># tokens is a list of all tokens in corpus >>>freq_dist = nltk.FreqDist(token) >>>rarewords = freq_dist.keys()[-50:] >>>after_rare_words = [ word for word in token not in rarewords] Rarewords
- 21. >>>import nltk >>>from nltk import word_tokenize >>>s = "I was watching TV" >>>print nltk.pos_tag(word_tokenize(s)) [('I', 'PRP'), ('was', 'VBD'), ('watching', 'VBG'), ('TV', 'NN')] What is Part of speech tagging
- 23. >>>from nltk.corpus import brown >>>import nltk >>>tags = [tag for (word, tag) in brown.tagged_words(categories='news')] >>>print nltk.FreqDist(tags) <FreqDist: 'NN': 13162, 'IN': 10616, 'AT': 8893, 'NP': 6866, ',': 5133, 'NNS': 5066, '.': 4452, 'JJ': 4392 >
- 50. clusters <- hclust(dist(iris[, 3:4])) plot(clusters)
- 52. clusters <- hclust(dist(iris[, 3:4]), method = 'average') plot(clusters)
- 53. K-Means
- 58. ggplot(iris, aes(Petal.Length, Petal.Width, color = iris $Species)) + geom_point(alpha = 0.4, size = 3.5) + geom_point(col = clusterCut) + scale_color_manual(values = c('black', 'red', 'green'))