Topic Modeling - NLP
0 likes928 views
The document discusses topic modeling using Latent Dirichlet Allocation (LDA) as a technique to identify underlying themes in texts. It outlines the steps for implementing LDA, including data preparation, creating a document-term matrix, training/testing model, and evaluating topic likelihoods. Additionally, it mentions various related methods and libraries utilized in the process, emphasizing how to determine the optimal number of topics.
![Topic Modeling - LDA
library(RTextTools)
library(topicmodels)
tweets<-read.csv(file.choose())
View(tweets)
names(tweets)
tweets1<-data.frame(tweets$text)
tweets1<-tweets[,c(6,11)]
names(tweets1)
dim(tweets1)
names(tweets1)[2]<-"tweets"
View(tweets1)
Rupak Roy](https://image.slidesharecdn.com/4-220114110938/85/Topic-Modeling-NLP-5-320.jpg)
![Topic Modeling - LDA
#Create a Document Term Matrix
matrix= create_matrix(cbind(as.vector(tweets1$airline),as.vector(tweets1$tweets)),
language="english",removeNumbers=TRUE, removePunctuation=TRUE,
removeSparseTerms=0,
removeStopwords=TRUE, stripWhitespace=TRUE, toLower=TRUE)
inspect(tweets.corpus[1:5])
#Choose the number of topics
k<- 15
#Split the Data into training and testing
#We will take a small subset of data
train <- matrix[1:500,]
test <- matrix[501:750,]
#train <- matrix[1:10248,]
#test <- matrix[10249:1460,]
Rupak Roy](https://image.slidesharecdn.com/4-220114110938/85/Topic-Modeling-NLP-6-320.jpg)
![Topic Modeling - LDA
#Build the model on train data
train.lda <- LDA(train,k)
topics<-get_topics(train.lda,5)
View(topics)
#by default it gives the highest topic with the document
terms<-get_terms(train.lda,5)
View(terms)
#by default it gives the most highly probable word in each topic
#Get the top topics
train.topics <- topics(train.lda)
#Test the model
test.topics <- posterior(train.lda,test)
test.topics$topics[1:10,1:15]
#[row, number of topics(upto 15topics)that is the value of K =15]
test.topics <- apply(test.topics$topics, 1, which.max)
#gives topic with highest probability
Rupak Roy](https://image.slidesharecdn.com/4-220114110938/85/Topic-Modeling-NLP-7-320.jpg)
![Topic Modeling - LDA
#Join the predicted Topic number to the original test Data
test1<-tweets[501:750,]
final<-data.frame(Title=test1$airline,Subject=test1$text,
Pred.topic=test.topics)
View(final)
table(final$Pred.topic)
#View each topic
View(final[final$Pred.topic==10,])
Rupak Roy](https://image.slidesharecdn.com/4-220114110938/85/Topic-Modeling-NLP-8-320.jpg)
![Topic Modeling - LDA
#Get the best value from the graph
k=final_best_model[which.max(final_best_model$log_likelihood),1]
cat("Best topic number k=",k)
Rupak Roy](https://image.slidesharecdn.com/4-220114110938/85/Topic-Modeling-NLP-10-320.jpg)
Topic Modeling - NLP
- 1. Topic Modeling By LDA Laten Dirichlet Allocation
- 2. Topic Modeling Topic modeling: is technique to uncover the underlying topic from the document, in simple words it helps to identify what the document is talking about, the important topics in the article. Types of Topic Models 1) Latent Semantic Indexing (LSI) 2) Laten Dirichlet Allocation (LDA) 3) Probalistic Latent Semantic Indexing (PLSI) Document topic words Rupak Roy
- 3. Topic Modeling - LDA Topics Technology Healthcare Business %topics in the documents 30 % 60% 17% Bag of words Google, Dell Radiology, Transactions, Apple, Microsoft Diagnose Bank, Cost DOCUMENT Behind LDA Topic 1: Technology: Google, Dell, Apple, Microsoft Topic 2: Healthcare: Radiology, Diagnose, Ct Scan Topic 3: Business: Transactions, Banks, Cost. Rupak Roy
- 4. Topic Modeling How often does “Diagnose appear in topic Healthcare ? If the ‘Diagnose’ word often occurs in the Topic Healthcare, then this instance of ‘Diagnose’ might belong to the topic Healthcare. Now how common is the topic healthcare in the rest of the document? This is actually similar to Bayes theorem. To find the probability of possible topic T Multiply the frequency of the word type W in T by the number of other words in document D that already belong to T Therefore the output is The probability that this word came from topic T=> => P(TW,D) = )words W in the topic T/words in the document )* words in D that belong to T Rupak Roy
- 5. Topic Modeling - LDA library(RTextTools) library(topicmodels) tweets<-read.csv(file.choose()) View(tweets) names(tweets) tweets1<-data.frame(tweets$text) tweets1<-tweets[,c(6,11)] names(tweets1) dim(tweets1) names(tweets1)[2]<-"tweets" View(tweets1) Rupak Roy
- 6. Topic Modeling - LDA #Create a Document Term Matrix matrix= create_matrix(cbind(as.vector(tweets1$airline),as.vector(tweets1$tweets)), language="english",removeNumbers=TRUE, removePunctuation=TRUE, removeSparseTerms=0, removeStopwords=TRUE, stripWhitespace=TRUE, toLower=TRUE) inspect(tweets.corpus[1:5]) #Choose the number of topics k<- 15 #Split the Data into training and testing #We will take a small subset of data train <- matrix[1:500,] test <- matrix[501:750,] #train <- matrix[1:10248,] #test <- matrix[10249:1460,] Rupak Roy
- 7. Topic Modeling - LDA #Build the model on train data train.lda <- LDA(train,k) topics<-get_topics(train.lda,5) View(topics) #by default it gives the highest topic with the document terms<-get_terms(train.lda,5) View(terms) #by default it gives the most highly probable word in each topic #Get the top topics train.topics <- topics(train.lda) #Test the model test.topics <- posterior(train.lda,test) test.topics$topics[1:10,1:15] #[row, number of topics(upto 15topics)that is the value of K =15] test.topics <- apply(test.topics$topics, 1, which.max) #gives topic with highest probability Rupak Roy
- 8. Topic Modeling - LDA #Join the predicted Topic number to the original test Data test1<-tweets[501:750,] final<-data.frame(Title=test1$airline,Subject=test1$text, Pred.topic=test.topics) View(final) table(final$Pred.topic) #View each topic View(final[final$Pred.topic==10,]) Rupak Roy
- 9. Topic Modeling - LDA #---------------Another method to get the optimal number of topics ---------# library(topicmodel) best.model <- lapply(seq(2,20, by=1), function(k){LDA(matrix,k)}) #seq(2,20) refers range of K values best_model<- as.data.frame(as.matrix(lapply(best.model, logLik))) #one of the methods to measure the performance is loglikehood & to find out #whether a model is good model or average model or bad model based on the parameter model uses. final_best_model <- data.frame(topics=c(seq(2,20, by=1)), log_likelihood=as.numeric(as.matrix(best_model))) #The higher the loglikelihood the better the model. #finds out ideal topic for every doc head(final_best_model) library(ggplot2) with(final_best_model,qplot(topics,log_likelihood,color="red")) #the higher the likelihood value in the graph better the topics are. Rupak Roy
- 10. Topic Modeling - LDA #Get the best value from the graph k=final_best_model[which.max(final_best_model$log_likelihood),1] cat("Best topic number k=",k) Rupak Roy
- 11. Steps Topic Modeling 1) Data 2) Create TDM 3) Choose number of topics (K) 4) Divide the data into train & test 5) Building model on train data 6) Get the topic 7) Test the model 8) Joining the predicted Topic Number to the original dataset 9) Analyize Rupak Roy