(Hierarchical) topic modeling
Download as PPTX, PDF2 likes564 views
This document provides an overview of hierarchical topic modeling. It begins with background on text summarization and topic modeling. Topic modeling aims to learn latent topics from a corpus using probabilistic models like PLSI and LDA. Hierarchical topic modeling uses non-parametric Bayesian models like the Chinese Restaurant Process to capture hierarchical structure in topics. The document explains the generative process of nested CRP models and provides examples of hierarchical topics. It also discusses parameter estimation methods and provides supplemental information on probabilistic graphical models and references for further reading.
(Hierarchical) topic modeling
- 1. (Hierarchical) Topic Modeling Yueshen Xu (lecturer) ysxu@xidian.edu.cn / xuyueshen@163.com Data and Knowledge Engineering Research Center Xidian University Text Mining & NLP & ML
- 2. Software Engineering2016/12/27 Outline Background Some Concepts Topic Modeling Probabilistic Latent Semantic Indexing (PLSI) Latent Dirichlet Allocation (LDA) Hierarchical Topic Modeling Chinese Restaurant Process (CRP) Parameter Estimation Supplement & Reference 2 Keywords: topic modeling, hierarchical topic modeling, probabilistic graphical model, Bayesian model Basics, not state-of-the-art
- 3. Software Engineering2016/12/27 Background Information Overloading 3 we need summarization Visualization Dimensional Reduction Big Data Cloud Computing Artificial Intelligence Deep Learning ,…, etc
- 4. Software Engineering2016/12/27 Background Text Summarization Document Summarization What do these docs (or this doc) talk about? Review Summarization What do these consumers care about or complain about? Short Text/Tweets Summarization What are people discussing about? 4 Automatic Applicable Explainable Basic Requirement Topic Modeling
- 5. Software Engineering2016/12/27 General Concepts Latent Semantic Analysis Text Mining Natural Language Processing Computational Linguistics Information Retrieval Dimension Reduction Topic Modeling Some Concepts 5 Information Retrieval Computational Linguistics Natural Language Processing LSA/Topic Model Text Mining LSA Data Mining Reduction Dimension Machine Learning Machine Translation Topic Modeling to learn the latent topics from a corpus/document
- 6. Software Engineering2016/12/27 Topic Modeling Topic modeling an example in Chinese (from my doctorate thesis) 6 继续实施稳健的货币政策，保 持松紧适度适时预调微调，做 好与供给侧结构，并综合运用 数量、价格等多种货币政策 从员额上来看，这次改革远远超 过了裁军的数量，它是一种结构 性的改革，是军队组织结构现代 化的一个关键步骤 美元作为主要国际货币的地位在 可预见的将来仍无可取代，唯一 的出路是推动全球治理向更均衡 的方向发展。国际货币基金组织 总裁拉加德日前在美国马里兰大 学演讲时就呼吁，国际治理改革 应认清新兴经济体越来越重要这 一现实。 独立学院从母体高校“断奶”后， 可能会面临品牌、招生等方面阵 痛，但是在国家和省市鼓励民间 资本进入教育领域的实施意见发 布后，一些独立学院果断切割连 接母体大学的“脐带”，自立门 户发展。 Corpus Doc 1 Doc2 Doc 3 Doc4
- 7. Software Engineering2016/12/27 Topic Modeling After topic modeling 7 继续实施稳健的货币政策，保 持松紧适度适时预调微调，做 好与供给侧结构，并综合运用 数量、价格等多种货币政策 政策 0.082 改革 0.063 … 金融 0.074 货币 0.051 … 学院 0.077 教育 0.071 … 军队 0.083 组织 0.079 … 从员额上来看，这次改革远远 超过了裁军的数量，它是一种 结构性的改革，是军队组织结 构现代化的一个关键步骤 美元作为主要国际货币的地位 在可预见的将来仍无可取代， 唯一的出路是推动全球治理向 更均衡的方向发展。国际货币 基金组织总裁拉加德日前在美 国马里兰大学演讲时就呼吁， 国际治理改革应认清新兴经济 体越来越重要这一现实。 独立学院从母体高校“断奶” 后，可能会面临品牌、招生等 方面阵痛，但是在国家和省市 鼓励民间资本进入教育领域的 实施意见发布后，一些独立学 院果断切割连接母体大学的 “脐带”，自立门户发展。 … … … … Corpus Doc 1 Doc 2 Doc3 Doc 4 Topic 2 Topic 3 Topic 4 Topic 1
- 8. Software Engineering2016/12/27 Topic Modeling A topic A word cluster a group of words Not clustered randomly, but meaningfully (not semantically) 8 Models Parametric models Latent Semantic Indexing (LSI) PLSI; Latent Dirichlet Allocation (LDA) Non-parametric models (Dirichlet Process) (Nested) Chinese Restaurant Process Indian Buffet Process Pitman-Yor Process
- 9. Software Engineering2016/12/27 Topic Modeling 9 pLSI Model w1 w2 wN z1 zK z2 d1 d2 dM ….. ….. ….. )(dp)|( dzp)|( zwp Assumption Pairs(d,w) are assumed to be generated independently Conditioned on z, w is generated independently of d Words in a document are exchangeable Documents are exchangeable Latent topics z are independent The generative process ∑∑ ∈∈ ZzZz dzpzwpdpdzwpdpdpdwpwdp )|()|()(=)|,()(=)()|(=),( Multinomial Distribution Multinomial Distribution One layer of ‘Deep Neutral Network’
- 10. Software Engineering2016/12/27 Topic Modeling 10 Latent Dirichlet Allocation (LDA) David M. Blei, Andrew Y. Ng, Michael I. Jordan Hierarchical Bayesian model; Bayesian pLSI θ z w N M α β iterative times Generative process of LDA Choose N ~ Poisson(𝜉); For each document d={𝑤1, 𝑤2 … 𝑤 𝑛} Choose 𝜃 ~𝐷𝑖𝑟(𝛼); For each of the N words 𝑤 𝑛 in d: a) Choose a topic 𝑧 𝑛~𝑀𝑢𝑙𝑡𝑖𝑛𝑜𝑚𝑖𝑛𝑎𝑙 𝜃 b) Choose a word 𝑤 𝑛 from 𝑝 𝑤 𝑛 𝑧 𝑛, 𝛽 , a multinomial distribution conditioned on 𝑧 𝑛
- 11. Software Engineering2016/12/27 Topic Modeling Parameter Estimation Variational Inference (+EM) || Gibbs Sampling (MCMC) 11 Variational EM Algorithm Aim: (𝛼 ∗ , 𝛽 ∗ )=arg max 𝑑=1 𝑀 𝑝 𝒘|𝛼, 𝛽 Initialize 𝛼, 𝛽 E-Step: compute 𝛼, 𝛽 through variational inference for likelihood approximation M-Step: Maximize the likelihood according to 𝛼, 𝛽 End until convergence I just hope you to know: EM is quite important
- 12. Software Engineering2016/12/27 Hierarchical Topic Modeling Topic modeling is not enough 12 Hierarchical Structure
- 13. Software Engineering2016/12/27 Hierarchical Topic Modeling 13 Chinese Restaurant Process (Dirichlet Process) A restaurant with an infinite number of tables, and customers (word) enter this restaurant sequentially. The ith customer (𝜃𝑖) sits at a table (𝜙 𝑘) according to the probability 𝜙 𝑘: Clustering == 1/2 unsupervised learning clustering, topic modeling (two layer clustering), hierarchical concept building, collaborative filtering, similarity computation…
- 14. Software Engineering2016/12/27 Hierarchical Topic Modeling 14 The generative process (nested CRP) Focus on the insight 1. Let 𝑐1 be the root restaurant (only one table) 2. For each level 𝑙 ∈ {2, … , 𝐿}: Draw a table from restaurant 𝑐𝑙−1 using CRP. Set 𝑐𝑙 to be the restaurant referred to by that table 3. Draw an 𝐿 -dimensional topic proportion vector 𝜃~𝐷𝑖𝑟(𝛼) 4. For each word 𝑤 𝑛: Draw 𝑧 ∈ 1, … , 𝐿 ~ Mult(𝜃) Draw 𝑤 𝑛 from the topic associated with restaurant 𝑐 𝑧 α zm,n N c1 c2 cL T γ wm,n M β k m Matryoshka (Russia) Doll
- 15. Software Engineering2016/12/27 Hierarchical Topic Modeling Examples 15 root topic analysis obtain base system concentration thermal polymer acid property diamine activity compound acid derivative active compound ligand group investigate synergistic reaction derivative yield synthesis microwave assay food quality content analysis decoction component radix quality constituent compound activity synthesize salt derivative antioxidant activity extract inhibitory flavonoid interaction cation metal energy solution
- 16. Software Engineering2016/12/27 Supplement 16 Some supplements Probabilistic Graphical Model Modeling Bayesian Network using plates and circles Generative Model & Discriminative Model: 𝑝(𝜃|𝑋/𝐷𝑎𝑡𝑎) Generative Model: p(θ|X) ∝ p(X|θ)p(θ) - Naïve Bayes, GMM, pLSA, LDA, HMM, HDP… : Unsupervised Learning Discriminative Model: 𝑝(𝜃|𝑋) - LR, KNN,SVM, Boosting, Decision Tree : Supervised Learning Also can be represented by graphical models
- 17. Software Engineering2016/12/27 Reference My previous tutorials/notes (ZJU/UIC/Netease/ITRZJU as a Ph.D) ‘Topic modeling (an introduction)’ ‘Non-parametric Bayesian learning in discrete data’ ‘The research of topic modeling in text mining’ ‘Matrix factorization with user generated content’ …, etc Website You can download all slides of mine http://web.xidian.edu.cn/ysxu/teach.html http://liu.cs.uic.edu/yueshenxu/ http://www.slideshare.net/obamaxys2011 https://www.researchgate.net/profile/Yueshen_Xu 17
- 18. Software Engineering2016/12/27 Reference • David Blei, etc. Latent Dirichlet Allocation, JMLR, 2003 • Yee Whye Teh. Dirichlet Processes: Tutorial and Practical Course, 2007 • Yee Whye Teh, Jordan M I, etc. Hierarchical Dirichlet Processes, American Statistical Association, 2006 • David Blei. Probabilstic topic models. Communications of the ACM, 2012 • David Blei, etc. The Nested Chinese Restaurant Process and Bayesian Inference of Topic Hierarchies. Journal of the ACM, 2010 • Gregor Heinrich. Parameter Estimation for Text Analysis, 2008 • T.S., Ferguson. A Bayesian Analysis of Some Nonparametric Problems. The Annals of Statistics, 1973 • Martin J. Wainwright. Graphical Models, Exponential Families, and Variational Inference • Rick Durrett. Probability: Theory and Examples, 2010 • Christopher Bishop. Pattern Recognition and Machine Learning, 2007 • Vasilis Vryniotis. DatumBox: The Dirichlet Process Mixture Model, 2014 18
Editor's Notes
- #6: 最后表述，由此引出本节：分组交换技术：数据报/虚电路