Lecture 5: Variational Estimation and Inference
1 like962 views
The document discusses variational estimation and inference techniques, focusing on exponential family models, particularly in relation to estimating models with partial observations and using methods like Expectation-Maximization (EM) and variational EM. It covers concepts such as conditional log-partitions, maximizing likelihood through latent variables, and introduces mean field methods to handle intractable expectations. The document outlines various strategies for managing observed and latent variables to optimize estimations effectively.
Lecture 5: Variational Estimation and Inference
- 2. Outline • Es$ma$on)of)Models)in)Exponen$al)Families • Es$ma$on)with)par$al)observa$ons)9)EM • Mean)Field)Methods 2
- 3. Factorized+Exponen0al+Family Consider)an)exponen-al)family)of)joint)distribu-ons) over) : Here,% %indicates%the%subset%of%components%involved% in%the% 6th%factor. 3
- 4. With%Complete%Observa2ons Given& ,&the&op,mal&es,mates&are&given&by • With&canonical¶meteriza1on,&this&is&convex. • Parameters&may&come&with&constraints. • There&can&be&analy%c'solu%ons,&otherwise,&one&can& solve&this&using&numerical&methods.& 4
- 5. Example:)GMM • GMM"involves:"observed*feature" "and"component* indicator" . • How%to%es)mate%if%both% %and% %are%observed? 5
- 8. Par$ally'Observed'Models Consider)an)exponen-al)family)involving)observed( variables) )and)latent(variables) : Here,% %and% %refer%to%the%observed%parts%and%latent% parts%of%the%en2re%sample%set. 8
- 11. Issues • The%condi&onal)log+par&&on% %as%below%is% o-en%very%difficult%to%evaluate: • We$usually$resort$to$Expecta(on+Maximiza(on0(EM)$ --$a$strategy$that$itera1vely$construct$and$maximize$ lower$bounds$of$ . 11
- 13. Lower&Bound&of& &(cont'd) Hence,&we&have& Hence,& &is&a&lower&bound&of& &for& any& . 13
- 14. Expecta(on+Maximiza(on The$Expecta(on+Maximiza(on0(EM)$algorithm$is$ coordinate0ascent$on$ : • E"step: • M"step: 14
- 15. E"step • Each&E"step&reduces&to&maximize& ,& the&op4mal&solu4on&is&the&expecta*on&of& : • By$conjugate*duality,$with$ ,$we$have$ ,$thus: 15
- 18. log L(✓|x) Q(✓; µ(t+1) ) Q(✓; µ(t) ) ✓(t 1) ✓(t) ✓(t+1) EM#Op&mizes# Sta$onary)point)is)a-ained)when) )and) )are)dually&coupled,)w.r.t.) both) )and) : 18
- 19. Info.&Geo.&Interpreta-on • A#parameter# #indicates#a#condi0onal# distribu0on#over# :# . • A#mean# #is#realized#by#another#condi0onal# distribu0on# #with# . • The#KL#divergence#between#them: 19
- 20. Info.&Geo.&Interpreta-on • For%any% %and% : • E#step:)minimize) )to)close)the)gap) between) )and) . • M#step:)M#projec;on)of) )onto) . 20
- 27. Varia%onal)EM • Basic&idea:"Use"a"distribu-on" "from"a"tractable" family" "to"approximate" ,"and"thus" "to"approximate" . • This"is"to"restrict" "to" . • The"lower"bound"becomes: 27
- 28. Varia%onal)EM)(cont'd) • Varia%onal)E+step:"with"restric+on"to" ," compu+ng" "is"tractable: ! • M"step:"remains"the"same 28
- 29. Varia%onal)E+step • "is"usually"chosen"to"be"an"exponen&al)family," parameterized"by" ."Then"the"varia&onal)E1step" reduces"into"two"steps. • Step"1:"Find"op=mal" "through"I1projec&on: • Step&2:&Compute& 29
- 30. Key$Problem • With& &given,& &remains&an&exponen3al& family&distribu3on:& &with& &and& . • &plays&a&key&role&in&model&es3ma3on. • Key$problem:&choose&a&tractable&distribu3on& & from& &to&approximate& &and&compute& 30
- 31. Mean%Field%Methods • Consider*an*exponen.al*family*distribu.on* *for* which*it*is*intractable*to*compute*the*mean*given* . • Mean%field%methods*use*a*distribu.on* *from*a* tractable*family,*usually*in*a*product%form,*to* approximate*the*given*distribu.on* ,*and*use* *to*approximate* .* 31
- 32. Product(Form • We$say$a$joint$distribu1on$over$ $ is$of$the$product(form,$if$its$density$can$be$wri8en: • An$exponen&al)family$of$product)form: 32
- 33. Product(Form((cont'd) • Log%par))on+func)on: • Expecta)on: • If$each$factor$is$tractable,$then$the$whole$ distribu5on$is$tractable. 33
- 36. Ising&Model&(approxima2on) To#find# #that#approximates# ,#we#perform#I" projec)on#of# #onto#the#factorized1family# : with% . 36
- 39. Mean%Field%Theory%(cont'd) The$difference$between$ $and$the$tractable(lower( bound$is$the$KL$divergence: with% .%The%op+ma% %is%the%I"projec)on: 39
- 41. Hence,&the&nega+ve&entropy&of& &can&be&factorized: The$op'ma$ $can$be$solved$by$minimizing: where% . 41
- 42. Naive&Mean&Field&(Op/ma) • This&problem&can&be&solved&by&coordinate*descent.& • When&op6ma&is&a7ained:& • Hence,'the'op,ma' 'is'given'by' 42
- 43. Naive&Mean&Field&(Discussion) • In$naive$mean$field,$while$ $is$of$a$product$form,$ the$parameters$associated$with$different$ components$are$generally$coupled$in$the$op;mal$ approxima;on. • The$I"projec)on$problem$in$naive$mean$field$is$non# convex$in$general.$In$prac;ce,$the$coordinate$ ascent$procedure$can$be$trapped$in$a$local-valley.$ • Generally,$it$is$unclear$how$far$ $is$from$ . 43
- 44. Varia%onal)EM)(Recap) • E#step((for(each(sample( ): • M#step: 44
- 45. M N nd ↵ ✓d zdi wdi k Latent&Dirichlet& Alloca/on • Variables • Parameters:. ,. • Observed:. • Latent:. ,. 45
- 46. Condi&onal)Distribu&on Let$ $and$ : Two$latent&suff.stats.:$ $and$ .$ 46
- 48. Varia%onal)E+Steps • For% : • For% : 48
- 49. M"Step 49