Ml4nlp04 1
3 likes1,439 views
The document discusses text classification using Naive Bayes classifiers. It presents the Naive Bayes algorithm for classification, including calculating the probabilities of words given a class and class priors. It shows how to estimate these probabilities from training data and classify new documents. An example is provided to demonstrate classification of sample documents into positive and negative classes.
Ml4nlp04 1
- 1. 4.1 4.2 . . 2010/10/12 4.1 4.2
- 2. AGENDA 1 2 . 3 . 4 . . 4.1 4.2
- 3. AGENDA 1 2 . 3 . 4 . . 4.1 4.2
- 4. 4 1 2 . . 4.1 4.2
- 5. AGENDA 1 2 . 3 . 4 . . 4.1 4.2
- 6. (classification categorization) 4.1 4.2
- 7. 2 SVM 4.1 4.2
- 8. AGENDA 1 2 . 3 . 4 . . 4.1 4.2
- 9. d P(c|d) c∈C P(c|d) 1 P(c)P(d|c) P(c|d) = P(d) . 2 P(d) P(c)P(d|c) c max . P(c)P(d|c) c max = arg max c P(d) = arg max P(c)P(d|c) c . 4.1 4.2
- 10. P(d|c) d d d P(d|c) d - - 4.1 4.2
- 11. AGENDA 1 2 . 3 . 4 . . 4.1 4.2
- 12. - P(d|c) d ∏ δ P(d|c) = pw,c (1 − pw,c )1−δw,d w,d w∈V V w pc P(c) pw,c pc 2 δ pw,c (1 − pw,c )1−δw,d w,d c w d 4.1 4.2
- 13. - ∏ δ P(c)P(d|c) = pc pw,c (1 − pw,c )1−δw,d w,d w∈V c pw,c P(d|c) P(d|c) pw,c 1 4.1 4.2
- 14. D D = {(d(1) , c(1) ), (d(2) , c(2) ), ..., (d|D| , c|D| )} ∑ log P( D) = log P(d, c) (d,c)∈D ∑ ∏ δw,d = log pc pw,c (1 − pw,c )1−δw,d (d,c)∈D w∈V ∑ ∑ log pc + = (δw,d log pw,c + (1 − ww,d ) log(1 − pw,c )) (d,c)∈D w∈V ∑ ∑∑ ∑∑ = N c log pc + Nw,c log pw,c + (N c − Nw,c ) log(1 − pw,c ) c c w∈V c w∈V Nc : c Nw,c : c w 4.1 4.2
- 15. pc max . log P(D) ∑ s.t. pc = 1. c L(θ, λ) ∑ L(θ, λ) = log P(D) + λ pc − 1 c θ: { pw,c }winV,c∈C , {pc } c∈C 4.1 4.2
- 16. ∂L(θ, λ) = 0 ∂ pw,c ∂L(θ, λ) = 0 ∂ pc ∂L(θ, λ) = 0 ∂λ 4.1 4.2
- 17. ∂L(θ, λ) ∂ ∑ ∑∑ = N c log pc + Nw,c log pw,c ∂pw,c ∂pw,c c c w∈V ∑∑ ∑ + (N c − Nw,c ) log(1 − pw,c ) + λ pc − 1 c w∈V c ∂(1− pw,c ) Nw,c ∂ pw,c = + (N c − Nw,c ) pw,c (1 − pw,c ) Nw,c (N c − Nw,c ) = − pw,c 1 − pw,c ∂L(θ, λ) ∑ ∂ ∑∑ = N c log pc + Nw,c log pw,c ∂pc ∂pc c c w∈V ∑∑ ∑ + (N c − Nw,c ) log(1 − pw,c ) + λ pc − 1 c w∈V c Nc = +λ pc 4.1 4.2
- 18. pw,c Nw,c (N c − Nw,c ) − = 0 pw,c 1 − pw,c (1 − pw,c )Nw,c − pw,c (N c − Nw,c ) = 0 pw,c (N c − Nw,c + Nw,c ) = Nw,c Nw,c pw,c = Nc 4.1 4.2
- 19. pc Nc +λ = 0 pc Nc pc = − λ ∑ pc = 1 c 1∑ − Nc = 1 λ c ∑ λ = − Nc c Nc Nc pc = − = ∑ λ c Nc 4.1 4.2
- 20. c w pw,c = c c pc = 4.1 4.2
- 21. 4.1 P 3 d(1) = ”good bad good good” d(2) = ”exciting exciting” d(3) = ”good good exciting boring” N 3 d(4) = ”bad boring boring boring” d(5) = ”bad good bad” d(6) = ”bad bad boring exciting” P N 4.1 4.2
- 22. 4.1 V = {bad, boring, exciting, good} N P = 3, N N = 3, N bad,P = 1, N bad,N = 3, N boring,P = 1, N boring,N = 2, Nexciting,P = 2, Nexciting,N = 1, N good,P = 2, N good,N = 1, NP NN pP = N P +N N = 3+3 = 0.50 3 pN = N p+NN = 3+3 = 0.50 3 N bad,P N bad,N pbad,P = N P = 1 = 0.33 3 pbad,N = NN = 3 = 1.00 3 N boring,P N bof ing,N pboring,P = N P = 3 = 0.33 1 pbof ing,N = NN = 2 = 3 0.67 Nexciting,P pexciting,P = N P = 2 = 0.67 3 Nexciting,N 1 pexciting,N = = 3 = 0.33 N good,P N good,N pgood,P = N P = 2 = 0.67 3 pgood,N = NN = 1 = 0.33 3 4.1 4.2
- 23. 4.2 4.1 d d = ”good good bad boring” pP pd|P pN pd|N pP pd|P = pP × pbad,P × pboring,P × (1 − pexciting,P ) × pgood,P = 0.5 × 0.33 × 0.33 × (1 − 0.67) × 0.67 = 0.012 pN pd|N = pN × pbad,N × pboring,N × (1 − pexciting,N ) × pgood,N = 0.5 × 1.00 × 0.67 × (1 − 0.33) × 0.33 = 0.074 4.1 d N 4.1 4.2
- 24. 4.2 4.1 d d = ”good good bad boring” pP pd|P pN pd|N pP pd|P = pP × pbad,P × pboring,P × (1 − pexciting,P ) × pgood,P = 0.5 × 0.33 × 0.33 × (1 − 0.67) × 0.67 = 0.012 pN pd|N = pN × pbad,N × pboring,N × (1 − pexciting,N ) × pgood,N = 0.5 × 1.00 × 0.67times(1 − 0.33) × 0.33 = 0.074 4.1 d N 4.1 4.2
- 25. 4.3 4.1 d(1) d(1) = ”good bad good good fine” d d = ”bad bad boring boring fine” 4.1 4.2
- 26. 4.3 “fine” fine N f ine,P N f ine,N p f ine,P = NP = 1 3 = 0.33 p f ine,N = NN = 0 3 = 0.00 pP pd|P = pP × pbad,P × pboring,P × (1 − pexciting,P ) × p f ine,P × (1 − pgood,P ) = 0.5 × 0.33 × 0.33 × (1 − 0.67) × 0.33 × (1 − 0.67) = 0.002 pN pd|N = pN × pbad,N × pboring,N × (1 − pexciting,N ) × p f ine,N × (1 − pgood,N ) = 0.5 × 1.00 × 0.67 × (1 − 0.33) × 0.00 × 0.67 = 0.00 P 4.1 4.2
- 27. 4.3 d “bad” ”boring” ”good” ”exciting” P p f ine,N = 0.00 N pN pd|N = 0.00 0 MAP 4.1 4.2
- 28. MAP 0.00 MAP ∏ ∏ ∑ × α−1 ( α−1 ) α−1 log P(θ) + log P(D) = log pc pw,c (1 − pw,c ) + log P(d, c) + (const.) c w,c (d,c)∈D ∑ ∑( ) = (α − 1) log pc + (α − 1) log pw,c + log(1 − pw,c ) c w,c ∑ ∏ δw,d + log pc ( pw,c (1 − pw,c )1−δw,d ) + (const.) (d,c)∈ D w∈V ∑ c p(c) = 1 4.1 4.2
- 29. MAP ∑ L(θ, λ) = log P(θ) + log P( D) + λ pc − 1 c ∂L(θ, λ) (α − 1) (α − 1) Nw,c N c − Nw,c = +− + − ∂ pw,c pw,c 1 − pw,c pw,c 1 − pw,c ∂L(θ, λ) (α − 1) N c = + +λ ∂pc pc pc 4.1 4.2
- 30. MAP ∑ 0 c pc = 1 Nw,c + (α − 1) pw,c = Nc + 2 Nc + 1 pc = ∑ c N c + |C| α 4.1 4.2
- 31. 4.4 4.3 MAP α=1 P 3 d(1) = ”good bad good good fine” d(2) = ”exciting exciting” d(3) = ”good good exciting boring” N 3 d(4) = ”bad boring boring boring” d(5) = ”bad good bad” d(6) = ”bad bad boring exciting” 4.1 4.2
- 32. 4.4 Table: MAP MAP pP 0.50 0.50 pN 0.50 0.50 pbad,P 0.33 0.40 pbad,N 1.00 0.80 pboring,P 0.33 0.40 pboring,N 0.67 0.60 pexciting,P 0.67 0.60 pexciting,N 0.33 0.40 p f ine,P 0.33 0.40 p f ine,N 0.00 0.20 pgood,P 0.67 0.60 pgood,N 0.33 0.40 MAP smoothing MAP 4.1 4.2
- 33. AGENDA 1 2 . 3 . 4 . . 4.1 4.2
- 34. V 1 |d| P(d|c) d w nw,d ∑ (∑ n )! ∏ w w,d P(d|c) = P K = nw,d ∏ nw,d qw,c w w∈V nw,d ! w∈V K: ( ∑ ) ∑ P K = w nw,d : w nw,d 4.1 4.2
- 35. c ∑ (∑ n )! ∏ w w,d pc P nw,d ∏ nw,d P(c)P(d|c) = qw,c w w∈V nw,d ! w∈V ∑ (∑ n )! ∏ w w,d arg max P(c)P(d|c) = arg max pc P nw,d ∏ n q w,d c c w w∈V nw,d ! w∈V w,c ∏ nw = arg max pc qw,c c w∈V ∏ nw c pc w∈V qw,c 4.1 4.2
- 36. 2 4.1 4.2
- 37. ∑ log P( D) = log P(d, c) (d,c)∈ D ∑ p(|d|)|d|! ∏ n w,d = log ∏ pc qw,c (d,c)∈ D w∈Vn ! w,d w∈V ∑ P(|d|)|d|! ∑ ∑ ∑ = log ∏ + log pc + nw,d log qw,c (d,c)∈ D w∈V nw,d ! (d,c)∈ D (d,c)∈D w∈V ∑ P(|d|)|d|! ∑ ∑∑ = log ∏ + log nc pc + nw,c log qw,c (d,c)∈ D w∈V nw,d ! c c w∈V max. log P( D) ∑ s.t. pc = 1. c∈C ∑ qw,c = 1; ∀c ∈ C w∈V 4.1 4.2
- 38. ∑ ∑ ∑ L(θ, β, γ) = log P(D) + βc qw,c − 1 + γ pc − 1 c∈C w∈V c∈C ∂L(θ, β, γ) = 0 ∂qw,c ∂L(θ, β, γ) = 0 ∂ pc ∂L(θ, β, γ) = 0 ∂β ∂L(θ, β, γ) = 0 ∂γ 4.1 4.2
- 39. ∂L(θ, β, γ) ∂ ∑ P(|d|)|d|! ∑ ∑∑ = log ∏ + nc log pc + nw,c log qw,c ∂qw,c ∂qw,c (d,c)∈D w∈V nw,d ! c c w∈V ∑ ∑ ∑ βc ( −1) + γ( pc − 1) c∈C w∈V c∈C nw,c = + βc = 0 qw,c nw,c qw,c = βc 4.1 4.2
- 40. βc ∑ qw,c = 1 w∈V 1 ∑ nw,c = 1 β c w∈V 1 βc = ∑ w∈V nw,c nw,c qw,c = ∑ w nw,c pc 4.1 4.2
- 41. c w qw,c = c c w pw,c = c 4.1 4.2
- 42. MAP 0.00 MAP MAP ∏ ∏ ∑ log P(θ) + log P(D) ∝ log pα−1 × qα−1 + c w,c log P(d, c) c w,c (d,c)∈D ∑ ∑ ∑ P(|d|)|d|! ∏ n w,d = (α − 1) log pc + log qw,c + log ∏ pc qw,c n ! c w,c (d,c)∈D w∈V w,d w∈V ∑ ∑ c p(c) = 1 w qw,c = 1 4.1 4.2
- 43. MAP L(θ, β, γ) = log P(θ) + log P(D) ∑ ∑ ∑ + βc pw,c − 1 + γ pc − 1 c∈C w∈V c∈C ∂L(θ, β, γ) (α − 1) nw,c = + + βc ∂qw,c qw,c qw,c ∑ 0 w∈V qw,c = 1 nw,c + (α − 1) qw,c = ∑ w nw,c + |W|(α − 1) 4.1 4.2
- 44. AGENDA 1 2 . 3 . 4 . . 4.1 4.2
- 45. d MAP 4.1 4.2
- 46. ( ) Ml for nlp chapter 4 4.1 4.2
- 47. 4.1 4.2