A Semi-naive Bayes Classifier with Grouping of Cases
0 likes627 views
The document presents a semi-naive Bayes classifier that incorporates grouping of cases to enhance performance over traditional naive Bayes classifiers. It introduces new joining and grouping criteria, including Bayesian Dirichlet equivalent, expected log-likelihood under leave-one-out, and log-likelihood ratio tests, evaluated through experiments on various datasets. The results indicate that the proposed methods provide efficiency and comparable performance to existing classifiers while reducing complexity.
A Semi-naive Bayes Classifier with Grouping of Cases
- 1. A Semi-naive Bayes Classifier with Grouping of Cases J. Abellán, A. Cano, A. R. Masegosa, S. Moral Department of Computer Science and A.I. University of Granada Spain
- 2. 2 Outline 1. Introduction. 2. Semi-Naive Bayes Classifier with Grouping of Cases. General Description The Joining Criterions The Grouping Criterions 3. Experimental Evaluation. 4. Conclusions and Future Work.
- 3. 3 Introduction Information from a data base Attribute variables Class variable Data Base Calcium Tumor Coma Migraine Cancer normal a1 absent absent absent high a1 present absent present normal a1 absent absent absent normal a1 absent absent absent high ao present present absent ...... ...... ...... ...... ......
- 4. 4 Introduction Naive Bayes (Duda & Hart, 1973) Attribute variables {Xi | i=1,..,r} Class variable C={c1,..,ck}. New observation z=(z1,..,zr) (X1=z1,..,Xr=zr). Select state of C: arg maxci (P(ci|Z)). Supposition of independecy known the class variable: arg maxci (P(ci) ∏r j=1 P(zj|ci)) … C X1 X2 Xr Graphical Structure
- 5. 5 Introduction Naive Bayes Classifiers Naive Bayesian Classifiers: NB’s performance is comparable with some state-of-the-art classifiers even when its independency assumption does not hold in normal cases. Question: “Can the performance be better when the conditional independency assumption of NB is relaxed?”
- 6. 6 Semi-Naive Bayesian Classifiers(SNB) A looser assumption than NB. Independency occurs among the joined variables given the class variable C. Introduction Semi-Naive Bayes Classifiers
- 7. 7 Introduction Semi-Naive Bayes Classifiers Main problems of Semi-NB approach: When to join two variables? Joining Criterion Kononenko’s criterion is entropy based. Pazzani’s criterion is accuracy based. Wrapper estimation. Very high complexity with high number of variables. Class entropy reduction
- 8. 8 A SNB with Grouping of Cases Joining Method Three new proposals for Joining Criterions. BDe: Bayesian Dirichlet Equivalent. L10: The Expected Log-likelihood under leaving-one-out. LRT: Log-likelihood Ratio Test.
- 9. 9 A SNB with Grouping of Cases Grouping Method Increment in Parameter Estimations Solution: “Grouping cases of the new variable”. Independent P (Xi | C)P(Xj | C) Nº Parameters: #(C) (#(Xi) + #(Xj)) Dependent P (Xi, Xj | C) Nº Parameters: #(C) #(Xi) #(Xj) Similar Information
- 10. 10 A SNB with Grouping of Cases Example … C X1 X2 Xr Joining Phase … C X5 x X9 X1 Xr Each pair of Variables is evaluated using a JC Grouping Phase Similar Information Each pair of Cases is evaluated using a GC … C X5 x X9 X1 Xr
- 11. 11 Joining Criterions BDe criterion Bayesian Dirichlet equivalent Metric (BDe) “Bayesian scores measure the quality of a model, M, as the posterior probability of the model given the learning data D” JC(BDe) = Score (M1:D) – Score(M2:D) C X Y C X x Y M1 M2
- 12. 12 Joining Criterions L1O criterion Expected Log-Likelihood Under Leave- One-Out (L1O). Leave-one-out EstimationLaplace Estimation “The estimation of the log-likelihood of the class is carried out with a leave-one-out scheme computed with a closed equation”
- 13. 13 Joining Criterions LRT criterion Log-likelihood Ratio Test (LRT): Corrector Factor: “Comparison of two nested models: M1 with merged variables and M2 variables are independent” Number of total comparisons over n active variables
- 14. 14 Grouping Method Hypotheses Hypotheses: Model Selection Problem Sample data D is restricted to X=xi or X=xj. Consider xi and xj the only possible cases of X. Grouping xi and xj implies X has only one case. Similar Information
- 15. 15 Grouping Method Criterions BDe score: L10 score: LRT score:
- 16. 16 Experimental Evaluation Details SNG was implemented in Elvira. Integrated in Weka for evaluation. Tested in 13 data bases without missing values from UCI repository. 10 fold-cross validation repeated 10 times. Comparison with a corrected paired t-test to 5%.
- 17. 17 The trade-off between Accuracy and log- likelihood is better for LRT. L10 works badly as joining criterion. Evaluating Joining Criterions Naive Bayes Comparison
- 18. 18 Evaluating Joining Criterions Pazzani’s semi-NB comparison LRT works slightly better than BDe. Similar performance with a lower time complexity. LRT is the best joining criterion
- 19. 19 Evaluating Grouping Criterions Naive Bayes Comparison LRT Joining + Grouping Method Not strong differences among criterions. L10 slightly better. L1O is the best grouping criterion
- 20. 20 Pazzani’s Semi-NB Comparison SNB-G = LRT Joining + L10 Grouping Similar performance: Dramatic building time reduction:
- 21. 21 State-of-the-art Classifiers AODE, TAN and LBR comparison Three wins against NB. 1 W vs 1 D against AODE. None difference against TAN and LBR. One Win against Pazzani’s Semi-NB.
- 22. 22 Conclusions and Future Work A preprocessing step for Naive Bayes: Method for joining variables. Combined method for grouping cases. Very efficient with similar performance respect to Pazzani’s Semi-NB classifier. Application to high-dimensionality data sets. Generalization of the methodology to another models: decision trees and TAN model.