Interpretable Discriminative Dimensionality Reduction and Feature Selection on the Manifold
Download as PPTX, PDF0 likes119 views
The document presents a novel method for interpretable discriminative dimensionality reduction and feature selection within a manifold context, aimed at enhancing class-based interpretations. It outlines objectives such as improving the separation of classes in embedded spaces and performing feature selection using multiple kernel representations. The proposed optimization framework balances interpretability with effective class discrimination, as demonstrated through experiments on various datasets.
Interpretable Discriminative Dimensionality Reduction and Feature Selection on the Manifold
- 1. Interpretable Discriminative Dimensionality Reduction and Feature Selection on the Manifold Babak Hosseini*, Barbara Hammer *Bielefeld University (formerly) Dortmund University (currently) Twitter: @Babak_hss ECML 2019, 19 September 2019
- 2. Babak Hosseini, Barbara Hammer ECML 2019, 19 September 2019 Outline: • Introduction • Proposed Method • Experiments • Conclusion
- 3. Babak Hosseini, Barbara Hammer ECML 2019, 19 September 2019 Dimensionality reduction (DR): • Mapping: • Visualization purpose • Lower down data complexity
- 4. Babak Hosseini, Barbara Hammer ECML 2019, 19 September 2019 Relational representation: • No vectorial representation 𝑿 anymore
- 5. Babak Hosseini, Barbara Hammer ECML 2019, 19 September 2019 DR on manifold: dim. reduction Input space Relational rep. Feature space Projected space
- 6. Babak Hosseini, Barbara Hammer ECML 2019, 19 September 2019 Interpretation of the projection: dim. reduction ? ? Feature space Projected space
- 7. Babak Hosseini, Barbara Hammer ECML 2019, 19 September 2019 Class-based Interpretation: • Applicable to Kernel-based DR method
- 8. Babak Hosseini, Barbara Hammer ECML 2019, 19 September 2019 Class-based Interpretation: • Kernel-PCA • Embedding dimensions • Each 𝒖𝑖 is recont. from a selection of data Q: all of them selected from one class? • If Yes dimension 𝒖𝑖 represents (or is related to) class q
- 9. Babak Hosseini, Barbara Hammer ECML 2019, 19 September 2019 Class-based Interpretation: • a & b: each dim. uses all classes • c & d: each dim. uses all almost one class • Separation of data in the label-space
- 10. Babak Hosseini, Barbara Hammer ECML 2019, 19 September 2019 Class-based Interpretation: Supervised K-based DR methods • e.g.: K-FDA (kernel fisher discriminant analysis) • Within-class (𝑆 𝑊) and between-class (𝑆 𝐵) covariance matrices • Good class-separation • Weak class-based interpretation
- 11. Babak Hosseini, Barbara Hammer ECML 2019, 19 September 2019 Outline: • Introduction • Proposed Method • Experiments • Conclusion
- 12. Babak Hosseini, Barbara Hammer ECML 2019, 19 September 2019 Notations: • Training Matrix: • Label matrix: • Mapping to RKHS (rel. rep.) • Embedding dimensions • Embedding of 𝒙
- 13. Babak Hosseini, Barbara Hammer ECML 2019, 19 September 2019 Objectives: • O1: Increasing the class-based interpretation of embedding dimensions. • O2: The embedding should make the classes more separated in the LD space. • O3: The classes should be locally more condensed in the embedded space. • O4: Performing feature selection if a multiple kernel representation is provided.
- 14. Babak Hosseini, Barbara Hammer ECML 2019, 19 September 2019 Objectives: • O1: Increasing the class-based interpretation of embedding dimensions. • O2: The embedding should make the classes more separated in the LD space. • O3: The classes should be locally more condensed in the embedded space. • O4: Performing feature selection if a multiple kernel representation is provided. Optimization framework:
- 15. Babak Hosseini, Barbara Hammer ECML 2019, 19 September 2019 Objectives: • O1: Increasing the class-based interpretation of embedding dimensions. • O2: The embedding should make the classes more separated in the LD space. • O3: The classes should be locally more condensed in the embedded space. • O4: Performing feature selection if a multiple kernel representation is provided.
- 16. Babak Hosseini, Barbara Hammer ECML 2019, 19 September 2019 Interpretability term (O1): • • Embedding vector: 1. 𝑎 𝑠𝑖, 𝑎 𝑡𝑖 non-zero small 2. 𝑎 𝑠𝑖 = 0 or 𝑎 𝑡𝑖 = 0 large Reconst. close data points in RKHS Smooth labeling in local neighborhoods
- 17. Babak Hosseini, Barbara Hammer ECML 2019, 19 September 2019 Objectives: • O1: Increasing the class-based interpretation of embedding dimensions. • O2: The embedding should make the classes more separated in the LD space. • O3: The classes should be locally more condensed in the embedded space. • O4: Performing feature selection if a multiple kernel representation is provided.
- 18. Babak Hosseini, Barbara Hammer ECML 2019, 19 September 2019 Inter-class dissimilarity (O2): • • • Projected vectors Goal: • To reduce the similarity of 𝒙𝑖 and other classes in the embedded space
- 19. Babak Hosseini, Barbara Hammer ECML 2019, 19 September 2019 Objectives: • O1: Increasing the class-based interpretation of embedding dimensions. • O2: The embedding should make the classes more separated in the LD space. • O3: The classes should be locally more condensed in the embedded space. • O4: Performing feature selection if a multiple kernel representation is provided.
- 20. Babak Hosseini, Barbara Hammer ECML 2019, 19 September 2019 Intra-class similarity (O3): • • • Works on non-zero entries in each 𝒂 𝑠 belonging to class(𝒙𝑖) Goal: • 𝒙𝑖: If 𝛾𝑠𝑖 is large embedding dim 𝒖 𝑠: 𝒂 𝑠 is const. the class(𝒙𝑖)
- 21. Babak Hosseini, Barbara Hammer ECML 2019, 19 September 2019 Objectives: • O1: Increasing the class-based interpretation of embedding dimensions. • O2: The embedding should make the classes more separated in the LD space. • O3: The classes should be locally more condensed in the embedded space. • O4: Performing feature selection if a multiple kernel representation is provided.
- 22. Babak Hosseini, Barbara Hammer ECML 2019, 19 September 2019 Feature-selection (O4): • m projections: • • • Multiple-kernel representation of 𝑿
- 23. Babak Hosseini, Barbara Hammer ECML 2019, 19 September 2019 Feature-selection (O4): • : dim. 𝑚 in 𝒙 e.g.: • multivariate time-series • multi-view image data • multi-domain information • … • Scaling of the RKHS: Goal: • Given the supervised information 𝑯 • 𝛽 𝑚 ≠ 0 dim. 𝑚 is chosen
- 24. Babak Hosseini, Barbara Hammer ECML 2019, 19 September 2019 Feature-selection (O4): Goal: • Given the supervised information 𝑯 • 𝛽 𝑚 ≠ 0 dim. 𝑚 is chosen • Injecting into the opt. framework • affine-constraint + non-negativity cons. interpretable solution 𝜷
- 25. Babak Hosseini, Barbara Hammer ECML 2019, 19 September 2019 Optimization scheme: Convexity of the terms: • PSD • non-convex term (w.r.t. 𝑨): • relaxation of the opt. problem • Alternating opt. scheme
- 26. Babak Hosseini, Barbara Hammer ECML 2019, 19 September 2019 Optimization scheme: • Close-form solution • ADMM algorithm • QP
- 27. Babak Hosseini, Barbara Hammer ECML 2019, 19 September 2019 Outline: • Introduction • Proposed Method • Experiments • Conclusion
- 28. Babak Hosseini, Barbara Hammer ECML 2019, 19 September 2019 Datasets: Different domains: • face, text, image, etc. • UCI & feature-selection rep. • A wide range of dimensions Alternative methods: • Supervised: K-FDA, LDR, SDR, KDR • Unsupervised: JSE, S-KPCA, KEDR
- 29. Babak Hosseini, Barbara Hammer ECML 2019, 19 September 2019 Dimensionality reduction results: • Classification accuracy (%) • 1-nn classifier based on the projected data • 10-fold CV
- 30. Babak Hosseini, Barbara Hammer ECML 2019, 19 September 2019 Dimensionality reduction results:
- 31. Babak Hosseini, Barbara Hammer ECML 2019, 19 September 2019 Interpretation of the embedding dimension: • Interpretability measure 𝑰𝑖: • becomes 𝟏 if 𝒂𝑖 is recon. using one class • close to 𝟎. 𝟓 if 𝒂𝑖 is recon. using all the class
- 32. Babak Hosseini, Barbara Hammer ECML 2019, 19 September 2019 Interpretation of the embedding dimension: • Projecting the emb. Dimensions on the label-space: • 𝑳 = 𝑯𝑨
- 33. Babak Hosseini, Barbara Hammer ECML 2019, 19 September 2019 Feature selection result: • MK representation of the data • non-zero entries in beta: • alternative methods: • MKL algorithms: MKL-TR, MKL-DR, KNMF-MKL, and DMKL • Classification accuracy &
- 34. Babak Hosseini, Barbara Hammer ECML 2019, 19 September 2019 Conclusion: • A novel method for discriminative dimensionality reduction. • Focused on the local neighborhoods in RKHS • Aimed the class-based interpretation of the embedding dimensions. • A good trade-off between interpretation and separation of classes. • Feature-selection extension using multiple-kernel data representation.
- 35. •Thank you very much! •Questions? Twitter: @Babak_hss 3535
Editor's Notes
- #2: 1