Clustering introduction
Download as PPTX, PDF3 likes814 views
This document outlines the agenda for a machine learning meetup. It will include sessions on supervised learning algorithms like regression, SVM, trees and Bayesian methods. Unsupervised learning sessions will cover clustering algorithms like K-means, DBSCAN, mean shift and hierarchical clustering. Dimension reduction techniques for visualization will also be discussed. Deep learning sessions will focus on neural networks, convolutional neural networks and training deep models. Cluster validation techniques will be introduced, including internal measures like within-cluster and between-cluster sum of squares and correlation between affinity and incidence matrices. Visualizing similarity matrices is also proposed to validate clustering results.
Clustering introduction
- 1. Clustering for New Discovery in Data Houston Machine Learning Meetup
- 2. 2 SCRŠ Roadmap: Method ⢠Tour of machine learning algorithms (1 session) ⢠Feature engineering (1 session) â Feature selection - Yan ⢠Supervised learning (4 sessions) â Regression models -Yan â SVM and kernel SVM - Yan â Tree-based models - Dario â Bayesian method - Xiaoyang â Ensemble models - Yan ⢠Unsupervised learning (3 sessions) â K-means clustering â DBSCAN - Cheng â Mean shift â Agglomerative clustering - Kunal â Dimension reduction for data visualization - Yan ⢠Deep learning (4 sessions) _ Neural network â From neural network to deep learning â Convolutional neural network â Train deep nets with open-source tools
- 3. 3 SCRŠ Roadmap: Application ⢠Business analytics ⢠Recommendation system ⢠Natural language processing ⢠Computer vision ⢠Energy industry
- 4. 4 SCRŠ Agenda ⢠Introduction ⢠Application of clustering ⢠K-means ⢠DBSCAN ⢠Cluster validation
- 5. 5 SCRŠ What is clustering Clustering: to discover the natural groupings of a set of objects/patterns in the unlabeled data
- 8. 8 SCRŠ Application: Pizza Hut Center Delivery locations
- 9. 9 SCRŠ Application: Discovering Gene functions Important to discover diseases and treatment
- 10. 10 SCRŠ Clustering Algorithm ⢠K-Means (King of clustering, many variants) ⢠DBSCAN (group neighboring points) ⢠Mean shift (locating the maxima of density) ⢠Spectral clustering (cares about connectivity instead of proximity) ⢠Hierarchical clustering (a hierarchical structure, multiple levels) ⢠Expectation Maximization (k-means is a variant of EM) ⢠Latent Dirichlet Allocation (natural language processing) âŚâŚ
- 13. 13 SCRŠ Cluster Validity ⢠For cluster analysis, the question is how to evaluate the âgoodnessâ of the resulting clusters? ⢠Then why do we want to evaluate them? â To avoid finding patterns in noise â To compare clustering algorithms â To determine the optimal number of clusters
- 14. 14 SCRŠ Cluster Validity ⢠Numerical measures: â External: Used to measure the extent to which cluster labels match externally supplied class labels. ⢠Entropy â Internal: Used to measure the goodness of a clustering structure without respect to external information. ⢠Sum of Squared Error (SSE) â Relative: Used to compare two different clusterings. ⢠Often an external or internal measurement is used for this function, e.g., SSE or entropy ⢠Visualization
- 15. 15 SCRŠ Internal Measures: WSE and BSE ⢠Cluster Cohesion: Measures how closely related are objects in a cluster â Example: SSE ⢠Cluster Separation: Measure how distinct or well-separated a cluster is from other clusters ⢠Example: Squared Error â Cohesion is measured by the within cluster sum of squares (SSE) â Separation is measured by the between cluster sum of squares â Where |Ci| is the size of cluster i ďĽ ďĽ ď ďď˝ i Cx i i mxWSS 2 )( ďĽ ďď˝ i ii mmCBSS 2 )(
- 16. 16 SCRŠ Internal Measures: WSE and BSE ⢠Example: SSE â BSS + WSS = constant 1091 9)35.4(2)5.13(2 1)5.45()5.44()5.12()5.11( 22 2222 ď˝ďŤď˝ ď˝ďď´ďŤďď´ď˝ ď˝ďďŤďďŤďďŤďď˝ Total BSS WSS 1 2 3 4 5 ď´ ď´ď´ m1 m2 m K=2 clusters: 10010 0)33(4 10)35()34()32()31( 2 2222 ď˝ďŤď˝ ď˝ďď´ď˝ ď˝ďďŤďďŤďďŤďď˝ Total BSS WSSK=1 cluster:
- 17. 17 SCRŠ Internal Measures: WSE and BSE ⢠Can be used to estimate the number of clusters 2 5 10 15 20 25 30 0 1 2 3 4 5 6 7 8 9 10 KSSE5 10 15 -6 -4 -2 0 2 4 6 WSS
- 18. 18 SCRŠ Internal Measures: Proximity graph measures ⢠Cluster cohesion is the sum of the weight of all links within a cluster. ⢠Cluster separation is the sum of the weights between nodes in the cluster and nodes outside the cluster. cohesion separation
- 19. 19 SCRŠ Correlation between affinity matrix and incidence matrix ⢠Given affinity distance matrix D = {d11,d12, âŚ, dnn } Incidence matrix C= { c11, c12,âŚ, cnn } from clustering ⢠Correlation r between D and C is given by ď ďď ďď ď˝ ďĽďĽ ďĽ ď˝ď˝ď˝ď˝ ď˝ď˝ n ji ij n ji ij n ji ijij ccdd ccdd r 1,1 2 _ 1,1 2 _ 1,1 __ )()( ))((
- 20. 20 SCRŠ Correlation with Incidence matrix ď ďď ďď ď˝ ďĽďĽ ďĽ ď˝ď˝ď˝ď˝ ď˝ď˝ n ji ij n ji ij n ji ijij ccdd ccdd r 1,1 2 _ 1,1 2 _ 1,1 __ )()( ))(( 0 0.2 0.4 0.6 0.8 1 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 x y 0 0.2 0.4 0.6 0.8 1 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 x y r = -0.9235 r = -0.5810
- 21. 21 SCRŠ Visualization of similarity matrix ⢠Order the similarity matrix with respect to cluster labels and inspect visually. 0 0.2 0.4 0.6 0.8 1 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 x y Points Points 20 40 60 80 100 10 20 30 40 50 60 70 80 90 100 Similarity 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
- 22. 22 SCRŠ ⢠Clusters in random data are not so crisp Points Points 20 40 60 80 100 10 20 30 40 50 60 70 80 90 100 Similarity 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0 0.2 0.4 0.6 0.8 1 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 x y Visualization of similarity matrix
- 23. 23 SCRŠ Final Comment on Cluster Validity âThe validation of clustering structures is the most difficult and frustrating part of cluster analysis. Without a strong effort in this direction, cluster analysis will remain a black art accessible only to those true believers who have experience and great courage.â Algorithms for Clustering Data, Jain and Dubes
- 24. 24 SCRŠ Roadmap: Method ⢠Tour of machine learning algorithms (1 session) ⢠Feature engineering (1 session) â Feature selection - Yan ⢠Supervised learning (4 sessions) â Regression models -Yan â SVM and kernel SVM - Yan â Tree-based models - Dario â Bayesian method - Xiaoyang â Ensemble models - Yan ⢠Unsupervised learning (3 sessions) â K-means clustering â DBSCAN - Cheng â Mean shift â Hierarchical clustering - Kunal â Dimension reduction for data visualization - Yan ⢠Deep learning (4 sessions) _ Neural network â From neural network to deep learning - Yan â Convolutional neural network â Train deep nets with open-source tools
- 25. 25 SCRŠ Thank you Slides will be posted on slide share: http://www.slideshare.net/xuyangela