K-Means Algorithm
3 likes3,238 views
The document discusses the k-means clustering algorithm, highlighting its purpose of minimizing cost by finding cluster centers for a given set of points. It explains the limitations of k-means, such as sensitivity to initialization and inability to handle clusters of varying densities and shapes. Additionally, it introduces k-means++, an improved version with better initialization that enhances performance and guarantees improved results.
![1
K-Means
Class Algorithmic Methods of Data Mining
Program M. Sc. Data Science
University Sapienza University of Rome
Semester Fall 2015
Lecturer Carlos Castillo http://chato.cl/
Sources:
● Mohammed J. Zaki, Wagner Meira, Jr., Data Mining and Analysis:
Fundamental Concepts and Algorithms, Cambridge University
Press, May 2014. Example 13.1. [download]
● Evimaria Terzi: Data Mining course at Boston University
http://www.cs.bu.edu/~evimaria/cs565-13.html](https://image.slidesharecdn.com/s3o456brqtm41i9vhb1q-signature-d503319da1301971aeeaa1b7e38d890bf394f7e6fb2415f661de7d4d58f0c518-poli-151222093950/85/K-Means-Algorithm-1-320.jpg)
![31
k-means--
● Algorithm 4.1 in [Chawla & Gionis SDM 2013]](https://image.slidesharecdn.com/s3o456brqtm41i9vhb1q-signature-d503319da1301971aeeaa1b7e38d890bf394f7e6fb2415f661de7d4d58f0c518-poli-151222093950/85/K-Means-Algorithm-31-320.jpg)
K-Means Algorithm
- 1. 1 K-Means Class Algorithmic Methods of Data Mining Program M. Sc. Data Science University Sapienza University of Rome Semester Fall 2015 Lecturer Carlos Castillo http://chato.cl/ Sources: ● Mohammed J. Zaki, Wagner Meira, Jr., Data Mining and Analysis: Fundamental Concepts and Algorithms, Cambridge University Press, May 2014. Example 13.1. [download] ● Evimaria Terzi: Data Mining course at Boston University http://www.cs.bu.edu/~evimaria/cs565-13.html
- 2. 2 Boston University Slideshow Title Goes Here The k-means problem • consider set X={x1,...,xn} of n points in Rd • assume that the number k is given • problem: • find k points c1,...,ck (named centers or means) so that the cost is minimized
- 3. 3 Boston University Slideshow Title Goes Here The k-means problem • k=1 and k=n are easy special cases (why?) • an NP-hard problem if the dimension of the data is at least 2 (d≥2) • in practice, a simple iterative algorithm works quite well
- 4. 4 Boston University Slideshow Title Goes Here The k-means algorithm • voted among the top-10 algorithms in data mining • one way of solving the k- means problem
- 6. 6 Boston University Slideshow Title Goes Here The k-means algorithm 1.randomly (or with another method) pick k cluster centers {c1,...,ck} 2.for each j, set the cluster Xj to be the set of points in X that are the closest to center cj 3.for each j let cj be the center of cluster Xj (mean of the vectors in Xj) 1.repeat (go to step 2) until convergence
- 7. 7 Boston University Slideshow Title Goes Here Sample execution
- 8. 8 1-dimensional clustering exercise Exercise: ● For the data in the figure ● Run k-means with k=2 and initial centroids u1=2, u2=4 (Verify: last centroids are 18 units apart) ● Try with k=3 and initialization 2,3,30 http://www.dataminingbook.info/pmwiki.php/Main/BookDownload Exercise 13.1
- 9. 9 Limitations of k-means ● Clusters of different size ● Clusters of different density ● Clusters of non-globular shape ● Sensitive to initialization
- 10. 10 Boston University Slideshow Title Goes Here Limitations of k-means: different sizes
- 11. 11 Boston University Slideshow Title Goes Here Limitations of k-means: different density
- 12. 12 Boston University Slideshow Title Goes Here Limitations of k-means: non-spherical shapes
- 13. 13 Boston University Slideshow Title Goes Here Effects of bad initialization
- 14. 14 Boston University Slideshow Title Goes Here k-means algorithm • finds a local optimum • often converges quickly but not always • the choice of initial points can have large influence in the result • tends to find spherical clusters • outliers can cause a problem • different densities may cause a problem
- 16. 16 Boston University Slideshow Title Goes Here Initialization • random initialization • random, but repeat many times and take the best solution • helps, but solution can still be bad • pick points that are distant to each other • k-means++ • provable guarantees
- 17. 17 Boston University Slideshow Title Goes Here k-means++ David Arthur and Sergei Vassilvitskii k-means++: The advantages of careful seeding SODA 2007
- 18. 18 Boston University Slideshow Title Goes Here k-means algorithm: random initialization
- 19. 19 Boston University Slideshow Title Goes Here k-means algorithm: random initialization
- 20. 20 Boston University Slideshow Title Goes Here 1 2 3 4 k-means algorithm: initialization with further-first traversal
- 21. 21 Boston University Slideshow Title Goes Here k-means algorithm: initialization with further-first traversal
- 22. 22 Boston University Slideshow Title Goes Here 1 2 3 but... sensitive to outliers
- 23. 23 Boston University Slideshow Title Goes Here but... sensitive to outliers
- 24. 24 Boston University Slideshow Title Goes Here Here random may work well
- 25. 25 Boston University Slideshow Title Goes Here k-means++ algorithm • interpolate between the two methods • let D(x) be the distance between x and the nearest center selected so far • choose next center with probability proportional to (D(x))a = Da(x) a = 0 random initialization a = ∞ furthestfirst traversal a = 2 kmeans++
- 26. 26 Boston University Slideshow Title Goes Here k-means++ algorithm • initialization phase: • choose the first center uniformly at random • choose next center with probability proportional to D2(x) • iteration phase: • iterate as in the k-means algorithm until convergence
- 27. 27 Boston University Slideshow Title Goes Here k-means++ initialization 1 2 3
- 28. 28 Boston University Slideshow Title Goes Here k-means++ result
- 29. 29 Boston University Slideshow Title Goes Here • approximation guarantee comes just from the first iteration (initialization) • subsequent iterations can only improve cost k-means++ provable guarantee
- 30. 30 Boston University Slideshow Title Goes Here Lesson learned • no reason to use k-means and not k-means++ • k-means++ : • easy to implement • provable guarantee • works well in practice
- 31. 31 k-means-- ● Algorithm 4.1 in [Chawla & Gionis SDM 2013]