Poggi analytics - clustering - 1

Buenos Aires, junio de 2016
Eduardo Poggi

Clustering
 Supervised vs. Unsupervised Learning
 Clustering Concepts
 Non-Hierarchical Clustering
 K-means
 EM-Algorithm
 Hierarchical Clustering
 Hierarchical Agglomerative Clustering (HAC)

Supervised vs. UnSupervised Learning
 Supervised Learning
 Classification: partition examples into groups according to pre-defined
categories
 Regression: assign value to feature vectors
 Requires labeled data for training
 Unsupervised Learning
 Clustering: partition examples into groups when no pre-defined
categories/classes are available
 Novelty detection: find changes in data
 Outlier detection: find unusual events (e.g. hackers)
 Only instances required, but no labels

Clustering Concepts
 El objetivo básico del análisis de clusters es descubrir grupos en los
datos, de modo tal que los objetos del mismo grupo sean similares,
mientras que los objetos de diferentes grupos sean tan disímiles
como sea posible.
 Partition unlabeled examples into disjoint subsets of clusters, such
that:
 Examples within a cluster are similar
 Examples in different clusters are different
 Discover new categories in an unsupervised manner (no sample
category labels provided).

Clustering Concepts (2)
 Las aplicaciones son muy numerosas, por ejemplo la clasificación de plantas y
animales, en ciencias sociales la clasificación de personas considerando sus
costumbres y preferencias, en marketing la identificación de grupos de consumidores
con necesidades parecidas, etc.
 Cluster retrieved documents (e.g. Teoma)
 to present more organized and understandable results to user
 Detecting near duplicates
 Entity resolution
 E.g. “Thorsten Joachims” == “Thorsten B Joachims”
 Cheating detection
 Exploratory data analysis
 Automated (or semi-automated) creation of taxonomies
 e.g. Yahoo-style

Clustering Concepts (3)
 Consideraremos dos tipos de algoritmos de clustering:
 Métodos de partición: clasifican los datos en k grupos que deben cumplir los
requerimientos de una partición
 Cada grupo debe contener al menos un objeto
 Cada objeto debe pertenecer exactamente a un grupo.
 Métodos jerárquicos:
 Aglomerativos: empiezan con n clusters de una observación cada uno, en cada paso
se combinan dos grupos hasta terminar en un sólo cluster con n observaciones.
 Divisorios: comienzan con un sólo cluster de n observaciones y en cada paso se divide
un grupo en dos hasta tener n clusters con una observación cada uno.

K-Means Clustering Method
1. Ask user how many clusters they’d like. (e.g. k=5)
2. Randomly guess k cluster Center locations
3. For each datapoint find out which Center it’s closest to. (Thus each
Center “owns” a set of datapoints)
4. For each Center find the centroid of the points it owns
5. …and jumps there
6. …Repeat until terminated!
(Are we sure it will terminate?)

K-Means Step by step (1 & 2)
1. Ask user how many
clusters they’d like.
(e.g. k=5)
2. Randomly guess k
cluster Center
locations

K-Means Step by step (3)
1. Ask…
2. Randomly guess k
cluster Center
locations
3. For each datapoint
find out which Center
it’s closest to. (Thus
each Center “owns” a
set of datapoints)

K-Means Step by step (4)
1. Ask…
2. Randomly guess…
3. For each datapoint
find out which Center
it’s closest to. (Thus
each Center “owns” a
set of datapoints)
4. For each Center find
the centroid of the
points it owns

K-Means Step by step (5 & 6)
1. Ask…
2. Randomly guess…
3. For each datapoint …
4. For each Center find
the centroid of the
points it owns
5. …and jumps there
6. …Repeat until
terminated!

K-Means Q&A
 What is it trying to optimize?
 Are we sure it will terminate?
 Are we sure it will find an optimal clustering?
 How should we start it?
 How could we automatically choose the number
of centers?

K-Means Q&A (2)
 This clustering method is simple and reasonably effective.
 The final cluster centers do not represent a global
minimum but only a local one.
 Completely different final clusters can arise from
differerences in the initial randomly chosen cluster
centers.

K-Means Q&A (3)
Are we sure it will terminate?
 There are only a finite number of ways of partitioning R records into k
groups.
 So there are only a finite number of possible configurations in which all
Centers are the centroids of the points they own.
 If the configuration changes on an iteration, it must have improved the
distortion.
 So each time the configuration changes it must go to a configuration it’s
never been to before.
 So if it tried to go on forever, it would eventually run out of configurations.

K-Means Q&A (4)
 Will we find the optimal configuration?
 Can you invent a configuration that has converged, but does not
have the minimum distortion?

K-Means Q&A (5)
 Will we find the optimal configuration?
 Can you invent a configuration that has converged, but does not
have the minimum distortion?

K-Means Q&A (6)
Trying to find good optima
 Idea 1: Be careful about where you start
 Neat trick:
 Place first center on top of randomly chosen datapoint.
 Place second center on datapoint that’s as far away as possible from first center:
 Place j’th center on datapoint that’s as far away as possible from the closest of
Centers 1 through j-1
 Idea 2: Do many runs of k-means, each from a different random start
configuration
 Many other ideas floating around.

K-Means Q&A (7)
Choosing the number of Centers
 A difficult problem
 Most common approach is to try to find the solution that minimizes
the Schwarz Criterion
 Trying k from 2 to n !!
 Incrementally (k=2, then do 2-Means for each cluster, and so on…)

Common uses of K-means
 Often used as an exploratory data analysis tool
 In one-dimension, a good way to quantize realvalued variables into k
non-uniform buckets
 Used on acoustic data in speech understanding to convert waveforms
into one of k categories (known as Vector Quantization)
 Also used for choosing color palettes on old fashioned graphical
display devices!

Single Linkage Hierarchical Clustering
1. Say “Every point is its
own cluster”

Single Linkage Hierarchical Clustering (2)
own cluster”
2. Find “Most similar” pair of
clusters

own cluster”
clusters
3. Merge it into a parent
cluster

own cluster”
clusters
cluster
4. Repeat... until you’ve
merged the whole dataset
into one cluster

Hierarchical Clustering Q&A
 How do we define similarity between clusters?
 Minimum distance between points in clusters (in which case we’re
simply doing Euclidian Minimum Spanning Trees)
 Maximum distance between points in clusters
 Average distance between points in clusters
 And more…

Hierarchical Clustering Q&A (bis)

Hierarchical Clustering Q&A (2)
 Single Linkage Comments
 Also known in the trade as Hierarchical Agglomerative Clustering (note
the acronym)
 It’s nice that you get a hierarchy instead of an amorphous collection of
groups
 If you want k groups, just cut the (k-1) longest links
 There’s no real statistical or information-theoretic foundation to this.
Makes your lecturer feel a bit queasy.

Cluster Silhouettes
 Para cada ejemplo i definimos a(i), con A el cluster asignado a i
 Luego calculamos d(i, C) para los clusters distintos a A
 Nos quedamos con b(i) como la menor distancia un cluster. El cluster B para el cual este mínimo
se cumple, es decir d(i,B) = b(i) se llama el vecino del objeto i. (La segunda opción de
pertenencia)

Cluster Silhouettes (2)
 Ahora definimos s(i) como:
 Para entender el significado de s(i) veamos que sucede en las situaciones extremas:
 Cuando s(i) es cercano a 1, a(i) es decir, el promedio de las disimilaridades entre i y los objetos de su cluster
son mucho más pequeñas que b(i) la disimilaridad entre i y el cluster vecino. Por lo tanto podemos decir que
i está bien clasificado.
 Cuando s(i) es cercano a 0, b(i) y a(i) son aproximadamente iguales no es claro si i debe ser asignado a A ó al
cluster vecino. El objeto i está tan lejos de uno como de otro.
 La peor situación se da cuando s(i) es cercano a –1, a(i) es mucho más grande que b(i), entonces i en
promedio está más cerca del cluster vecino que de A.

Cluster Silhouettes (3)
0.0 0.2 0.4 0.6 0.8 1.0
Li
J
Le
P
Ti
I
K
Ta
Silhouettewidth
Averagesilhouettewidth:0.8
C1
C2
SC Interpretación
0.71-1 Fuerte estructura
0.51-0.7 Razonable estructura
0.26-0.5 La estructura es débil y podría ser artificial
< 0.25 No se ha hallado estructura

eduardopoggi@yahoo.com.ar
eduardo-poggi
http://ar.linkedin.com/in/eduardoapoggi
https://www.facebook.com/eduardo.poggi
@eduardoapoggi

Poggi analytics - clustering - 1

More Related Content

What's hot (20)

Viewers also liked (17)

Similar to Poggi analytics - clustering - 1 (20)

More from Gaston Liberman (15)

Recently uploaded (20)

Poggi analytics - clustering - 1

Editor's Notes