Graph mining 2: Statistical approaches for graph mining

Graph mining 2
Statistical approaches for graph mining
Nathalie Villa-Vialaneix
nathalie.villa@toulouse.inra.fr
http://www.nathalievilla.org
Advanced mathematics for network analysis
Luchon, May 3rd 2016
Nathalie Villa-Vialaneix | Graph mining 2 1/48

Talk map...
Who am I? Statistician working in biostatistics at INRA Toulouse
My research interests are: data mining, network inference and
mining, machine learning
Purpose of this talk: presenting a few statistical tools for graph
mining (graph structure, important vertices) and clustering

Background
Unlike said so, G:
undirected and connected graph;

Background
Unlike said so, G:
with vertices V = {x1, ..., xn};
with set of edges E;

Background
Unlike said so, G:
with vertices V = {x1, ..., xn};
with set of edges E;
eventually with (positive and symmetric) weights on edges, wij
(st wii = 0, no self loop)
adjacency matrix A = (wij)i,j=1,...,n

Examples are made with...
the toy example “Les Misérables” (co-appearance network in
Hugo’s novel)
Myriel
Napoleon
MlleBaptistine
MmeMagloire
CountessDeLoGeborand
Champtercier
Cravatte
Count
OldMan
Labarre
Valjean
Marguerite
MmeDeR
Isabeau
Gervais
Tholomyes
Listolier
Fameuil
Blacheville
Favourite
Dahlia
Zephine
Fantine
MmeThenardier
Thenardier
Cosette
Javert
Fauchelevent
Bamatabois
Perpetue
Simplice
Scaufflaire
Woman1
Judge
Champmathieu
Brevet
Chenildieu
Cochepaille
Pontmercy
Boulatruelle
Eponine
Anzelma
Woman2
MotherInnocent
Gribier
Jondrette
MmeBurgon
Gavroche
Gillenormand
Magnon
MlleGillenormand
MmePontmercy
MlleVaubois
LtGillenormand
Marius
BaronessT
Mabeuf
Enjolras
Combeferre
Prouvaire
Feuilly
Courfeyrac
Bahorel
Bossuet
Joly
Grantaire
MotherPlutarch
GueulemerBabet
Claquesous
Montparnasse
Toussaint
Child1Child2
Brujon
MmeHucheloup

Hugo’s novel)
software and especially the R package igraph

Hugo’s novel)
software and especially the R package igraph
the full script and the dataset is available on my website at:
http://www.nathalievilla.org/teaching/toconet.html

Basic description of the graph
lesmis
## IGRAPH U--- 77 254 --
## + attr: layout (g/n), id (v/n), label (v/c), value (e/n)
## + edges:
## [1] 1-- 2 1-- 3 1-- 4 3-- 4 1-- 5 1-- 6 1-- 7 1-- 8 1-- 9 1--10
## [11] 11--12 4--12 3--12 1--12 12--13 12--14 12--15 12--16 17--18 17--19
## [21] 18--19 17--20 18--20 19--20 17--21 18--21 19--21 20--21 17--22 18--22
## [31] 19--22 20--22 21--22 17--23 18--23 19--23 20--23 21--23 22--23 17--24
## [41] 18--24 19--24 20--24 21--24 22--24 23--24 13--24 12--24 24--25 12--25
## [51] 25--26 24--26 12--26 25--27 12--27 17--27 26--27 12--28 24--28 26--28
## [61] 25--28 27--28 12--29 28--29 24--30 28--30 12--30 24--31 31--32 12--32
## [71] 24--32 28--32 12--33 12--34 28--34 12--35 30--35 12--36 35--36 30--36
## + ... omitted several edges
U--- means: Undirected, not Named (no name attribute for the
vertices), not Weighted (no weight attribute for the edges) and not
Bipartite

System information
## R version 3.2.5 (2016-04-14)
## Platform: x86_64-pc-linux-gnu (64-bit)
## Running under: Ubuntu 14.04.4 LTS
##
## locale:
## [1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
## [3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
## [5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
## [7] LC_PAPER=en_US.UTF-8 LC_NAME=C
## [9] LC_ADDRESS=C LC_TELEPHONE=C
## [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
##
## attached base packages:
## [1] stats graphics grDevices utils datasets methods base
##
## other attached packages:
## [1] igraph_1.0.1 knitr_1.12.3
##
## loaded via a namespace (and not attached):
## [1] magrittr_1.5 formatR_1.3 tools_3.2.5 stringi_1.0-1
## [5] highr_0.5.1 stringr_1.0.0 evaluate_0.8.3

Outline
Numerical characteristics
Clustering
Modularity optimization
Spectral clustering
Model based clustering

Sketch of this section
Issue at stake:
a graph is given

Issue at stake:
a graph is given
numerical characteristics describing the graph, the nodes, are
a standard approach to describe it

Issue at stake:
a graph is given
numerical characteristics describing the graph, the nodes, are
a standard approach to describe it
how to know that the observed value are unexpected
according to a so-called “null model”?

Standard (global) characteristics
density: |E|
n(n−1)/2
graph.density
number of triangles: triangles (see also motifs)
transitivity: number of triangles divided by the number of
triplets with at least two edges transitivity
diameter: length of the longest shortest paths between two
nodes diameter
radius: minimal length, over all vertices in the graph, of the
longest shortest path linking this vertex to another vertex
radius
girth: length of the shortest circle in the graph girth
cohesion: minimum number of vertices to remove to
disconnect the graph

Standard (global) characteristics for “Les misérables”
graph.density(lesmis); triangles(lesmis); length(triangles(lesmis))/3
## [1] 0.08680793
## + 1401/77 vertices:
## [1] 12 1 3 12 1 4 12 3 4 12 24 32 12 24 13 12 24 25 12 24 30 12 25
## [24] 71 12 25 70 12 25 69 12 25 27 12 26 24 12 26 25 12 26 27 12 26 72 12
## [47] 26 71 12 26 70 12 26 69 12 27 73 12 27 52 12 27 50 12 27 44 12 28 73
## [70] 12 28 24 12 28 25 12 28 26 12 28 27 12 28 29 12 28 30 12 28 32 12 28
## [93] 34 12 28 44 12 28 72 12 28 59 12 28 69 12 28 70 12 28 71 12 29 45 12
## [116] 30 39 12 30 38 12 30 37 12 30 35 12 30 36 12 35 39 12 35 38 12 35 36
## [139] 12 35 37 12 36 39 12 36 38 12 36 37 12 37 39 12 37 38 12 38 39 12 49
## [162] 26 12 49 28 12 49 56 12 49 59 12 49 65 12 49 69 12 49 70 12 49 72 12
## [185] 50 52 12 56 26 12 56 27 12 56 65 12 56 50 12 56 52 12 56 59 12 59 71
## [208] 12 59 65 12 69 72 12 69 71 12 69 70 12 70 72 12 70 71 12 71 72 49 26
## + ... omitted several vertices
## [1] 467
transitivity(lesmis); diameter(lesmis); radius(lesmis); girth(lesmis)
## [1] 0.4989316
## [1] 5
## [1] 3
## $girth
## [1] 3
##
## $circle
## + 3/77 vertices:
## [1] 3 1 4

Comparison with random graphs...
Erdos-Renyi model with the same number of nodes and the same
number of edges than the original graph (uniform probability to
observe an edge between two given nodes)

Method: compare the observed values with those of a large
number of randomly generated random graphs (with no loop, only
connected graphs are kept)
sample_gnm(vcount(lesmis), ecount(lesmis))

Results of the comparison with random graphs...
For B = 500 graphs (only connected graphs are kept), we have:
## density triangles transitivity diameter
## Min. :0.08681 Min. :31.00 Min. :0.05834 Min. :4.000
## 1st Qu.:0.08681 1st Qu.:43.00 1st Qu.:0.07907 1st Qu.:4.000
## Median :0.08681 Median :47.00 Median :0.08701 Median :5.000
## Mean :0.08681 Mean :47.55 Mean :0.08660 Mean :4.627
## 3rd Qu.:0.08681 3rd Qu.:52.00 3rd Qu.:0.09415 3rd Qu.:5.000
## Max. :0.08681 Max. :67.00 Max. :0.11793 Max. :6.000
## radius girth cohesion
## Min. :3.000 Min. :3 Min. :1.000
## 1st Qu.:3.000 1st Qu.:3 1st Qu.:1.000
## Median :3.000 Median :3 Median :2.000
## Mean :3.004 Mean :3 Mean :1.599
## 3rd Qu.:3.000 3rd Qu.:3 3rd Qu.:2.000
## Max. :4.000 Max. :3 Max. :3.000
compared to: 0.0868079, 467, 0.4989316, 5, 3, 3, 1
⇒ all values are standard except for:
the number of triangles and the transitivity which are larger:
local connectivity is strongest than expected in Erdos-Renyi
random graphs
the cohesion which is in the lowest values of what is expected
in Erdos-Renyi random graphs: this again indicates a
strongest local connectivity

Standard (local) characteristics
... for the vertex xi:
degree: {xj : (xi, xj) ∈ E, j i} degree (or strength for the
weighted version, j i wij)
betweenness (or centrality): number of shortest paths
between any pair of vertices in the graph which pass through
xi betweenness
eccentricity: maximal length of all the shortest paths going
from xi to any other vertex in the graph eccentricity
closeness (or closeness centrality): 1
j i d(xi,xj)
in which d(xi, xj)
is the length of the shortest path between xi and xj closeness
...and their distributions among all vertices.

Standard (local) characteristics for “Les misérables”
summary(degree(lesmis))
## Min. 1st Qu. Median Mean 3rd Qu. Max.
## 1.000 2.000 6.000 6.597 10.000 36.000
summary(betweenness(lesmis))
## 0.00 0.00 0.00 62.36 22.92 1624.00
summary(eccentricity(lesmis))
## 3.00 4.00 4.00 4.13 5.00 5.00
summary(closeness(lesmis))
## 0.003378 0.004484 0.005181 0.005123 0.005435 0.008475

Method: compare the observed values (average betweenness and
degree) with those of a large number of randomly generated
random graphs (with no loop, only connected graphs are kept)
sample_gnm(vcount(lesmis), ecount(lesmis))

For B = 500 graphs (only connected graphs are kept), we have:
## degree betweenness eccentricity closeness
## Min. :6.597 Min. :54.64 Min. :3.597 Min. :0.005249
## 1st Qu.:6.597 1st Qu.:55.93 1st Qu.:3.779 1st Qu.:0.005322
## Median :6.597 Median :56.32 Median :3.857 Median :0.005340
## Mean :6.597 Mean :56.36 Mean :3.863 Mean :0.005340
## 3rd Qu.:6.597 3rd Qu.:56.71 3rd Qu.:3.909 3rd Qu.:0.005361
## Max. :6.597 Max. :58.79 Max. :4.688 Max. :0.005430
compared to: 6.597, 62.364, 4.13, 0.00512
⇒ the observed average betweenness is higher and the observed
average closeness is smaller for all the randomly generated
graphs: this seems to indicate that, in average, shortest paths in
the graphs are longer than expected for graphs with uniform
distribution of the edges.

Degree distribution for “Les misérables”
+
+
+
+
+
+
+
+
+
+
+++
++++ +
0 1 2 3
−4.0−3.5−3.0−2.5−2.0−1.5
log(k)
log(P(k))
Estimation of power law ﬁt (left: α = 1.49) with
fit_power_law(degree(lesmis) + 1, implementation =
"R.mle")

Scale free model with a parameter for the power law identical to
the one previously estimated and the same number of nodes.
Barabási and Albert model is used with a number of edges added
at each step which is chosen so that the ﬁnal number of edges
resembles that of the original graph (3 edges, which gives 225
edges in the ﬁnal graph, compared to 254)
P(degree = k) = k−α

Scale free model with a parameter for the power law identical to
the one previously estimated and the same number of nodes.
Barabási and Albert model is used with a number of edges added
at each step which is chosen so that the ﬁnal number of edges
resembles that of the original graph (3 edges, which gives 225
edges in the ﬁnal graph, compared to 254)
P(degree = k) = k−α
Method: compare the observed values with those of a large
number of randomly generated random graphs
sample_pa(vcount(lesmis), m = 3, power = ..., directed =
FALSE)

For B = 500 graphs, we have:
## density triangles transitivity diameter
## Min. :0.0769 Min. : 72 Min. :0.1075 Min. :3.000
## 1st Qu.:0.0769 1st Qu.:102 1st Qu.:0.1250 1st Qu.:4.000
## Median :0.0769 Median :112 Median :0.1307 Median :4.000
## Mean :0.0769 Mean :113 Mean :0.1303 Mean :3.988
## 3rd Qu.:0.0769 3rd Qu.:124 3rd Qu.:0.1359 3rd Qu.:4.000
## Max. :0.0769 Max. :153 Max. :0.1530 Max. :5.000
## radius girth cohesion degree betweenness
## Min. :2.000 Min. :3 Min. :3 Min. :5.844 Min. :41.86
## 1st Qu.:2.000 1st Qu.:3 1st Qu.:3 1st Qu.:5.844 1st Qu.:47.88
## Median :2.000 Median :3 Median :3 Median :5.844 Median :49.55
## Mean :2.314 Mean :3 Mean :3 Mean :5.844 Mean :49.35
## 3rd Qu.:3.000 3rd Qu.:3 3rd Qu.:3 3rd Qu.:5.844 3rd Qu.:50.97
## Max. :3.000 Max. :3 Max. :3 Max. :5.844 Max. :55.73
## eccentricity closeness
## Min. :2.935 Min. :0.005407
## 1st Qu.:3.130 1st Qu.:0.005695
## Median :3.221 Median :0.005788
## Mean :3.234 Mean :0.005805
## 3rd Qu.:3.325 3rd Qu.:0.005901
## Max. :3.662 Max. :0.006334
compared to: 0.087, 467, 0.499, 5, 3, 3, 1, 6.597, 62.364, 4.13, 0.00512
⇒ the number of triangles, the transitivity, the radius, the average degree, the
average betweenness and the eccentricity are larger than in power law graphs
with power 1.495, whereas the cohesion and the closeness are smaller.

Limits of the previous approaches
Until now, we have compared the real graph to graphs randomly
generated according to a given random model but:
this approach only gives information about global
characteristics of the observed graph;
none of the distributions of the current characteristics is
preserved during the process, especially not the degree
distribution which is central for controlling local/global
connectivity, counts of speciﬁc patterns...

A null model closer to the real graph...
Sketch of statistical tests on graphs
1. sample at random within the set of graphs with the same
degree distribution than the observed graph (B times)
2. compute a numerical statistics for each of these randomly
generated graphs
3. comparing the observed value of the statistics and its
distribution over the random graphs, a p-value can be derived
(for B large enough)

A null model closer to the real graph...
Sketch of statistical tests on graphs
1. sample at random within the set of graphs with the same
degree distribution than the observed graph (B times)
2. compute a numerical statistics for each of these randomly
generated graphs
3. comparing the observed value of the statistics and its
distribution over the random graphs, a p-value can be derived
(for B large enough)
Two main approaches to sample at random with fixed degrees:
configuration model [Bender and Canfield, 1978]
permutation approach [Rao et al., 1996, Roberts Jr., 2000]

Sampling at random within the set of graphs with a given
degree distribution
Aim:
all graphs can exhaustively be sampled
all graphs have the same probability to be sampled
⇒ MCMC approach

Sampling at random within the set of graphs with a given
degree distribution
Aim:
all graphs can exhaustively be sampled
all graphs have the same probability to be sampled
⇒ MCMC approach
Method:
1: Start from the observed graph G
2: for t = 1 → T do
3: Select uniformly at random two edges e1
= (x1
i , x1
j ) and e2
= (x2
i , x2
j ) ∈ E
4: E ← E {e1
, e2
} ∪ {e1
s , e2
s } with e1
s = (x1
i , x2
j ) and e2
s = (x2
i , x1
j )
5: if G = (V, E ) is simple and connected then
6: G ← G
7: end if
8: end for
9: return G

In practice...
This method is used in [Milo et al., 2004] with T = 100. It can be
performed using rewire(lesmis, keeping_degseq(n = 100))
Number of triangles
Frequency
200 300 400
020406080100120
transitivity
Frequency
0.25 0.35 0.45
020406080100

In practice... for the vertex characteristics
Find a(n empirical) p-value for all vertices which indicates if its
betweenness is higher or lower than expected with respect to its
degree: ratio of random graphs for which the observed
betweenness is higher (resp. lower) than 95% of the
betweennesses for the corresponding vertex in random graphs.
Myriel
Valjean
Listolier
Fameuil
Blacheville
Favourite
Dahlia
Zephine
Fantine
Judge
Champmathieu
Brevet
Chenildieu
Cochepaille
LtGillenormand
Marius
Combeferre
Prouvaire
FeuillyCourfeyrac
BahorelJoly
Grantaire
GueulemerBabet
Claquesous
MontparnasseBrujon
MmeHucheloup

More on random graphs generation
Sometimes, one wants to compare the observed graph with a
more sophisticated (constrained) null model (taking into account
some additional information on edges or nodes for instance):
This can be achieved using the same principle and throwing
away the random graphs which do not satisfy the constrains.

Warning: The more sophisticated the model is, the more
costly the simulation would be. For instance, only removing
graphs with multiple edges and graphs which are not
connected leads to throw away 47 simulations over 500.

Warning: The more sophisticated the model is, the more
costly the simulation would be. For instance, only removing
graphs with multiple edges and graphs which are not
connected leads to throw away 47 simulations over 500.
Possible solution: [Tabourier and Cointet, 2011] use multiple edge
switching to improve the simulations such simulations.

Outline
Numerical characteristics
Clustering
Modularity optimization
Spectral clustering
Model based clustering

Issue at stake:
short overview of different types of methods for vertex
clustering
only simple clustering (although some methods for
overlapping clustering, clustering according to vertex/edge
attributes, clustering of bipartite graphs... also exist)
statistical relevance and comparison of clustering results

A short overview of vertex clustering
Purpose: Find communities or modules (i.e., groups of vertices) st
vertices inside the community are strongly connected whereas
vertices between two communities are slightly connected.

A short overview of vertex clustering
Purpose: Find communities or modules (i.e., groups of vertices) st
vertices inside the community are strongly connected whereas
vertices between two communities are slightly connected.
Some approaches to perform such task:
optimizing a given criterion (e.g., modularity maximization)
spectral clustering
model based clustering
... (see [Fortunato and Barthélémy, 2007, Schaeffer, 2007,
Brohée and van Helden, 2006])

Clustering based on criterion optimization
“Cut” criteria: Given a number of clusters, K, ﬁnd the partition
of V, C1, . . . , CK such that it solves the mincut problem, i.e., it
minimizes
cut(A1, . . . , AK ) =
1
2
K
k=1 xi∈Ak , xj Ak
wij

of V, C1, . . . , CK such that it solves the mincut problem, i.e., it
minimizes
cut(A1, . . . , AK ) =
1
2
K
k=1 xi∈Ak , xj Ak
wij
Problem: The mincut problem often separates individual
vertices from the rest of the graph.

of V, C1, . . . , CK such that it solves the “RatioCut” problem,
i.e., it minimizes
RatioCut(A1, . . . , AK ) =
1
2
K
k=1 xi∈Ak , xj Ak
wij
|Ak |
(forces larger communities than the mincut problem).

of V, C1, . . . , CK such that it solves the “NCut” problem, i.e.,
it minimizes
NCut(A1, . . . , AK ) =
1
2
K
k=1 xi∈Ak , xj Ak
wij
Vol(Ak )
in which Vol(Ak ) = xi, xj∈Ak
wij (also forces larger
communities than the mincut problem).

“Cut” criteria
“Modularity” criterion [Newman and Girvan, 2004]: Given a
number of clusters, K, ﬁnd the partition of V, C1, . . . , CK
which maximizes
Q(A1, . . . , Ak ) =
1
2m
K
k=1 xi, xj∈Ck
(wij − Pij)
with Pij: weight of a “null model” (graph with the same degree
distribution but no preferential attachment): Pij =
didj
2m with
di = 1
2 j i wij.

Advantages and drawbacks
mincut is not adapted to vertex clustering in practice (clusters
with isolated vertices)
the other three methods are NP hard to solve...

the modularity takes into account asymmetry in degree
distribution by correcting the importance of a vertex by its
degree: it is often more adapted to real life graphs
[Fortunato and Barthélémy, 2007] showed that modularity has a
small resolution issue. [Bickel and Chen, 2009] gave conditions
for consistency of the clusters obtained by modularity
optimization in Stochastic Block Models (SBM).

the modularity takes into account asymmetry in degree
distribution by correcting the importance of a vertex by its
degree: it is often more adapted to real life graphs
[Fortunato and Barthélémy, 2007] showed that modularity has a
small resolution issue. [Bickel and Chen, 2009] gave conditions
for consistency of the clusters obtained by modularity
optimization in Stochastic Block Models (SBM).
Remark: Relaxation of RatioCut problem and NCut problem gives
spectral clustering. Modularity optimization is often solved by
approximation methods.

A short description of approximation methods for
modularity optimization
simple greedy algorithms ([Newman, 2004] and
[Clauset et al., 2004] for a fast version): hierarchical clustering
which merges pairs of vertices with the highest contribution to
modularity cluster_fast_greedy

multi-level greedy algorithms ([Blondel et al., 2008], also known
as “Louvain algorithm” and [Noack and Rotta, 2009] for an
improved version): hierarchical approach in which vertices are
sometimes re-assigned to a different community in a greedy
way cluster_louvain

way cluster_louvain
simulated annealing ([Reichardt and Bornholdt, 2006] uses a
spin-glass model which, in some cases, is equivalent to
modularity maximization) cluster_spinglass(..., gamma
= 1, update.rule = "config")

way cluster_louvain
simulated annealing ([Reichardt and Bornholdt, 2006] uses a
spin-glass model which, in some cases, is equivalent to
modularity maximization) cluster_spinglass(..., gamma
= 1, update.rule = "config")
...to be compared (when usable) with the exact optimization
cluster_optimal.

Examples
res_time <- cbind(
system.time(res_hierarchical <- cluster_fast_greedy(lesmis)),
system.time(res_multilevel <- cluster_louvain(lesmis)),
system.time(res_annealing <- cluster_spinglass(lesmis)),
system.time(res_exact <- cluster_optimal(lesmis))
)[3, ]
## hierarchical multilevel annealing exact
## 0.002 0.002 1.907 21.656

Computational time (greedy approaches)
Difference (computational time) between the ﬁrst two approaches
(100 evaluations):
library(microbenchmark)
res_micro <- microbenchmark(cluster_fast_greedy(lesmis),
cluster_louvain(lesmis))
cluster_fast_greedy(lesmis)
cluster_louvain(lesmis)
1000
Time [microseconds]

Accuracy of the clustering
hierarchical − 0.5006 − 5 multilevel − 0.5556 − 6
simulated annealing − 0.5596 − 7 exact − 0.56 − 6

Assessing the relevance of a clustering
Given a graph, the modularity optimization will always return a
clustering: how to know that this clustering is meaningful? (i.e.,
that its modularity is large)

Assessing the relevance of a clustering
Given a graph, the modularity optimization will always return a
clustering: how to know that this clustering is meaningful? (i.e.,
that its modularity is large)
Similarly as previously, compare the maximum modularity to the
maximum modularity over a large number of randomly generated
graphs (with same degree sequence).
Modularity
Frequency
0.30 0.35 0.40 0.45 0.50 0.55
020406080

Relation between RatioCut and Laplacian
[von Luxburg, 2007] shows that minimizing
RatioCut(A1, A2) =
1
2
2
k=1 xi∈Ak , xj Ak
wij
|Ak |
is equivalent to the following constrained problem:
min
A1, ,A2
v Lv st v ⊥ 1n and v =
√
n
for v the vector of Rn
obtained from the partition by:
vi =
(|A2|)/|A1| if vi ∈ A1
− |A1|/(|A2|) otherwise.
and L is the Laplacian of the graph, n × n-matrix with entries:
Lij =
−wij if i j
di = j i wij otherwise
.

... and more remarks
this is a discrete (since v can only have two values) and
NP-hard problem;

NP-hard problem;
the same relation holds between NCut problem and
normalized Laplacian D−1/2
LD−1/2
is which
D = Diag(d1, . . . , dn);

NP-hard problem;
the same relation holds between NCut problem and
normalized Laplacian D−1/2
LD−1/2
is which
D = Diag(d1, . . . , dn);
a generalization of these results exist for K > 2.

Some properties of the Laplacian
Relations with the graph structure:
1
2
3
4
5
has a null space spanned by the vectors


1
1
1
0
0


and


0
0
0
1
1


.

Relations with the graph structure: the vector 1n spans the null
space for connected graphs.

Random walk point of view: If we consider a random walk on the
graph with probability to jump from one node to the other equal to
wij
di
then NCut(A1, A2) is interpreted as the probability to go from A1
to A2 or from A2 to A1.

Random walk point of view: If we consider a random walk on the
graph with probability to jump from one node to the other equal to
wij
di
then the average time to go from one node to another
(commute time) is given by L+ [Fouss et al., 2007].

Spectral clustering: relaxing the constrains
K has to be given. Solving minA1, ,A2
Tr(U LU) for a K × n matrix U
st U U = 1:
1. Compute the ﬁrst K eigenvectors of L, u1
, . . . , uK
and write
U = (u1
, . . . , uK
) (a n × K matrix).

st U U = 1:
, . . . , uK
and write
U = (u1
, . . . , uK
2. For i = 1, . . . , n, denote ui ∈ RK
the i-th row of U. Cluster the
points (ui)i=1,...,n using a clustering algorithm (e.g., k-means).

st U U = 1:
, . . . , uK
and write
U = (u1
, . . . , uK
2. For i = 1, . . . , n, denote ui ∈ RK
the i-th row of U. Cluster the
points (ui)i=1,...,n using a clustering algorithm (e.g., k-means).
embed_laplacian_matrix(..., no = ..., which = "sa",
scaled = ...) et kmeans(..., centers = ..., nstart =
10)

Spectral clustering in practice
res_time_spec <- system.time({
spec_embed <- embed_laplacian_matrix(lesmis, no = 6, which = "sa",
scaled = FALSE)
res_spectral <- kmeans(spec_embed$X[ ,-1], centers = 6, nstart = 1)
})[3]
res_time_spec
## elapsed
## 0.017
Time is between the greedy approaches for modularity
optimization and simulated annealing for modularity optimization.

spectral clustering − 0.4461 − 6 exact − 0.56 − 6
Modularity is smaller (as expected) and clusters tend to be more
unbalanced. An empirical comparison between the performance of
spectral clustering and modularity optimization is provided in
[Bickel and Chen, 2009]. [Lei and Rinaldo, 2015] gives conditions for the
consistency of spectral clustering in stochastic block models.

A mixture model for networks
[Snijders and Nowicki, 1997]: The observed network G is supposed to
be the realization of some random graph model in which vertices
are organized in groups.
description of the model
vertices xi belong to an unknow class in {C1, ..., CK } (K is
given) ⇒ latent (unobserved) variables
Zi ∼ M(1, α = (α1, . . . , αK ))
in which αk is the probability that xi belongs to Ck

Zi ∼ M(1, α = (α1, . . . , αK ))
given the class membership, the probabilities to have an edge
between xi and xj are all independant and obtained by:
wij = 1|Zik Zik = 1 ∼ L(., πkk )
for a given distribution L

Zi ∼ M(1, α = (α1, . . . , αK ))
given the class membership, the probabilities to have an edge
between xi and xj are all independant and obtained by:
typically, the Bernouilli distribution with probability πkk with
πkk =
p1 if k = k
p0 if k k
for p1 > p0.

Basic principle for using SBM
1. assignments of vertices to groups;
2. parameter estimation ((αk )k and (πkk )k,k );
3. estimation of the number of groups.

Estimation is made by Bayesian or frequentist approaches and
Variational EM (see e.g., [Daudin et al., 2008] for the more
computationally efﬁcient frequentist approach). Number of nodes
can be chosen using ICL [Biernacki et al., 2000].

Estimation is made by Bayesian or frequentist approaches and
Variational EM (see e.g., [Daudin et al., 2008] for the more
computationally efﬁcient frequentist approach). Number of nodes
can be chosen using ICL [Biernacki et al., 2000].
All this is implemented in the package blockmodels [Léger, 2016].
BM_bernoulli("SBM_sym", as_adjacency_matrix(...,
sparse = FALSE))
BM_bernoulli$estimate()

SBM in practice
library(blockmodels)
res_time_sbm <- system.time({
res_sbm <- BM_bernoulli("SBM_sym",
as_adjacency_matrix(lesmis, sparse = FALSE))
res_sbm$estimate()
})[3]
res_time_sbm
## elapsed
## 1.821
opt_K <- which.max(res_sbm$ICL)
opt_K
## [1] 6
sbm_clust <- apply(res_sbm$memberships[[opt_K]]$Z, 1, which.max)

SBM clustering − 0.4556 − 6 exact − 0.56 − 6
Modularity is smaller (as expected) but groups can be interpreted
by being sets of vertices with similar connecting patterns.

Comparing clustering
Various metrics ((di)similarities) exist to compare clustering,
among which:
Rand Index [Rand, 1971] compare(..., method = "rand"):
number of agreements between the two clusterings
n
Normalized Mutual Information [Danon et al., 2005]
compare(..., method = "nmi")
K1
k=1
K2
k =1
nkk
n
log


nkk n
n1
k
n2
k


in which Kj is the number of clusters in clustering j, n
j
k
is the
number of vertices classiﬁed into cluster k for clustering j and
nkk is the number of vertices classiﬁed into cluster k for
clustering 1 and cluster k for clustering 2. The similarity is
normalized so that it is between 0 and 1 (1 is for a perfect
match).

How do clusterings relate?
Method:
1. compute a dissimilarity based on Rand index or NMI
(1 − value)
2. perform clustering (of the results of vertex clustering) using
hierarchical clustering hclust

How do clusterings relate?
sbm
spectral
hierarchical
multilevel
annealing
exact
0.00.10.20.3
Rand index
hclust (*, "complete")
as.dist(compare_rand)
Height
sbm
spectral
hierarchical
multilevel
annealing
exact
0.00.20.40.6
NMI
hclust (*, "complete")
as.dist(compare_nmi)
Height

Any question?

Bender, E. and Canﬁeld, E. (1978).
The asymptotic number of labeled graphs with given degree sequences.
Journal of Combinatorial Theory, Series A, 24(3):296–307.
Bickel, P. and Chen, A. (2009).
A nonparametric view of network models and Newman-Girvan and other modularities.
Proceedings of the National Academy of Sciences, USA, 106(50):21068–21073.
Biernacki, C., Celeux, G., and Govaert, G. (2000).
Assessing a mixture model for clustering with the integrated completed likelihood.
IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(7):719–725.
Blondel, V., Guillaume, J., Lambiotte, R., and Lefebvre, E. (2008).
Fast unfolding of communites in large networks.
Journal of Statistical Mechanics: Theory and Experiment, P10008:1742–5468.
Brohée, S. and van Helden, J. (2006).
Evaluation of clustering algorithms for protein-protein interaction networks.
BMC Bioinformatics, 7(488).
Clauset, A., Newman, M. E. J., and Moore, C. (2004).
Finding community structure in very large networks.
Physical Review E, 70:066111.
Danon, L., Diaz-Guilera, A., Duch, J., and Arenas, A. (2005).
Comparing community structure identiﬁcation.
Journal of Statistical Mechanics, page P09008.
Daudin, J., Picard, F., and Robin, S. (2008).
A mixture model for random graphs.
Statistics and Computing, 18:173–183.
Fortunato, S. and Barthélémy, M. (2007).
Resolution limit in community detection.

In Proceedings of the National Academy of Sciences, volume 104, pages 36–41.
doi:10.1073/pnas.0605965104; URL: http://www.pnas.org/content/104/1/36.abstract.
Fouss, F., Pirotte, A., Renders, J., and Saerens, M. (2007).
Random-walk computation of similarities between nodes of a graph, with application to collaborative
recommendation.
IEEE Transactions on Knowledge and Data Engineering, 19(3):355–369.
Léger, J. (2016).
Blockmodels: a R-package for estimating in LBM and SBM, with many pdf, with or without covariates.
Preprint arXiv 1602.07587v1. Submitted for publication.
Lei, J. and Rinaldo, A. (2015).
Consistency of spectral clustering in stochastic block models.
The Annals of Statistics, 43(1):215–237.
Milo, R., Kashtan, N., Itzkovitz, S., Newman, M., and Alon, U. (2004).
On the uniform generation of random graphs with prescribed degree sequences.
eprint arXiv: cond-mat/0312028v2.
Newman, M. (2004).
Fast algorithm for detecting community structure in networks.
Physical Review E, 69:066133.
Newman, M. and Girvan, M. (2004).
Finding and evaluating community structure in networks.
Physical Review, E, 69:026113.
Noack, A. and Rotta, R. (2009).
Multi-level algorithms for modularity clustering.
In SEA 2009: Proceedings of the 8th International Symposium on Experimental Algorithms, pages 257–268,
Berlin, Heidelberg. Springer-Verlag.
Rand, W. (1971).

Objective criteria for the evaluation of clustering methods.
Journal of the American Statistical Association, 66(336):846–850.
Rao, A., Jana, R., and Bandyopadhyay, S. (1996).
A markov chain monte carlo method for generating random (0, 1)-matrices with given marginals.
Sankhyã: The Indian Journal of Statistics, Series A (1961-2002), 58(2):225–242.
Reichardt, J. and Bornholdt, S. (2006).
Statistical mechanics of community detection.
Physical Review, E, 74(016110).
Roberts Jr., J. (2000).
Simple methods for simulating sociomatrices with given marginal totals.
Social Networks, 22(3):273 – 283.
Schaeffer, S. (2007).
Graph clustering.
Computer Science Review, 1(1):27–64.
Snijders, T. and Nowicki, K. (1997).
Estimation and prediction for stochastic block-structures for graphs with latent block structure.
Journal of Classiﬁcation, 14:75–100.
Tabourier, L.and Roth, C. and Cointet, J. (2011).
Generating constrained random graphs using multiple edge switches.
ACM Journal of Experimental Algorithmics, 16(1):1.7.
von Luxburg, U. (2007).
A tutorial on spectral clustering.
Statistics and Computing, 17(4):395–416.

Graph mining 2: Statistical approaches for graph mining

More Related Content

What's hot (18)

Viewers also liked (20)

Similar to Graph mining 2: Statistical approaches for graph mining (20)

More from tuxette (20)

Recently uploaded (20)

Graph mining 2: Statistical approaches for graph mining