Non-negative Matrix Factorization

Non-negative Matrix Factorization:
Applications and Algorithms
Trial Lecture
Akanksha Agrawal

Matrix
1 2 3
1 55 119 11
2 -112 456 154
3 513 33 223
4 324 123 543
4 x 3
A =

Matrix
1 2 3
1 55 119 11
2 -112 456 154
3 513 33 223
4 324 123 543
4 x 3
A =
A;1

Matrix
1 2 3
1 55 119 11
2 -112 456 154
3 513 33 223
4 324 123 543
4 x 3
A =
A2;

Matrix
1 2 3
1 55 119 11
2 -112 456 154
3 513 33 223
4 324 123 543
4 x 3
A =
a32

n x m
= xA W H
Minimize k
n x k
k x m
Factor of a Matrix

n x m
= xA W H
Minimize k
k is the rank, r of A
n x k
k x m
Factor of a Matrix

n x m n x k
= x
A basis
k ≤ r
k x m
W H
Factor of a Matrix

n x m
= x
k ≤ r
n x r
r x m
H
Factor of a Matrix
A basis

n x m
= x
k ≤ r
a1 a2 ar+ + +
n x r
r x m
H
Factor of a Matrix
A basis
j

n x m
= x
k ≤ r
a1 a2 ar+ + +
j
a1
a2
ar
n x r
r x m
Factor of a Matrix
A basis
j

n x m
= x
r ≤ k
n x k
k x m
A H
Factor of a Matrix

n x m
= x
r ≤ k
n x k
k x m
Can obtain a generating
set of the vector space
spanned by columns of A
A H
Factor of a Matrix

Non-Negative Matrix
1 2 3
1 55 119 11
2 -112 456 154
3 513 33 223
4 324 123 543
4 x 3
All elements are non-negative

Non-Negative Matrix
1 2 3
1 55 119 11
2 112 456 154
3 513 33 223
4 324 123 543
4 x 3
All elements are non-negative

Non-negative (Exact) Factor of a Non-negative
Matrix
n x m
= xA W H
Minimize k
n x k
k x m
non-negative non-negative non-negative

Non-negative (Exact) Factor of a Non-negative
Matrix
n x m
= xA W H
Minimize k
n x k
k x m
Non-negative
rank

Decision Version of the Problem
Exact Non-negative Matrix Factorization (ENMF)
Input:
Question:
An n x m non-negative matrix A and an
integer k.
Are there non-negative matrices W and H
such that A = W x H, W is of order
n x k, and H is of order k x m?

(A,k)
n x m
A
a11 a12 a1m
a21 a22 a2m
an1 an2 anm
(Cohen and Rothblum)
A Simple Algorithm for ENMF

= W
x
n x k
w11 w12 w1k
w21 w22 w2k
wn1 wn2 wnk
k x m
h11 h12 h1m
h21 h22 h2m
wk1 wk2 wkm
Create variables
n x m
A
a11 a12 a1m
a21 a22 a2m
an1 an2 anm
H

n x m
A
W
a11 a12 a1m
a21 a22 a2m
an1 an2 anm
x
n x k
w11 w12 w1k
w21 w22 w2k
wn1 wn2 wnk
k x m
h11 h12 h1m
h21 h22 h2m
wk1 wk2 wkm
Create variables
=
H
Create polynomial constraints:
[Const(A,k)]
1. For all i,j wij, hij ≥ 0.

n x m
A
W
a11 a12 a1m
a21 a22 a2m
an1 an2 anm
x
n x k
w11 w12 w1k
w21 w22 w2k
wn1 wn2 wnk
H
k x m
h11 h12 h1m
h21 h22 h2m
wk1 wk2 wkm
Create variables
Create polynomial constraints:
[Const(A,k)]
1. For all i,j wij, hij ≥ 0.
2. For all i,j, aij = wik hkj.∑
k
=

(A,k)
n x m
A
a11 a12 a1m
a21 a22 a2m
an1 an2 anm
Number of variables:
Number of polynomial
constraints:
nk + km
nm + nk + km
(A,k) is a yes-instance of ENMF if and only if
Const(A,k) is satisﬁable (over reals)

(A,k)
n x m
A
a11 a12 a1m
a21 a22 a2m
an1 an2 anm
constraints:
nk + km
nm + nk + km
We can ﬁnd a solution to
a set of polynomial
inequalities in time (Dp)O(x)

(A,k)
n x m
A
a11 a12 a1m
a21 a22 a2m
an1 an2 anm
constraints:
nk + km
nm + nk + km
a set of polynomial
x: number of variables
p: number of inequalities
D: Maximum degree of a
polynomial inequality

(A,k)
n x m
A
a11 a12 a1m
a21 a22 a2m
an1 an2 anm
constraints:
nk + km
nm + nk + km
We can decide if Const(A,k) is
satisﬁable in time O((nm)O(k(n+m)))

An Illustration of Variable Reduction
Simplicial Factorization
Input:
Question:
An n x m non-negative matrix A of rank k.
Are there non-negative matrices W and H
such that A = W x H, W is of order n x k,
and H is of order k x m?

n x m
xA W H
n x k
k x m
Rank k
=

n x m
= xA W H
n x k
k x m
Rank k Full column rank Full row rank

n x m
x
k ≤ r
n x r
r x m
H
Factor of a Matrix
A basis
=

n x m
xA W H
n x k
k x m
Rank k Full column rank Full row rank
=

Goal: To design an algorithm for Simplicial
Factorization that runs in time O((nm)O(r )).2

Goal: To design an algorithm for Simplicial
Factorization that runs in time O((nm)O(r )).2
Follow similar approach as the algorithm for ENMF, but
apply with reduced number of variables.

Pseudo Inverse
Consider a full column (or row) rank
matrix Mp,q of rank p (q).
M+ has all real entries;
M+ has order q x p;
M+ x M = Iq,q and M x M+ = Ip,p.
The (unique) pseudo inverse M+, of M satisﬁes
the following:

n x m
= xA W H
n x k
k x m
W+
W+ has order k x n and H+ has order m x k;
W+ x W = Ik,k and H x H+ = Ik,k.
H+Pseudo inverse:

n x m
= xA W H
n x k
k x m
A;i
H;i
W+ A;i = W+ W H;i = H;i

n x m
= xA W H
n x k
k x m
A;i
H;i
k x k k x 1
W+ A;i = W+ W H;i = H;i

n x m
= xA W H
n x k
k x m
A;i
W+ A;i = W+ W H;i = H;i
H;i

n x m
= xA W H
n x k
k x m
A;i
H;i
aji Wj;
W+ A;i = W+ W H;i = H;i

n x m
= xA W H
n x k
k x m
A;i
H;i
W+ A;i = W+ W H;i = H;i
Aj; H+ = Wj; H H+ = Wj;

C = {U1, U2,…, Uk} : A column basis for A.
R = {V1, V2,…, Vk} : A row basis for A.
A
Columns of A
expressed in basic C
a1U1 + a2U2 + … + akUk
j AC
k x mn x m
a1
a2
ak
j

AC AR
Columns of A
expressed in basic C
Rows of A
expressed in basic R
n x kk x m
C = {U1, U2,…, Uk} : A column basis for A.
R = {V1, V2,…, Vk} : A row basis for A.

TC AC and AR TR are non-negative;
AR TR TC AC = A.
Lemma: A has a simplicial factor if and only if the for
every column and row basis C and R of A
there are k x k matrices TC and TR such that:

AR TR TC AC = A.
A has a simplicial factors
by the two conditions and the
construction of AC and AR.
n x k k x m

AR TR TC AC = A.
A = W x H
n x k k x m
U and V be column and
row basis respectively

A = W x H
n x k k x m
U
n x k
V
k x m

A = W x H
n x k k x m
U
n x k
V
k x m
TC = W+ x U
TR = V x H+

A = W x H
n x k k x m
row basis respectively k x k
TC = W+ x U
TR = V x H+

A = W x H
n x k k x m
TC = W+ x U
TR = V x H+
k x k
TC x AC = W+ x U x AC = H
W+ A;i = W+ W H;i = H;i
Aj; H+ = Wj; H H+ = Wj;
(non -ve)

A = W x H
n x k k x m
TC = W+ x U
TR = V x H+
k x k
TC x AC = W+ x U x AC = H
W+ A;i = W+ W H;i = H;i
Aj; H+ = Wj; H H+ = Wj;
AR x TR = AR x V x H+ = W
(non -ve)
(non -ve)

A = W x H
n x k k x m
TC = W+ x U
TR = V x H+
k x k
TC x AC = W+ x U x AC = H (non -ve)
AR x TR = AR x V x H+ = W (non -ve)
AR TR TC AC = A.

U n x k V k x m
n x m
A
a11 a12 a1m
a21 a22 a2m
an1 an2 anm

n x m
A
a11 a12 a1m
a21 a22 a2m
an1 an2 anm
AR AC
k x mn x k k x k
w11 w12 w1k
w21 w22 w2k
wk1 wk2 wkk
TR
k x k
h11 h12 h1k
h21 h22 h2k
hk1 hk2 hkk
TC
=

n x m
A
a11 a12 a1m
a21 a22 a2m
an1 an2 anm
constraints:
2k2
poly(n,m,k)
a set of polynomial
x: number of variables
p: number of inequalities
D: Maximum degree of a
polynomial inequality

n x m
A
a11 a12 a1m
a21 a22 a2m
an1 an2 anm
constraints:
2k2
poly(n,m,k)
We can solve Simplicial Factorization
in time O((nm)O(k )).
2

Other Results on ENMF
[Vavasis] ENMF is known to be NP-Hard.

[Arora et al.] Assuming ETH, there is no algorithm for
ENMF running in time O((nm)o(k)).

2
[Arora et al.] Assuming ETH, there is no algorithm for
ENMF running in time O((nm)o(k)).
[Moitra] EMNF admits an algorithm running in time
O((nm)O(k )).

n x m
= xA W H
n x k
k x m
Exact Non-negative Matrix Factorization
For most applications, close
approximation is good enough.

n x m
xA W H
n x k
k x m
Non-negative Matrix Factorization
For most applications, close
approximation is good enough.
≈

Notions of closeness
Is f symmetric?
Distance
f: M x M —> R [ {1}
Divergence

Example: Distance Function
Square of Euclidean distance:
For matrices A and B (of same order)
|| A - B ||2 = (Aij - Bij)2∑
i,j
|| A - B ||2 = 0 if and only if A = B

Example: Divergence Function
For matrices A and B (of same order)
D(A || B ) = (Aij log (Aij/Bij) - Aij + Bij)∑
i,j
D(A || B ) = 0 if and only if A = B

General Scheme of Algorithm: Non-negative
Matrix Factorization
Input:
Output:
A, W(0), H(0), and t=1.
W and H.

General Scheme of Algorithm: Non-negative
Matrix Factorization
1. Fix H(t-1) and find W(t), such that D(A, W(t)H(t-1)) ≤
D(A, W(t-1)H(t-1)).
2. Fix W(t) and find H(t), such that D(A, W(t)H(t)) ≤ D(A,
W(t)H(t-1)).
3. If convergence satisfied return W and H.
4. t=t+1.
Input:
Output:
A, W(0), H(0), and t=1.
W and H.
While true

Main Challenges in Designing Better NMF
Algorithms
Getting a good seeding for initialisation of W and H.

Algorithms
Devising updating rules for W and H at subsequent
iterations.

Algorithms
iterations.
Selecting distance/ divergence norms based on the
application.

Algorithms
iterations.
Selecting distance/ divergence norms based on the
application.
Proving/ giving enough evidences for convergence of
the algorithm.

Why are non-
negative matrices and
non-negative
factorisations
important?

Origin of Non-negative Matrix Factorization
Evolved from Principal Component Analysis, which is
used for dimension reduction.
Disadvantage: Both positive and
negative elements appear in
principal components and
coefﬁcients in linear combinations.
Hard to interpret results in
applications like storing pixel
brightness

Applications
Image Processing.
The work due to Lee and Seung (1970)
attracted lot of attention to NMF

Applications
Image Processing.
Data represented as a non-negative matrix of
pixels.
NMF can ﬁnd A W x H≈
W is the basis matrix, its
column can be regarded as
parts like nose, ear, eye,
etc.

Applications
Image Processing.
pixels.
H encodes weights of each
of basic parts of the face.

Applications
Image Processing.
pixels.
Therefore, we obtain a
compressed from of data.

Applications
Clustering
This is regarded as one of the most
successful applications of NFM.

Applications
Clustering
pixels.
Columns of it are samples
described by n features

Applications
Clustering
pixels.
W which is of order n x k,
k, which denotes the
number of clusters.

Applications
Clustering
pixels.
H is used as the cluster
membership indicator
matrix.

Applications
Clustering
pixels.
Sample i is in cluster j
if Hji is the largest
value in H;i.

Applications
Financial Data Mining
The stock price ﬂuctuations seem to be
dominated by several underlying factors. NMF
has been used to obtain underlying trends
from the stock market data.

Non-negative Matrix Factorization

More Related Content

What's hot (20)

Similar to Non-negative Matrix Factorization (20)

More from AkankshaAgrawal55 (17)

Recently uploaded (20)

Non-negative Matrix Factorization