pca analysis principal component pca.ppt

Principle Component Analysis
Linkon Chowdhury
Dept. of Computer Science & Engineering, CUET

2 Department of CSE, CUET
Outline
• Introduction
• Objective
• Coordinate System
• PCA Visualization
• Steps of Principle Component Analysis
• Variance & Covariance
• Eigenvector & Eigenvalue
• Conclusion

Introduction
PCA (Principle Component Analysis) is defined as an
orthogonal linear transformation that transforms the
data to a new coordinate system such that the greatest
variance comes to lie on the first coordinate, the second
greatest variance on the second coordinate and so on.

Objective
Principal component analysis (PCA) is a way to reduce
data dimensionality
PCA projects high dimensional data to a lower dimension
PCA projects the data in the least square sense– it captures
big (principal) variability in the data and ignores small
variability

Philosophy of PCA
Introduced by Pearson (1901) and Hotelling
(1933) to describe the variation in a set of
multivariate data in terms of a set of uncorrelated
variables
We typically have a data matrix of n observations
on p correlated variables x1,x2,…xp
PCA looks for a transformation of the xi into p
new variables yi that are uncorrelated

Data set

Principal Component Analysis
Each Coordinate in Principle Component Analysis
is called Principle Component.
Ci = bi1 (x1) + bi2 (x2) + … + bin(xn)
where, Ci is the ith
principle component, bij is the
regression coefficient for observed variable j for
the principle component i and xi are the
variables/dimensions.

Principal Component Analysis[cont..]
From k original variables: x1,x2,...,xk:
Produce k new variables: y1,y2,...,yk:
y1 = a11x1 + a12x2 + ... + a1kxk
y2 = a21x1 + a22x2 + ... + a2kxk
...
yk = ak1x1 + ak2x2 + ... + akkxk

y1 = a11x1 + a12x2 + ... + a1kxk
y2 = a21x1 + a22x2 + ... + a2kxk
...

y1 = a11x1 + a12x2 + ... + a1kxk
y2 = a21x1 + a22x2 + ... + a2kxk
...
such that:
yk's are uncorrelated (orthogonal)
y1 explains as much as possible of original variance in data set
y2 explains as much as possible of remaining variance etc.

PCA: Visually
Data points are represented in a rotated orthogonal coordinate system:
the origin is the mean of the data points and the axes are provided by
the eigenvectors

Steps to Find Principle Component
1. Adjust the dataset to zero mean dataset.
2. Find the Covariance Matrix M
3. Calculate the normalized Eigenvectors and Eigenvalues
of M
4. Sort the Eigenvectors according to Eigenvalues from
highest to lowest

Eigenvector and Principle Component
It turns out that the Eigenvectors of covariance matrix of
the data set are the principle components of the data set.
Eigenvector with the highest eigenvalue is first principle
component and with the 2nd
highest eigenvalue is the
second principle component and so on

Example
AdjustedData Set=Original Data-Mean
Original Data set Adjusted Data Set
X Y
2.5 2.4
0.5 0.7
2.2 2.9
1.9 2.2
3.1 3.0
2.3 2.7
2 1.6
1 1.1
1.5 1.6
1.1 0.9
X Y
0.69 0.49
-1.31 -1.21
0.39 0.99
0.09 0.29
1.29 1.09
0.49 0.79
0.19 -0.31
-0.81 -0.81
-0.31 -0.31
-0.71 -1.01

Variance & Covariance
The variance is a measure of how far a set of numbers is
spread out.
The equation of variance is
  
1
)
( 1






n
X
X
X
X
x
Var
n
i
i
i

Variance & Covariance (cont..)
• Covariance measure how much to random variable change
together.
Equation of Covariance:
  
1
)
,
( 1






n
y
y
x
x
y
x
Cov
n
i
i
i

Covariance Matrix
A covariance matrix n*n matrix where each element can be
defined as
A Covariance Matrix on 2-Dimensional Data Set:
)
,
cov( j
i
Mij 




)
,
(
)
,
(
x
y
Cov
x
x
Cov
M 


)
,
(
)
,
(
y
y
Cov
y
x
Cov

Covariance Matrix(Cont...)







716555556
.
0
615444444
.
0
615444444
.
0
6
0.61655555
M

Eigenvector & Eigenvalue
The eigenvectors of a square matrix A are the
non-zero vectors x such that, after being multiplied by
the matrix, remain parallel to the original vector.






1
1
1
2






 3
3
 





 3
3

Eigenvector & Eigenvalue(cont..)
For each Eigenvector, the corresponding Eigenvalue is the
factor by which the eigenvector is scaled when multiplied
by the matrix.






1
1
1
2






 3
3
 





 3
3
.
1

The vector x is an eigenvector of the matrix A with
eigenvalue λ (lambda) if the following equation holds:
0
)
(
,
0
,





x
I
A
or
x
Ax
or
x
Ax




Calculating Eigenvalues
Calculating Eigenvector
0

 I
A 
  0

 x
I
A 

Example…
Suppose A is a matrix
Finding Eigenvalue using
or,






2
1
1
A
2
2
0






3
1
1
0

 I
A 




 
2
1
1 
2
2
0










3
1
1
0

   
3
,
2
,
1
0
3
2
1











Example…
Finding Eigenvector using
For ,λ=1
So, Let, x=k and y=-k
Eigenvector x1 is
  0

 x
I
A 





2
1
0
2
1
0






2
1
1










z
y
x











0
0
0
0
0





z
y
x
z











0
k
k












0
1
1

Example…
For λ=2,
Eigenvector x2 =
For λ=3,
Eigenvector x3 =
So, Normalized Eigenvector x =











2
1
2












2
1
1






0
1
1
2
1
2








2
1
1

4.0 4.5 5.0 5.5 6.0
2
3
4
5
1st Principal
Component, y1
2nd Principal
Component, y2
PCA Presentation

PCA Scores
4.0 4.5 5.0 5.5 6.0
2
3
4
5
xi2
xi1
yi,1 yi,2

PCA Eigenvalues
4.0 4.5 5.0 5.5 6.0
2
3
4
5
λ1
λ2

Application
Uses:
Data Visualization
Data Reduction
Data Classification
Trend Analysis
Factor Analysis
Noise Reduction
Examples:
How many unique “sub-sets” are in the
sample?
How are they similar / different?
What are the underlying factors that influence
the samples?
Which time / temporal trends are
(anti)correlated?
Which measurements are needed to
differentiate?
How to best present what is “interesting”?
Which “sub-set” does this new sample
rightfully belong?

Thanks to All

pca analysis principal component pca.ppt

More Related Content

Similar to pca analysis principal component pca.ppt (20)

Recently uploaded (20)

pca analysis principal component pca.ppt