Sparse representation and compressive sensing

Advanced Signal Processing

Sparse Representation and Compressive Sensing

Dr. M. Sabarimalai Manikandan

Assistant Professor
Center for Excellence in Computational Engineering and Networking
Amrita University, Coimbatore Campus
E-mail: msm.sabari@gmail.com

September 16, 2011

Dr. M. Sabarimalai Manikandan Sparse Representation and Compressive Sensing


Why Signal Processing?

◮ Most natural signals are non-stationary and have highly
complex time-varying spectro-temporal characteristics.

◮ Mixture of many sources

◮ Composition of mixed events

◮ Various kinds of noise and artifacts
◮ The SP is challenging task because the natural signals are
typically having diﬀerent shapes, amplitudes, durations and
frequency content, which are not known in many diﬀerent
applications and systems



Signal Representation Using Basis Functions

◮ A set {ψn }N is called a orthonormal basis for RN if the
n=1
vectors in the set span RN and are linearly independent

◮ Let x ∈ RN×1 be the input signal that is spanned by N basis
functions {ψn }N . Then, a discrete-time signal x can be
n=1
represented as
N
x= αn ψ n = Ψα (1)
n=1

where α = [α1 , α2 , α3 , ......αN ] is the transform coefficients
vector that is computed as αn = x, ψ n .
◮ For some transform matrix, the transform coefficients vector α
has a small number of large amplitude coefficients and a large
number of small amplitude coefficients



Some of Representation or Transform Matrices

◮ Fourier transform matrix
◮ discrete cosines (DCT matrix) and discrete sines (DST matrix)
◮ Haar transform matrix
◮ wavelet and wavelet packets matrices
◮ Gabor ﬁlters
◮ curvelets, ridgelets, contourlets, bandelets, shearlets
◮ directionlets, grouplets, chirplets
◮ Hermite polynomials, and so on



Limitations of Fixed Representation Matrix

◮ The Fourier transform is suitable for analysis of the
steady-state sinusoidal signals but it fails to capture the sharp
changes and discontinuities in the signals.
◮ In the STFT-based methods, the choices for widths of the
time-window aﬀect the frequency and time resolution.

◮ The common problem in well-known wavelet transform-based
methods is which mother wavelet function and characteristic
scale provides the best time-frequency resolution for detection
of transients and non-transients.



Sparse Representation/Recovery

◮ Deﬁnition: The sparse representation theory has shown that
sparse signals can be exactly reconstructed from a small
number of elementary signals (or atoms).

◮ The sparse representation of natural signals can be achieved
by exploiting its sparsity or compressibility.

◮ A natural signal is said to be sparse signal if that can be
compactly expressed as a linear combination of a few small
number of basis vectors.

◮ Sparse representation has become an invaluable tool as
compared to direct time-domain and transform-domain signal
processing methods.



Sparse Representation: Applications

◮ audio/image/video processing tasks (compression, denoising,
deblurring, inpainting, and superresolution)

◮ speech enhancement and recognition

◮ signal detection and classiﬁcation

◮ face recognition, array processing, blind source separation

◮ sensor networks and cognitive radios

◮ power quality disturbances

◮ underwater acoustic communications

◮ data acquisition and imaging technologies



Sparse Signal: Sparsity

◮ Definition: A signal can be sparse or compressible in some
transform matrix Ψ when the transform coefficients vector α
has a small number of large amplitude coefficients and a large
number of small amplitude coefficients.

◮ Observations: Most of the energy is concentrated in a few
transform coefficients in a vector α

◮ The other N − K coefficients have less contribution in
representing a signal vector x ∈ RN×1 .

◮ The insignificant coefficients are set to zero in coding scheme.



Sparse Signal: Sparsity

◮ The value of K is computed as K = α 0 , where . 0 denotes
the ℓ0 -norm which counts the number of non-zero entries in
α.

◮ Concluding Remarks: A sparse signal x can be exactly
represented or approximated by the linear combination of K
basis functions with shorter transform coeﬃcients vector.

◮ In such a reconstruction process, the reconstruction error by a
K -term representation decays exponentially as K increases.



Sparse Representation: Dictionary Learning

◮ Need for Dictionary Learning: In practice, a signal is
composed of impulsive and oscillatory transients, spikes and
low-frequency components.

◮ Nature: The composite signal may not exhibit sparsity in one
transform basis matrix because some of its components are
sparse in one domain while other components are sparse in
another domain.

◮ The signals may exhibit sparsity in either time-domain or
frequency-domain.

◮ For example, the 50 Hz powerline signal is sparse in the
frequency-domain and the impulse or spikes component is
sparse in the time-domain.



Sparse Representation: Dictionary Learning

◮ Problem with Fixed Basis: In practice, the composite signal
(spike is superimposed on powerline signal) exhibits sparsity in
neither time-domain nor frequency-domain.

◮ In such cases, a fixed orthogonal basis functions are not
flexible enough to capture the complex local waves of a signal.

◮ For example, a fixed elementary cosine waveforms of discrete
cosine transform (DCT) matrix fails to capture transient parts
of biosignals.

◮ Detection and suppression of impulsive noise in speech
waveform.

◮ Compression of slow varying signals with spikes.


Sparse Representation: Need for Best Basis Functions

◮ Remedies: To improve the sparsity of composite signals, one
has to construct a transform matrix with the best basis
functions.

◮ One way to process such signal is to work with an large
dictionary matrix.

◮ A best basis set from a dictionary matrix used to sparsify the
data may yield highly compact representations of many
natural signals.

◮



Sparse Representation: What is Dictionary?

◮ Dictionary: A dictionary is a collection of elementary
waveforms or prototype atoms or basis functions.

◮ A dictionary matrix D of dimension N × L can be represented
as D = {ψ 1 |ψ 2 |ψ 3 |..........|ψ L }.

◮ The column vectors {ψ l }L of an dictionary D are
l=1
discrete-time elementary signals of length N × 1, called
dictionary atoms or basis functions.

◮ The atoms in the pre-deﬁned dictionary may be pairwise
orthogonal, linear independent, linear dependent, or not
orthogonal.



Sparse Representation: Classification of Dictionaries

◮ Based on the number of atoms L and the signal length N, the
pre-defined dictionary, D ∈ RN×M , could be classified into
three categories:

◮ (i ) when L > N, D is called overcomplete, or redundant
dictionary.

◮ (ii ) when L < N, D is called undercomplete dictionary.

◮ (iii ) D is said to be complete dictionary if L = N.



Some of Sparse Transform Matrices

◮ dirac and heaviside functions
◮ Fourier transform matrix and Fourier, short-time Fourier
transform (STFT)
◮ discrete cosines (DCT matrix) and discrete sines (DST matrix)
◮ Haar transform matrix
◮ wavelet and wavelet packets matrices
◮ Gabor ﬁlters
◮ curvelets, ridgelets, contourlets, bandelets, shearlets
◮ directionlets, grouplets, chirplets
◮ Hermite polynomials, and so on



Sparse Representation: Research Problems
◮ The dirac dictionary can be used to detect the spikes in a
signal and the discrete cosines dictionary can provide
sinusoidal waveforms
◮ The SR from redundant dictionaries may provide better ways
to reveal/capture the structures in nonstationary
environments
◮ The SR may offer better performance in signal modeling and
classification problems
◮ An efficient and flexible dictionary matrix has to be built for
separation of mixtures of events
◮ Many researchers have attempted to build dictionary for
specific signal processing tasks
◮ How to learn the dictionary from the training datasets


The CS Measurement System
◮ Performing reduction/compression when sensing analog
signals
◮ The CS is a new data acquisition theory
◮ The number of measurements is typically below the number of
samples obtained from the Nyquist sampling theorem
◮ The nonadaptive linear measurements of the input signal
vector are computed as
y = Φx (2)

where y is an M × 1 measurement vector, M ≪ N and Φ is
an M × N measurement/sensing matrix.
◮ Measurements using a second basis matrix Φ ∈ RM×N that is
incoherent with the sparsity basis matrix Ψ ∈ RN×N


The CS System: Reduction and Information

◮ The measurement system actually performs dimensionality
reduction

◮ Measurements are able to completely capture the useful
information content embedded in a sparse signal

◮ Measurements are information of the signals and thus can be
used as features for signal modeling

◮ If the Φ consists of elementary sinusoid waveforms, then α is
a vector of Fourier coeﬃcients.
◮ If the Φ consists of Dirac delta functions, then α is a vector of
sampled values of continuous time signal x(t).



The CS Recovery: Issues

◮ The y can be written as

y = ΦΨα (3)

◮ We define the matrix D = ΦΨ with a size of M × N.

◮ The major problem associated with CS concept is that we
have to solve an underdetermined system of equations to
recover the original signal x from the measurement vector y .

◮ This system has infinitely many solutions since the number of
equations is less than the number of unknowns
◮ It is necessary to impose constraints such as “sparsity” and
“incoherence” that are introduced for for this signal recovery
to be efficient



CS Reconstruction: Incoherence

◮ the CS recovery relies on two basic principles [?]:

◮ (i ) the row vectors of the measurement matrix Φ cannot
sparsely represent the column vectors of the sparsity matrix Ψ,
and vice versa

◮ (ii ) the number of measurements M is greater than
N
O(cKlog ( K ))

◮ these conditions can ensure that it is possible to recover the
set of nonzero elements of sparse vector α from measurements
y.

◮ the input signal x can be reconstructed by the linear
transformation of α: .



CS Reconstruction: Incoherence
◮ Sparsity basis matrix Ψ is orthonormal and the sensing matrix
Φ consists of M row vectors drawn randomly from some basis
˜
matrix Φ ∈ RN×N
◮ The mutual coherence is computed as:
√
µ(Φ, Ψ) = N max | Φk , Ψj | (4)
1≤k,j≥N

◮ It measures the largest correlation between any two elements
√
of Φ and Ψ and will take a value between 1 and N.
◮ The value of coherence is large when the elements of Φ and
Ψ are highly correlated and thus CS system requires more
measurements.
◮ The smaller value µ(Φ, Ψ) indicates maximally incoherent
bases and hence, the number of measurements will be less



CS Recovery: How many measurements?
◮ The recovery performance is perfect and optimal when the
bases are perfectly incoherent, and unavoidably decreases
when the mutual coherence µ increases.
◮ The number of measurements M required for perfect signal
reconstruction can be computed as:
M ≥ c · K · µ2 (Φ, Ψ) · log(N) (5)
where c is positive constant, µ is the mutual coherence, K is
the sparsity factor, and N is the length of the input vector.
◮ The value of coherence is large when the elements of Φ and
Ψ are highly correlated and thus CS system requires more
measurements.
◮ The smaller value µ(Φ, Ψ) indicates maximally incoherent
bases and hence, the number of measurements in (4) can be
the smallest


CS Recovery: How many measurements?

◮ Under very low mutual coherence value, k-sparse signal can
be reconstructed from k.log (N) measurements using basis
pursuit

◮ Examples of such pairs (maximal mutual incoherence)
are: Φ is the spike basis and Ψ is the Fourier basis

◮ Φ is the noiselet basis and Ψ is the wavelet basis. Noiselets
are also maximally incoherent with spikes and incoherent with
the Fourier basis.

◮ Φ is a random matrix and Ψ is any ﬁxed basis



Compressive Sensing/Measurement Matrices

◮ The the entries of Φ are: (i ) samples of independent and
identically distributed (iid) Gaussian or Bernoulli entries

◮ (ii ) randomly selected rows of an orthogonal N × N matrix

◮ The RIP says that D acts as an approximate isometry on the
set of vectors that are K -sparse, and a matrix D satisﬁes the
K −restricted isometry property if there exists the smallest
number, δs ∈ [0 1], such that
(1 − δs ) α 2 ≤ Dα 2 ≤ (1 + δs ) α 2 .
2 2 2
The constant δs depends on K , Φ, and α.



CS Recovery by ℓ1 -norm Optimization

◮ The goal of a sparse recovery algorithm is to obtain an
estimate of α given only y and D = ΦΨ

◮ The recovery of the K -sparse signal x from the measurements
y is ill-posed since M < N

◮ The CS system of equations is underdetermined
◮ the sparest vector is computed by solving the well-known
underdetermind problem with sparsity constraint,

ˆ
α = arg min α 0 subject to y = ΦΨα = Dα (6)
α

where • denotes ℓ0 -norm that counts the number of
nonzero entries in a vector.




◮ By allowing a certain degree of reconstruction error given by
the magnitude of the noise

◮ the optimization constraint is now relaxed:
λ
α = arg min{ α
ˆ 0 + y − ΦΨα 2 }
2 (7)
α 2
where λ ∈ R+ , which controls the relative importance applied
to the reconstruction error term and the sparseness term.

◮ the solution needs a combinatorial search among all possible
sparse α, which is infeasible for most problems of interest



◮ To overcome this problem, many nonlinear optimization-based
methods have been proposed to obtain sparest vector α by
converting (6) into a convex problem which relaxes the
ℓ0 -norm to an ℓ1 -norm problem

α = arg min α
ˆ 1 subject to y = ΦΨα = Dα (8)
α
λ
α = arg min{ α
ˆ 1 + y − ΦΨα 2 }
2 (9)
α 2
◮ which can be solved by linear programming such as BP, MP
and OMP
◮ The solution to equation (8) is exact or optimal if the number
of measurements K is large enough compared to the sparsity
factor K , K < M < N and the measurements are chosen
uniformly at random


SR Applications: Transients Detection
2

1

amplitude
0

−1

−2
0 20 40 60 80 100
(a)

1

0
amplitude

−1

1

0

−1
0 20 40 60 80
(b)

1
amplitude

0

−1

−2
0 20 40 60 80 100 120 140
(c) sample number

Figure: Examples of measured transients with 50 Hz power supply
waveforms: (a) spike, (b) microinterruption, and (c) oscillatory transient.


◮ The over-complete dictionary Ψ with size of N × 2N is
constructed as
Ψ = [ I C] (10)
where I is the N × N identity (or spike-like) matrix, and C is
the N × N DCT matrix.
◮
1
√ , i = 0, 0 ≤ j ≤ N −1
M
Cij = 2 π(2j+1)i (11)
M
cos( 2N ), 1 ≤ i ≤ N − 1, 0 ≤ j ≤ N −1

and the spike like matrix is constructed as
 
1 0 ··· 0 0
0 1 ··· 0 0
 
Iij = 0 0 1 0 0 (12)

. . .

. . . .. 
. . . . 0
0 0 0 ··· 1 N×N



◮ the signal x can be written as

x ≈ Ψα = [ I
˜ C]˜ = Id + Ca.
α (13)

and can be rewritten as
N N
y= dn I n + an Cn . (14)
n=1 n=1

◮ The common problem in well-known wavelet transform-based
methods is which mother wavelet function and characteristic
scale provides the best time-frequency resolution for detection
of transients and non-transients.




1. Input: N × 1 input signal vector y .
2. Specify the value of regularization parameter λ.
3. Read the N × 2N over-complete dictionary matrix Ψ.
4. Solve the ℓ1 -norm minimization problem:
α = arg minα { Ψα − y 2 + λ α 1 }
˜ 2
5. Obtain the detail and approximation coeﬃcient vectors.
6. Process detail vector for detecting boundaries of transient event.
7. Output: time-instants and transient portions



SR Applications: Impulsive Transients

Powerline with impulsive noise
1

original
0

−1

0.5
component
detail

0

−0.5
approximation

1
component

0.5
0
−0.5

0 0.02 0.04 0.06 0.08 0.1 0.12
Time (sec)

Figure: Illustrates the detail and approximation components extracted by
using the proposed method. The power supply waveform is corrupted by
spikes.




50 Hz powerline with microinterruption
1

original
0

−1
0 0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08 0.09 0.1

0.8
component

0.6
detail

0.4
0.2
0
0 0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08 0.09 0.1

1
approximation
component

0

−1
0 0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08 0.09 0.1
Time (sec)

Figure: Illustrates the detail and approximation components extracted by
using the proposed method. The The power supply waveform with
microinterruption.




low−amplitude transients high−amplitude transients
original signal

1

orignal signal
1
0
0

−1 −1
0 0.01 0.02 0.03 0.04 0.05 0.06 0 0.01 0.02 0.03 0.04 0.05 0.06
(a) (b)
(detail signal extracted)

(detail signal extracted)
1 1
detected transient

detected transient
0.5 0.5
0
0
−0.5
−0.5
−1
−1 −1.5
0 0.01 0.02 0.03 0.04 0.05 0.06 0 0.01 0.02 0.03 0.04 0.05 0.06
(c) (d)
signal extracted

signal extracted
approximation

approximation
1 1

0 0

−1 −1

0 0.01 0.02 0.03 0.04 0.05 0.06 0 0.01 0.02 0.03 0.04 0.05 0.06
Time (sec) Time (sec)
(e) (f)

Figure: Example of waveforms S1 and S2 illustrates signals corrupted by
low-amplitude transient S1 and high-amplitude transient S2 due to
capacitor switching, respectively. The detected transient events by using
our method.


0.4
with noise

signal with
0.2 1
spike

spike
0
0
−0.2
−1
0.01 0.02 0.03 0.04 0.05 0 0.01 0.02 0.03 0.04 0.05
(a1) (a2)
wavelet method

wavelet method
0.4
(First Detail)

(First Detail)
0.2 1
0 0
−0.2
−1
0 0.01 0.02 0.03 0.04 0.05 0 0.01 0.02 0.03 0.04 0.05
(b1) (b2)
wavelet method

wavelet method
(Second Detail)

(Second Detail)
0.4
1
0.2
0
0
−0.2

0 0.01 0.02 0.03 0.04 −1
(c1) 0 0.01 0.02 0.03 0.04 0.05
(c2)
0.2
by our method

detected spike
by our method
detected spike

1
0.1
0
0
−1
0.01 0.02 0.03 0.04 0.05 0 0.01 0.02 0.03 0.04 0.05
(d1) (d2)

Figure: Example of transient signals S3 and S4 : (a1) the spike buried in
strong noise with SNR value of -10 dB; (a2) the 50 Hz sinusoidal signal
aﬀected by a superimposed spike; Plots are the outputs from the
wavelet-based methods and the proposed method.


SR Applications: Removal of Powerline

original ECG signal original ECG signal

0.5 0.5
a m p litu d e

am plitude
0 0

-0.5 -0.5

200 400 600 800 1000 1200 1400 1600 1800 2000 200 400 600 800 1000 1200 1400 1600 1800 2000
original ECG signal plus powerline (10 degree) original ECG signal plus powerline (86 degree)

0.5 0.5

am plitude
a m p litu d e

0 0

-0.5
-0.5

200 400 600 800 1000 1200 1400 1600 1800 2000
200 400 600 800 1000 1200 1400 1600 1800 2000
Time (sec)
Time (sec)
Output of CS-based approach
0.5
0.5

am plitude
a m p litu d e

0
0

-0.5
-0.5
200 400 600 800 1000 1200 1400 1600 1800 2000
200 400 600 800 1000 1200 1400 1600 1800 2000 Time (sec)
Time (sec)

Figure: Removal of Powerline from ECG Signal



SR Applications: Removal of Artifacts
original ECG signal

0.5
amplitude

0

-0.5

-1
500 1000 1500 2000 2500 3000
T ime (sec)
original ECG signal plus powerline

0.5
amplitude

0

-0.5
-1
500 1000 1500 2000 2500 3000
T ime (sec)
0.5
amplitude

0

-0.5

500 1000 1500 2000 2500 3000
T ime (sec)
Output of CS-based baseline wander removal
0.4
amplitude

0.2
0
-0.2
-0.4
500 1000 1500 2000 2500 3000
T ime (sec)

Figure: Simultaneous removal of Powerline and LF artifact from ECG
Signal


Sparse Representation and Compressive Sensing

Thanks for your Attention!


Sparse representation and compressive sensing

More Related Content

What's hot (20)

Viewers also liked (7)

Similar to Sparse representation and compressive sensing (20)

More from Indian Institute of Technology Bhubaneswar (6)

Recently uploaded (20)

Sparse representation and compressive sensing