A Unified PDE model for image multi-phase segmentation and grey-scale inpainting phd-kanpur

A Unified model of Cahn-Hilliard Greyscale
Inpainting and Multimodal Classification
Rowthu Vijayakrishna
Y9108070
Dept. of Mathematics and Statistics,
IIT Kanpur
Thesis Supervisor: Dr. B. V. Rathish Kumar
(Open Seminar)

Overview
● Image Basics, Segmentation and Inpainting
● Modified Cahn-Hilliard model(MCH) for Binary Inpainting
● Threshold based MCH for Binary Segmentation
● Proposed model using the Image Histogram
● Wellposedness
● Various regularisers for smooth histograms
● Convexity Splitting scheme (in time) for the Implementation of the model and
the space-time error estimates
● Segmentation and Inpainting Experiments
● Colour Inpainting and Medical data Segmentation
● Conclusion and References

Image
● An Image u is a mapping from a domain to a collection of image
intensities (say ).
● A digital image is a discrete version of u, i.e. a Matrix with entries as integers
in the range of 0-255 (unsigned 8-bit).
0 0 0 0 0 0 0 0 0 0
0 0 0 1 1 1 1 0 0 0
0 0 1 0 0 0 0 1 0 0
0 1 0 1 1 1 1 0 1 0
0 1 0 1 0 0 1 0 1 0
0 1 0 1 1 0 0 0 1 0
0 1 0 1 1 0 1 0 1 0
0 0 1 0 1 1 0 1 0 0
0 0 0 1 1 1 1 0 0 0
0 0 0 0 0 0 0 0 0 0

Image Processing
● Denoising : Reducing the effects from undesired intensities.
● Registration : Alignning a sequence of images for the
extraction of common features.
● Segmentation (Classification) : Detection of a meaningful
object.
● Inpainting : Filling-in of intensities on a damaged region.
● Compression : Optimal representatio of an image to save
the storage space and also to speed up the communcation
process.
● Superesolution : Enhancing the visibility of details.

Segmentation
● Segmentation of an image involves determining various classes
of objects present in that image.
● Segments are defined as the class of subregions
each representing a meaningful object or collection of objects in that image.
● Sometimes a part of pixels in the domain may belong to more than one
object class, standing for overlapped objects, called as fuzzy objects.

Inpainting
● Image inpainting is an art of retouching a partially damaged image to its
original state.
● In mathematical terms, it is an extrapolation of an image data onto the
domain D where intensities are missing or corrupted.
● Numerous models have been developed to resolve inpainting issue. While
some models are good at handling the binary images only and some can
handle cartoon type grey images.

Notations
● -Thickness of the interfacial boundary
● -Fidelity parameter
● -Time step (in Numerical Schemes)
● -Baseline width (in Multimodal Potential)
● -Dirac delta symbol
● D -Inpainting Domain
● -Standard Sobolev space notation
●
-either -norm or -norm, accordingly

Modified cahn-Hilliard model
● Andrea L. Bertozzi, Selim Esedoglu, and Alan Gillette devised an inpainting
technique for binary intensity images by modifying a fourth order nonlinear
parabolic PDE, known as Cahn-Hilliard equation (CH).
where with critical points at
● The CH equation explains a pattern formation phenomenon occurring in
mixed alloys, glasses, and polymer solutions based on molecular phase
transition theory.

Well potential function
W(u)=u2
(1-u)2

Cahn-Hilliard Binary
Segmentation(CHBS)
● We experimented for the binary classification of an image into two seperate
objects using a weaker fidelity parameter and by varying the unstable
node to which acts as a threshold of seperation between the two
classes.
● When the resulting well-potential shape will be of the form

CHBS for various thresholds
Original Source to be segmented Modified CH with threshold 0.5
Modified CH with Otsu threshold 0.37647 Modified CH with MaxEntropy threshold 0.31765

Image Histogram
● Histogram of an image is an intensity distribution representing
the frequency of each intensity k present in the range of
● A new formula for computing the histogram of an image u for all real intensity
values k in is proposed to have some theoretical advantage for
analysis and also to test variations in its form for practical purpose.

Generalized Histogram
● Define the Generalized Histogram of u as,
where with as counting measure or
Lebesgue measure accordingly as u is discrete or continuous.
● Histogram evaluation for k in using the indicator function of a
singleton set {k} is given by,

Generalized Histogram(Contd.)
● With the help of Dirac delta symbol , the Generalized Histogram can
also be expressed as,
● Moreover, the proposed new formula of generalized histogram also satisfy
the following properties.
●
● For any
●

Histogram based multi-well potential
● First, a primitive form of a multi-well potential is constructed and then a
novel method for the baseline extraction is explained which, later, be
subtracted from the potential for a much better refined multi-well potential
form.
● Define a normalized histogram for as
● Construct a multi-well potential based on the histogram of the initial image
as

Baseline Corrected potential
● Define a Baseline to the potential as
● Hence, the baseline correction for using the operator is given
by,
● Baseline correction is not always necessary.
● In the case of Discrete images,
For

Multi-well potential as a polynomial
● After the simplification, the Histogram based multi-well potential with
baseline subtraction will be of he form
● For theoretical feasibility, in place of the square of its approximation
has been used to make it an even degree polynomial. i.e.

Proposed Model
Cahn-Hilliard phase separation model with histogram based multi-well potential
is given by
● For multiphase segmentation:
● For Greyscale Inpainting:

Wellposedness
● To find a weak solution u to the classification problem (1) for the Neumann
boundary conditions, we define a subspace V of admissible class of
functions from as,
with the -norm
and represent the dual of V as V'

● Define a bilinear form
on V x V , which is known to be continuous on V x V but not coercive.
● Next, with the help of a(u, v) we give a meaning to the linear operator A on
its domain D(A), given by,
as, for , is defined by
which means in a lighter sense,

Weak form
● Taking an inner product with on both sides of the equation and
applying the Integration by parts formula followed by the usage of boundary
conditions leads to
with

Existence of Weak solution
● Theorem 1: For every given in , the initial-boundary value problem (1)
for the Multiphase segmentation possesses a unique solution u which
belongs to
Moreover, the mapping is continuous in
● Proof: To prove the existence of a weak solution u to (1) in V, we use
Faedo-Galerkin approach that involves representing u as a combination of
a countable, ordered, eigen basis of the Laplacian operator in V.

● Let be an orthonormal eigen basis to the Laplacian operator in V.
Then for some constants for each , can be written as,
● For each , consider the finite dimensional approximate version of the
problem (1) with
as

Some a-priori estimates
● For the substitution we finally arive at
● Similarly for

Regularised histograms
It may be extremely difficult to handle the numerical computations during the
evolution of the model unless the nonlinear and irregular behaviour of the
potential is not properly dealt.
● Gaussian Convolution
● Parametrizing the Generalized Histogram
● Least-Square Polynomial Approximation

Gaussian Convolution
● To regularize a histogram an easier choice is to convolve it with a mollifier of
higher order smoothness viz. Gaussian kernel with parameter . So we
define the new smoothed version of the histogram as as
where stands for the standard convolution operator and

Parametrizing the Generalized
Histogram
● Considering a smooth approximation to Dirac delta symbol using various
approximations of Heaviside function can turn the Generalized histogram
into another useful alternative for the construction of a multi-well potential for
the Inpainting application.
● Using an inverse tangent approximation to the Heaviside function the
following regular version to the histogram formula was obtained,

Least-Square Polynomial
Approximation
● An approximated polynomial of degree 2n to the histogram can lead the
segmentation model to achieve at most n number of phases associated to
the local maxima of the histogram.
● For a given degree n, define a Least squares polynomial approximation
to the histogram as,

Convexity-Splitting
● For the evolution equations which do not follow a variational principle, the
convexity splitting schemes are of great help for not only solving the gradient
systems but also a wide range of optimization problems.
● Let and be the exact and discrete-time solution to the
problem (1) at . , then the Convexity-Splitting scheme is given by,

Error due to time-discretisation
Theorem 2: There exists such that
the following error estimates uniform in time hold true for some positive numbers
P, Q and R .

Fourier Approximation
Theorem:[Ben-Yu Guo] Suppose the discrete-time solution at
with and let be the discrete Fourier approximation of
computed with N spatial grid points on .
Denote
then

Error bounds for the space-time
discretization
Theorem 3:
let be the exact solution to the proposed parabolic non-linear PDE (1)
at time with the discrete-time solution and its discrete-space Fourier
approximation as with N spatial grid points on .
Further, suppose then the combination of convexity splitting in
time with the discrete Fourier approximation in space results in a first order
convergence in time, second order convergence in space, and
in space-time combination.

Explicit formula
● Given , the solution at , we solve for , the solution at
, by applying the two dimensional Discrete Fourier Transform
(DFT) on to get an explicit formula for in frequency domain, given by
Here, the matrix entries for k=0, 1 , 2, ...., N-1 and l=0, 1 , 2, ..., M-1
are given by,

Two stage approach
● The final solutions to both the problems of Multiphase classification and the
Image Inpainting are obtained through a Two-stage process given by
Bertozzi et. al in their work on Modified Cahn-Hilliard Inpainting.
● First stage with an large enough and a weaker fidelity parameter ,
to allow the formation of larger segments but with no anticipation of sharp
structure on the interfacial boundaries.
● In the second stage, the value for the Bi-harmonic smoothing term
was always chosen very small or sometimes zero.
● In both the stages C1 and C2 were taken larger than the values of and
, respectively as per the requirements of the convexity splitting scheme.

Multiphase Segmentation
● Geometrtic Objects
● Trials on Bekeley Segmentation Data set.
● Brain-tumor segmentation
● Segmentation of layers in Interstellar Galactic space

Segmentation of Natural Objects

A Unified PDE model for image multi-phase segmentation and grey-scale inpainting phd-kanpur

Brain-tumor classification
● Segmenting complex structures in CT/MRI scan of a brain is always a
challenge. Minor structural differences in the segmentation outputs that are
generated using various algorithms are of interest to pathological experts as
there is no single general segmentation algorithm that works for all cases.
The object of interest decides the selection of an algorithm. Hence, in order
to produce few segmentations with minor structural changes for an image
we tested on a brain tumour image "Brain2" of size 377x500, published
online at UPMC repository.

Using the potential obtained from
Gaussian smoothing
(Left)Multiphase output-(center)Colour labelling-(right)Contours

Using Generalized Histogram
approximation

Using Least squares polynomial
approximation of the Histogram

On Low-Contrast images (special case)

Cahn-Hilliard Inpainting
● Our first attempt was to validate the model for some Binary images with the
Modified Cahn-Hilliard and models.
● In addition, parametric sensitive study has also been conducted for each of
the proposed regularisation techniques using a simple formula of absolute
differences.
● In the Greyscale case we have compared our results with both and
methods.

Binary Stripe
● In this case, we considered a binary image "strip" of size 100x50 containing
a single horizontal strip. The inpainting region is of size 31x11 filled by a
fixed grey value between 0 and 1.
● The existing phases 0 and 1 are linearly mapped to some other pair of
intermediate intensity values between 0 and 1, to have smooth shoulders
around the newly mapped binary phases.
● The final output is again remapped to their original intensity values by its
corresponding inverse map.

Gaussian vs MCH vs TV-H-1
Gen.Hist. vs MCH vs TV-H-1

● This particular binary inpainting experiment is to study the transition of phases
near the corner type structures in an inpainting region.
● The initial filling of the occluded region for except the , are done by
a fixed grey intensity value between 0 and 1.
● In the case we opted for "random" filling since the the constant
filling of the inpaint region, failed to produce any phase transitions, right from
the initial iterations.
Binary Cross

Gaussian vs MCH vs TV-H-1
Gen.Hist. vs MCH vs TV-H-1
Modified Cahn-Hilliard

Grey Stripes
● For the "Grey stripes" image of size 318x204 px an occlusion is made of
gradually increasing thicknesses of pixels 2, 4, 6, 8, 10, and 12, to assess
the capability of our model for various scales of occlusions, all at a single
shot.
● In addition to the parametric study a comparison with and
are also being done.

Gaussian vs TV-L2
vs TV-H-1
Gen.Hist. vs TV-L2
vs TV-H-1

Kaleidoscope Image Inpainitng
(A challenging benchmark)
● These type of images are made up of a largely varying curved objects of
different sizes. We selected an image "Kaleidoscope" of size 800x800 with
occlusions of thickness ranging from 1px to 12px placed at different
locations on the image.
● The histogram for this image is not sparse which allows us to try the
baseline removal option with varying baseline width parameters (e.g. 0, 5,
and 10) to generate extra phases for the potential.

To be inpainted on black scratches Origina Data / Expected Data

(Without baseline removal)
Generalized Histogram approach for and various baseline adjustments

Generalized Histogram approach for and various baseline adjustments

Inpainting outcome for some occlusions of 3 px width

Colour Image Inpainting
● For a colour image inpainting, the previous grey scale algorithm has to be
repeated for each channel individually.
● Let be a colour image with three channels, namely
Red, Green and Blue. Then the following system of pdes corresponding to
each of the channels give us the inpainting model for a colour image.
with initial-boundary conditions

Inpainting of Bharath-Dushyanth image (500x572) for an artificiallly made
occlusions of 3 px width

Inpainting of Raja Ravi verma's “Saariclad Woman” image (660x822) for
an artificiallly made occlusions of 6 px width

Segmentation on Medical data
● Apart from the motives guided by Robotic Vision Research the Image
segmentation algorithms have got great demand and urgency from the
research groups of Medical Imaging and Diagnostics for a variety of possible
segmentations out of which the best method can be employed in further
studies.
● None from the current state of art segmentation algorithms has the ability to
detect any kind of object(segment) with its natural boundary (as perceived
by human eye).
● By examining various kinds of segmentations of a particular anatomy that
are generated using different approaches, a physician only can pick the right
method with its parameters out of them that is giving a closer representation
to the organ of interest from its real anatomy.

● Our proposed segmentation method has high amount of flexibility interms of
parameter selection that results in a variety of segmentations of the same
input.
● Almost all of variational or PDE based segmentation models must require an
initial guess of the contour of the Object of Interest (ObI).
● Each different initialization of the contour leads to a different segmentation
output because of the non-linear components in the models. And also,
sometimes, fails to catch the smaller interior segments that are lying inside a
bigger objects boundary resulting from the non-convex nature of the
underlying functional.

Middle Ear-Cochlea detection
from CT data
● CT imaging technique provides 3D positional information and it offers
excellent contrast for different tissue types and can be done even with the
implant device in place.
● As a result, CT imaging has become a very useful tool in research and
development for the improvement of cochlear implants.
● Before a surgery, scans are commonly performed to identify abnormal
anatomical structures and to assist in surgery planning.

Original data – Otsu Thresholding- Level sets evol.- Multimodal Cahn-Hilliard

Controlled Cortical Impact(CCI)
● The consruction of the
multimodal potential is
obtained through the Least
Squares polynomial approach
to the histogram with a degree
of 20.
● In both the stages of the
computation we have taken
and in
Stage-I and in Stage-2.

Conclusion
● Introduced a new formula for an Image Histogram that suits for all types of
Image data.
● A new baseline formula has been introduced to enhance the number of wells
in the multi-well potential.
● A non-standard way of constructing a multi-well potential has been derived
through the given source Image.
● Our proposed model does not require an initialisation of the object
characteristics such as contours or phase values into the computations.
● Well-posedness has been established for the proposed model.
● Combined error estimates were obtained for both the space and time
discretizations.
● Both the Multiphase classification and the Greyscale Inpainting applications
have been handled by a single method.

References
● Cahn, John W., and John E. Hilliard. "Free energy of a nonuniform system. I.
Interfacial free energy." The Journal of chemical physics 28.2 (1958): 258-
267.
● Bertozzi, Andrea L., Selim Esedoglu, and Alan Gillette. "Inpainting of binary
images using the Cahn-Hilliard equation." IEEE Transactions on image
processing 16.1 (2007): 285-291.
● Schönlieb, Carola-Bibiane, et al. "Image Inpainting Using a Fourth-Order
Total Variation Flow." (2009).
● Chan, Tony F., and Luminita Vese. "Active contours without edges." Image
processing, IEEE transactions on 10.2 (2001): 266-277.
● Barroso, Elisa Maria Lamego. "Computational Processing and Analysis of
Ear Images." (2011).

References
● Otsu, Nobuyuki. "A threshold selection method from gray-level histograms."
Automatica 11.285-296 (1975): 23-27.
● Chan, Tony F., and Jianhong Shen. "Nontexture inpainting by curvature-
driven diffusions." Journal of Visual Communication and Image
Representation 12.4 (2001): 436-449.
● Temam, Roger. Infinite-dimensional dynamical systems in mechanics and
physics. Vol. 68. Springer Science & Business Media, 2012.
● Ben-Yu, Guo. "Spectral methods and their applications." World Scietific,
Singapore (1998).

A Unified PDE model for image multi-phase segmentation and grey-scale inpainting phd-kanpur

More Related Content

What's hot (20)

Similar to A Unified PDE model for image multi-phase segmentation and grey-scale inpainting phd-kanpur (20)

Recently uploaded (20)

A Unified PDE model for image multi-phase segmentation and grey-scale inpainting phd-kanpur