Image Inpainting Using Cloning Algorithms

Shrilaxmi Deshpande & Shalini Bhatia
International Journal for Image Processing (IJIP), Volume (6): Issue (6): 2012 397
Image Inpainting Using Cloning Algorithms
Ms. Shrilaxmi Deshpande shrivd6686@gmail.com
Computer Engineering Department
Thadomal Shahani Engineering College
Mumbai-400050, India
Ms. Shalini Bhatia shalini.tsec@gmail.com
Computer Engineering Department
Thadomal Shahani Engineering College
Mumbai-400050, India
Abstract
In image recovery image inpainting has become essential content and crucial topic in research of a
new era. The objective is to restore the image with the surrounding information or modifying an
image in a way that looks natural for the viewer. The process involves transporting and diffusing
image information. In this paper to inpaint an image cloning concept has been used. Multiscale
transformation method is used for cloning process of an image inpainting. Results are compared
with conventional methods namely Taylor expansion method, poisson editing, Shepard’s method.
Experimental analysis verifies better results and shows that Shepard’s method using multiscale
transformation not only restores small scale damages but also large damaged area and useful in
duplication of image information in an image.
Keywords: Image inpainting; Multiscale transformation; seamless cloning; poisson editing;
1. INTRODUCTION
Since from the period of Renascence the technique of inpainting for missing area has been in
practice. Image inpainting is nothing but restoring damaged region with information available in the
surrounding area. It also includes modifying image with the help of inpainting algorithms. It is
essential to fill the damaged region in such a way that it should not be identifiable by the viewer.
The artist starts canvassing from the external damaged region towards internal region to fill the
damaged region, so that the image looks even. Based on this idea Bertalmio has proposed PDE-
based algorithm for image inpainting [1] and [2], called as BSCB model. Chan and Shen presented
a Total Variation model based on Rudin-Osher-Fatemi’s Denoising bounded variation image model
[3]. Atzori and de Natale introduced edge based algorithm for image inpainting [4]. A number of
approaches have been introduced in later stages. But these can apply to repair small cracks and
small scale damages of inpainted image. Another method has been proposed by using block of
image to fill damaged region by Criminisi called block-based structure synthesis [5] and [6] and by
Efros [7]. The main idea of this technique is to select appropriate size block of the damaged image
in such a way that block should fill the damaged region. But this kind of algorithms need long time to
repair damaged area.
In recent year mean-value-coordinate-theory [8] and adaptive grid techniques are used for seamless
cloning which is used in image fusion and image reproduction. But these algorithms need lot of
preprocessing and take much duration for seamless cloning. Image inpainting using Shepard’s
method using multiscale transformation method takes less time for cloning of a target region and as
well as filling block within damaged region. The results are compared with conventional methods
namely Taylor Series Method, Poisson editing, cloning using multiscale transformation.
Experimental results show validity of image inpainting technique using Shepard’s method.

2. IMAGE INPAINTING METHODS
Consider area R and area to be inpainted as . Following methods are used for image inpainting.
2.1 Taylor Series Method
Taylor series expansion [9] for image inpainting is used to repair damaged image and to remove
unwanted objects in an image. Taylor series expansion uses heat equation of the Partial Differential
Equation (PDE). This fills the omega by using the information present on the left and the right side of
the damaged area depending on the shape, color and texture. Second order Taylor series is
obtained by approximating on Taylor series expansion given as:
(1)
This method recovers damaged area and removes unwanted objects of image but the process is
slow and recovered area is not seamless.
2.2 Cloning Algorithms
In recovery from the damaged image cloning algorithms are used, in which the user specifies the
co-ordinates for the area known as source domain. Damaged area is target domain. To interpolate
the source domain with target domain seamlessly cloning algorithms are used. Steps of algorithms:
1. Identify the co-ordinate of target domain.
2. Specify the approximate co-ordinates of source domain which is to be used for recovery.
3. Mask source domain and select the region of interest.
4. Apply cloning algorithms namely poisson editing, Multiscale transformation method,
Shepard’s method to get the seamless inpainted image.
2.2.1 Poisson Editing
Poisson editing method [10] is a mathematical tool used for seamless editing and deriving cloning of
selected region. Poisson editing includes Poisson equation:
(2)
With Dirichlet boundary condition:
(3)
where g is target cloning domain and f is source cloning domain. Point represents point on
boundary for interpolation of source and target domain. This algorithm uses laplacian pyramid
[11]. This incorporates cloning to remove and add objects seamlessly.
2.2.2 Multiscale transformation method
The method consists of multiscale scheme which resembles the Laplacian pyramid[12]. Repeatedly
upsampling and downsampling are performed over image and convolved with and fixed width
kernels, so as to operate on all scales of images [9].
Multiscale transformation is performed on both source region and masked target region. The
forward transformation consists of convolving signal with filter h1and by factor of two it is
subsampled. On subsampled data the process is repeated. At each level unsampled and unfiltered
data is kept and compute:
(4)
(5)
Where denotes level and denotes unfiltered data. ↓ represents subsampling operator.
initiates the transfer where a represent input signal.

The backward signal consists of upsampling signal by adding zero in between two samples and
convolving with filter h2. Combine the upsampled data with stored data at each level after
convolving it with another filter g as :
(6)
↑ denotes upsampling by zero. Choose h1, h2, g filter so as to accurately isolate and reconstruct
lower frequency bands of original data. To keep number of operations O(n)the filter must be small
and finite.
2.2.3 Shepard’s method
By constructing smooth membrane it is possible to formulate seamless image cloning as boundary
value problems can be effectively solved. By approximating Shepard’s scattered data interpolation
method using a convolution pyramid is easy to construct suitable membrane faster. If region of
interest is denoted by , and b(x) is the boundary value to interpolate these values inside ,
Shepard’s method defines the interpolant r at x as a weighted average of known boundary values:
(7)
Where are the boundary points. The weight function of satisfactory membrane interpolation is
obtained by:
(8)
Defining as an extension of b to the entire domain, to rewrite Shepard’s method in terms of
convolutions is given by:
(9)
If is the characteristic function corresponding to , the ratio of convolutions is as follows:
(10)
3. RESULTS AND COMPARISON
In order to verify the image inpainting algorithm based on cloning concept is used on more than ten
images with different level of damages. Consider image1 of figure 1 is damaged image which shows
the image of a wall but some text has been written on image. The algorithms discussed in this paper
are applied on image1 to remove the text. Images of figure 2 (a), (b), (c), (d) show inpainted image
after applying Taylor Series method, Poisson editing method, multiscale transformation method and
Shepard’s method respectively. Inpainted images are compared with original images using mean
square error (MSE) and peak signal noise ratio (PSNR). By comparing results of cloning algorithms
and traditional method it shows that Shepard’s methods achieve high efficiency in terms of time and
error factors. These methods used for recovering large scale damaged region and for duplicating
objects of images unlike Taylor expansion method which is used only on small scale damages.
Hence cloning algorithm used for image inpainting achieves good results and overcomes the issues
of traditional method to repair damaged image.
Multiscale transformation, Shepard’s method, poisson editing algorithm give results more seamless
than and Taylor series. Time complexity of MVSC and Taylor series are higher.Taylor Series
algorithms are used only for object removal but algorithms of convolution pyramid can replicate the
objects within the image as modification in the image.User interaction is more in Taylor expansion.

Figure 3 is considered as a sample image for object duplication. Images of figure 4 (a), (b), (c) are
the inpainted images using poisson editing, multiscale transformation and Shepard’s method
respectively for manipulation in a sample image with duplicated small bird which is seamless.
Figure1: Damaged image Considered as image1.
(a) (b) (c) (d)
Figure2: Inpainted images with removed distorted area.
Taylor series Poisson Editing Multiscale
Transformation
Shepard’s
Method
PSNR MSE PSNR MSE PSNR MSE PSNR MSE
Image 1 37.34 0.2678 46.62 0.2173 66.35 0.1507 66.4 0.1506
Image 2 24.18 0.4135 32.96 0.3282 37.29 0.2681 37.31 0.25
Image 3 65.68 0.1167 68.65 0.1056 93.46 0.1053 95.13 0.0954
Table1: MSE and PSNR values of inpainted images
Figure 3: Sample image

(a) (b) (c)
Figure 4: Inpainted images with duplicated object for image manipulation.
4. CONCLUSION
This paper discusses not only methods for removing occlusion as well as scratches from image but
also replicate the objects. For cloning of target region which is to be filled in damaged region is
cloned using Poisson editing, multiscale transformation and Shepard’s method. Taylore series
method uses PDE based algorithm and poisson editing algorithm uses Laplacian membrane and
Dirichlet equation where as Multiscale Transformation and Shepard’s method use optimized filters
which are commonly used in computer graphics. Time complexity of Multiscale transformation,
Shepard’s method, poisson editing algorithm is less than Taylor series algorithm. The experimental
result shows that the Shepard’s method using multiscale transformation is more efficient than other
methods and algorithm is fast, iterative, simple to implement and provides good results.
5. FURTHER WORK
Using above mentioned methods it is possible to duplicate objects in an image and recover
damaged image. But it is not possible to deform the objects of an image. Future work includes
embedding mesh deformation algorithm for an object, so that inpainted image will have deformed
object in it.
6. REFERENCES
[1] Bertalmio M.; Sapiro G.; V. Caselles et al; “Image Inpainting” In Kurt Akeley, editor, Siggraph
2000, Computer Graphics Proceedings. ACM Addison Wesley Longman, 2000. Page: 417-424.
[2] M. Bertalmio, Processing of flat and non-flat image information on arbitrary manifolds using
Partial Differential Equations. PhD thesis, 2001.
[3] Chan F T, Shen J H. “Variational Image Inpaining” [EB/OL].
http://www.math.ucla.edu/imagers/htmls/inp.html, 2004-11-28.
[4] L.Atzori and F. G. D Natale, “Error Concealment in video Transmission over Packet networks by
a Sketch-based Approach,” Image Communication, vol. 15 sep 1999. Page: 57-76.
[5] A. Criminisi, P. Pérez, and K. Toyama, "Region filling and object removal by exemplar- based
image inpainting," IEEE Transactions on Image Processing sep 2004, vol. 13. Page(s): 1200-
1212.
[6] C. P. Perez, Toyama K. “Region filling and object removal by Exemplar- based image
inpainting[j].” IEEE Trans. Image Processing, 2004, 13(9). Page: 1200-1212.

[7] A. A. Efros and W. T. Freeman, "Image quilting for texture synthesis and transfer," in
SIGGRAPH 2001, Computer Graphics Proceedings (E. Fiume, ed.), ACM Press / ACM
SIGGRAPH, 2001. Page(s): 341-346.
[8] Z Hua; Y Li; Jinjiang L; “Image inpainting algorithm based on contour features and improved
MVSC” In Computer Design and Applications (ICCDAS), 2010 International Conference Page:
V1-212-V1-216.
[9] S. Xiaoli; X. Chen; “Image denoising and image inpainting model based on Taylor Expansion”
Computational Intelligence and Security, 2009. International Conference, Page(s): 666-670.
[10] P. Perez, M. Gangnet, A. Blake; “poisson Image Editing” ACM Trans Graph 22, (3). Page:313-
318.
[11] Burt, P. J., and Adelson, E. H. 1983. “The Laplacian Pyramid as a image code.” IEEE Trans.
Comm 31, (4). Page(s): 532-540.
[12] Z. Farbman, R. Fattal, D. Lischinski.” Convolution pyramids” ACM Trans Graph 2011 30,(6).
Page(s):771-779

Image Inpainting Using Cloning Algorithms

More Related Content

What's hot (18)

Similar to Image Inpainting Using Cloning Algorithms (20)

Recently uploaded (20)

Image Inpainting Using Cloning Algorithms