Behavior study of entropy in a digital image through an iterative algorithm

International Journal of Soft Computing, Mathematics and Control (IJSCMC),Vol. 4, No. 3, August 2015
DOI : 10.14810/ijscmc.2015.4301 1
BEHAVIOR STUDY OF ENTROPY IN A DIGITAL IMAGE
THROUGH AN ITERATIVE ALGORITHM OF THE
MEAN SHIFT FILTERING
Esley Torres, Yasel Garces, Osvaldo Pereira and Roberto Rodriguez
Digital Signal Processing Group,
Institute of Cybernetics, Mathematics and Physics (ICIMAF), Havana, Cuba
ABSTRACT
Image segmentation is a critical step in computer vision tasks constituting an essential issue for pattern
recognition and visual interpretation. In this paper, we study the behavior of entropy in digital images
through an iterative algorithm of mean shift filtering. The order of a digital image in gray levels is defined.
The behavior of Shannon entropy is analyzed and then compared, taking into account the number of
iterations of our algorithm, with the maximum entropy that could be achieved under the same order. The
use of equivalence classes it induced, which allow us to interpret entropy as a hyper-surface in real m-
dimensional space. The difference of the maximum entropy of order n and the entropy of the image is used
to group the the iterations, in order to caractrizes the performance of the algorithm.
KEYWORDS
Maximum entropy of order n, equivalence classes, image segmentation, mean shift, relative entropy
1. INTRODUCTION
Image segmentation, that is, classification of the image gray-level values into homogeneous areas
is recognized to be one of the most important steps in any image analysis system. This allows one
to analyze and to interpret the relevant zones according to the aims of observer. Although, most
of the time, the final result of segmentation depends largely on the interest of the observer; it is
possible to develop unsupervised algorithms, which reach the expected results of the interpreter.
The creation of image segmentation algorithms with a fewer adjustment parameters is a task that
has been addressed in the last decade which makes the manipulation of these algorithms easier
and less complicated.
Mean shift (MSH) is a robust technique which has been applied in many computer vision tasks,
for example: image segmentation, visual tracking, etc. [13]. MSH technique was proposed by
Fukunaga and Hostetler [5] and largely forgotten until Cheng’s paper [1] rekindled interest in it.
MSH is a versatile nonparametric density analysis tool and can provide reliable solutions in many
applications [3], [2]. In essence, MSH is an iterative mode detection algorithm in the density
distribution space. The MSH procedure moves to a kernel-weighted average of the observations
within a smoothing window. This computation is repeated until convergence is obtained at a local
density mode. This way the density modes can be located without explicitly estimating the
density. An elegant relation between the MSH and other techniques can be found in [13].
The Mean Shift iterative algorithm (‫ܵܯ‬ℎ݅) that is used in this paper is based on the mean shift
and was previously introduced and applied in several works [4], [7], [9], [10], [11]. The proposed

2
algorithm uses entropy as a stopping criterion. As result of applying this algorithm the segmented
image is obtained without loss of quality of segmentation.
The aim of this work is to study the function of entropy (E) in real digital images, with the
purpose of analyzing the behavior of this entropy regarding the probabilities of occurrence of the
gray levels, while the proposed algorithm of the mean shift is running. For more details about
entropy function see [14]. In order to achieve the results of this study it is necessary to deﬁne the
associated equivalent classes with the gray levels and the maximum entropy of n order.
The work continues as follows. In section 2, the related theoretical aspects with the largest value
reached by the entropy in images with the same quantity of gray levels will be discussed. Section
3 will present the obtained experimental results according to the comparison of entropy of the
images with respect to its maximum value. Also, an analysis of these experimental results is
carried out. In section 4, the most important conclusions are given.
This document describes, and is written to conform to, author guidelines for the journals of WSP
series. It is prepared in Microsoft Word as a .doc document. Although other means of
preparation are acceptable, final, camera-ready versions must conform to this layout. Microsoft
Word terminology is used where appropriate in this document. Although formatting instructions
may often appear daunting, the simplest approach is to use this template and insert headings and
text into it as appropriate.
2. THEORETICAL ASPECTS
In this section the important theoretical aspects corresponding to this study will be exposed, with
the aim that one can understand with more clarity the analysis that will be carried out in the
transformations of the images in the application of the iterative algorithm of the mean shift [9],
[11].
2.1. Mean Shift
We first review the basic concepts of the MSh algorithm [5]. One of the most popular
nonparametric density estimators is kernel density estimation. Given ݊ data points ‫ݔ‬௜ ,
݅ = 1, 2, 3,··· , ݊, in a neighborhood of radius ℎ, drawn from a population with density
function ݂(‫,)ݔ‬ ‫ݔ‬ ∈ ℝௗ
the estimated general multivariate kernel density at ‫ݔ‬ is defined
by:
݂(‫ݔ‬) =
1
݊ℎௗ
෍ ‫ܭ‬
௡
௜ୀଵ
ቀ
‫ݔ‬ − ‫ݔ‬௜
ℎ
ቁ, (1)
By the use of theory and profile notation given in [2], the Mean Shift vector is given by
‫ܵܯ‬ℎ௛,ீ(‫)ݔ‬ =
∑ ‫ݔ‬௜݃ ቀቛ
‫ݔ‬ − ‫ݔ‬௜
ℎ
ቛቁ௡
௜ୀଵ
∑ ݃ ቀቛ
‫ݔ‬ − ‫ݔ‬௜
ℎ
ቛቁ௡
௜ୀଵ
− ‫)2( ݔ‬
where g is the profile of G and K is a shadow kernel of G. The length of the Mean Shift vector
give a measure of how much it is close a local maximum from the point x. For more details about
this topic see [1], [2], [3].

3
In [13] was proved that the segmentation algorithm, by recursively applying mean shift, guarantee
the convergence. By simplicity, in this paper, we only approach the case d = 1 corresponding to
gray level images. For more details about this algorithm see [4], [9], [10], [11].
Therefore, if the individual mean shift procedure is guaranteed to converge, a recursively
procedure of the mean shift also converges. For more details about this algorithm see [5], [19],
[20], [21], [22].
2.2. Entropy
The entropy of a digital image is a statistical measure that expresses the randomness of gray
levels (colors) and it is defined as:
Definition 1: Comparable images
‫ܧ‬ଵ = − ෍ ‫݌‬௜݈‫݃݋‬ଶ‫݌‬௜
ଶಳିଵ
௜ୀ଴
,
where ‫ܤ‬ is the total quantity of bits of the digitized image and by agreement ݈‫)0(2݃݋‬ = 0, also
‫݌‬௜ =
௞೔
௠ଵ ∙௠ଶ
, is the probability of occurrence of color ݅,
݇௜, is the frequency of occurrence of color ݅ in the image,
݉ଵ, is number of rows of the image,
݉ଶ, is number of columns of the image,
‫݌‬௜ ∈ [0; 1] and must satisfy that ∑ ‫݌‬௜ = 1௡
௜ୀଵ , as a condition of probability ‫.݌‬
One can observe in the first expression 1 that the entropy of a digital image is a sum of terms that
depend on the probability of occurrence of gray levels of pixels, in this way its value will not only
depend on pi, but also on the quantity of gray levels present in the image. This characteristic of
entropy denotes that it is a non-trivial function to analyze its values, when several images are
compared that do not have the same quantity of gray levels. Basically, we are interested in
knowing the maximum of the function of entropy under the conditions in which the gray levels
have a fixed number.
2.3. Relative Entropy
The Relative Entropy or Kullback Leibler Distance is a nonsymetric mesure of the distance of
two distributions [14]. This function is defined as follows:
Definition 2 (Relative Entropy):
‫݌(ܦ‬ ∥ ‫)ݍ‬ = ෍ ‫݌‬௜
௫∈॒
݈‫݃݋‬
‫݌‬௜
‫ݍ‬௜
, (2)
where ‫݌‬ and ‫ݍ‬ are two probability distributions of occurrence of variable ‫ݔ‬ at the set ॒.
2.4. Classes of equivalence Abstract
Definition 3 (Relation of Equivalence)
Let K be a given non-empty set and ܴ a defined binary relationship over ‫.ܭ‬ It is said that ܴ is a
relation of equivalence if this satisfies the following properties [8]:

4
1) Reflexivity: ∀‫ݔ‬ ∈ ‫ܭ‬ ⟹ ‫.ݔܴݔ‬
2) Symmetry: ∀‫,ݔ‬ ‫ݕ‬ ∈ ‫,ܭ‬ ‫ݕܴݔ‬ ⟹ ‫.ݔܴݕ‬
3) Transitivity: ∀‫,ݔ‬ ‫,ݕ‬ ‫ݖ‬ ∈ ‫,ܭ‬ ‫,ݕܴݔ‬ ‫ݖܴݕ‬ ⟹ ‫.ݖܴݔ‬
A relation of equivalence ܴ over a ‫ܭ‬ set can be denoted as the ordered pair (‫,ܭ‬ ~ ). The relation
of equivalence denoted by the symbol ~, defines disjoint sets in ‫ܭ‬ called classes of equivalence,
i.e., given an element ‫ݔ‬ ∈ ‫ܭ‬ , the given set by all the related elements with ‫ݔ‬ , i.e., ‫ܥ‬௜ =
ሼ‫ݕ‬ ∈ ‫ݔܴݕ |ܭ‬ሽ, is called the class of equivalence associated to element ‫.ݔ‬ The element ‫ݔ‬ is called a
representative of the class.
Definition 4 (Order of Relation of Equivalence)
The order of relation of equivalence is defined as the number of classes that generates a relation,
it is denoted by the letter ݊, and the number of elements of the classes ‫ܥ‬௫ is denoted by the
symbol ݈௫. The concept of a class of equivalence is very important for digital image processing.
Indeed, given a set of objects or abstract entities, relations of equivalence based on some criterion
can be created, where the resulting classes are the “types” in which one can classify the entire
range of objects. Later, we will build classes of equivalence by using ‫ܵܯ‬ℎ݅ and we will create a
link to the relation of equivalence defined in this section.
2.5. Class of equivalence of gray levels
Definition 5 (Relation of equivalence among pixels)
Let ‫ܩ‬ be a digital image in gray levels. Let ‫ݕ,ݔ‬ be the pixels and let ܿ‫,)ݔ(݈݋‬ ܿ‫)ݕ(݈݋‬ be their
respective gray levels, the symbolized relation on ‫ܩ‬ is such that ‫ݕ ~ݔ‬ if ܿ‫)ݔ(݈݋‬ = ܿ‫)ݕ(݈݋‬,
∀݅ = 0, 1, 2, ⋯ , 2஻
− 1 (see section 2.4). By using the notation of classes of equivalence, this can
be written as ‫ܥ‬௫ = ሼ‫ݕ‬ ∈ ‫)ݔ(݈݋ܿ|ܩ‬ = ܿ‫)ݕ(݈݋‬ሽ, which represents all pixels in image whose gray
level coincides with x. Due to the fact that the pixels in the images are visually represented by
gray levels, it is not convenient to refer them by classes, but by the gray levels.
Suppose that col(x) = i, then it follows that ‫ܥ‬௫ = ሼ‫ݕ‬ ∈ ‫)ݔ(݈݋ܿ|ܩ‬ = ܿ‫)ݕ(݈݋‬ሽ, provided that
col(x) = i. Taking in consideration the above-mentioned, it is convenient to establish the classes
of equivalence as ‫ܥ‬௜ = ሼܾ ∈ ‫)ܾ(݈݋ܿ|ܩ‬ = ݅ሽ ∀݅ = 0, 1, 2, ⋯ , 2஻
− 1. This class represents the set
of pixels having gray level ݅, so the order or the size of the class ‫ܥ‬௜ coincides with what is known
as ݇௜, i.e., the frequency of gray level ݅ in the image.
In a digital image, it may occurs that a certain gray level has ‫݌‬௜ = 0, this means that physically
this gray level is not visually present for the observer. For this reason, the agreement log2(0) = 0
is assumed, and therefore these terms do not affect the domain of the function of entropy. On the
other hand, the trivial case ‫݌‬௜ = 1 may occurs, which means that all pixels in the image have the
same colour, i.e., the region is quite homogeneous, and its entropy is equal to zero. Therefore, we
are not interested in these cases because entropy reaches the minimum value. In this way, the
values ‫݌‬௜ = 0 and ‫݌‬௜ = 1 are ignored. This means that the image to analyze has a certain number
of gray levels (such as minimum 2).
Let ‫ܫ‬ = ሼ݅ ∈ [0, 2஻
− 1] |݊௜ ≠ 0ሽ be the set of gray levels whose classes of equivalence are non-
zero. It is not difficult to see that the order of the relation of equivalence coincides with the
quantity of gray levels that can be appreciated in ‫.ܩ‬ In this way, one can say that the image has

5
order ݊. Therefore, it is possible to give a more precise definition of the order of an image in gray
levels.
Definition 6 (Order of an image in gray levels):
Let ‫ܩ‬ be a digital image in gray levels, the order of ‫ܩ‬ is defined as ݊, as the quantity of elements
of the set ሼ݅ ݃‫݌| ݈݁ݒ݈݁ ݕܽݎ‬௜ ≠ 0ሽ.
Figure 1 represents a region of a digital image of size 5x10 (50 pixels), where one can see the
gray levels corresponding to the values 15 , 16 , 49 and 159 . Note that the order of the
equivalence relation is ݊ = 4, such that the equivalence classes presented on the region are
‫ܥ‬ଵହ, ‫ܥ‬ଵ଺, ‫ܥ‬ସଽ, ‫ܥ‬ଵହଽ being respectively, ݈ଵହ = 42, ݈ଵ଺ = 6, ݈ସଽ = 1, ݈ଵହଽ = 1 and therefore, ‫݌‬ଵହ =
0.84, ‫݌‬ଵ଺ = 0.32, ‫݌‬ସଽ = 0.02, ‫݌‬ଵହଽ = 0.02. It is not casual that the value of the order of the
relation of equivalence coincides with the value of the quantity of gray levels in the image. In fact,
we are referring to the same physical entity under different names. Therefore, starting from now,
we will refer to the number of gray levels in the image with the letter ݊ (order of the relation of
equivalence), and we will indistinctly use the terms: “order of the relation of equivalence”, “order
of image” and “number of gray levels in the image”. Therefore, taking into consideration Figure 1,
the set of probabilities ‫݌‬௜ for a given image in gray levels can be expressed in the following way:
‫ܫ‬ = ሼ݅ ∈ [0, 2஻
− 1] |݇௜ ≠ 0ሽ. (3)
Figure 1. Classes of equivalence of gray levels with respect to classes of equivalence.
Observe that the difference in the expression 3 with regarding of Definition 1, it is that we now
only take into account those gray levels whose probabilities are not zero. In other words, here, we
do not consider the gray levels that are not present in the image.
2.6. Other considerations about notation of the entropy formula
It is known from information theory that the given data by a symbol ݅ of an object is supplied by
−݈‫݃݋‬ଶ ‫ݍ‬௜ , where ‫ݍ‬௜ is the relative frequency of symbol ݅ [12]. Taking the mathematical
expectation of the given information by n symbols, we obtain the entropy formula
‫ܧ‬ = − ෍ ‫ݍ‬௜݈‫݃݋‬ଶ‫ݍ‬௜
௡
௜ୀ଴
,

6
where −‫ݍ‬௜݈‫݃݋‬ଶ ‫ݍ‬௜, it is the average contribution of each symbol ݅. For such a reason, entropy is a
measure of the information average contained from each symbol of the object that is being
analyzed. Digital images in gray levels are a particular case of the information theory in which
images are objects and each gray levels represent the symbols.
Suppose one has a gray level image (assuming ݊ > 1), and starting from now, we assume that the
݊ gray levels in the image always will be present. This supposition permits that the entropy
formula can be written in the following way:
‫ܧ‬ = − ෍ ‫ݍ‬௜݈‫݃݋‬ଶ‫ݍ‬௜
௜∈ூ
, (4)
where ‫ܫ‬ = ሼ݅ ∈ [0, 2஻
− 1] |݇௜ ≠ 0ሽ, (see Section 2.4). Without loss of generality and to obtain a
better manipulation of the previous formula, the same can be expressed as,
‫ܧ‬ = − ෍ ‫ݍ‬௜݈‫݃݋‬ଶ‫ݍ‬௜
௡
௜ୀ଴
, (5)
Starting from now, the expression 5 will be widely used and we shall call entropy of order ݊,
which is interpreted as that entropy that is composed of the sum of n terms and the associated
image has ݊ gray levels, which are visually present to the observer.
2.7. Maximum entropy of an image of order ࢔
It is known from information theory that entropy is seen as a measure of information [12], this
reaches its maximum value when all symbols are equally probable (equal probability of
occurrence). The classical formula for the entropy for a digital image in gray levels is given by
expression 1 and it is defined in (2஻
− 1)-dimensional real space. However, in most of the cases
it is possible to reduce the quantity of variables of the space by using expression 5, especially in
images where a good level of segmentation is attained. In this section, we present a theorem about
the maximum value reached by entropy when expression 5 is applied.
Theorem 1 (Maximum entropy of an image of order ݊)
Let ‫ܩ‬ be an arbitrary digital image of order ݊, then has maximal entropy of order n, if ‫݌‬௜ =
ଵ
௡
,
∀݅ ∈ ‫,ܫ‬ where ‫ܫ‬ = ሼ݅ ∈ [0, 2஻
− 1]|݇௜ ≠ 0ሽ.
The proof of this theorem appears on appendix.
Lemma 1 (Maximum entropy of an image of order ݊)
Let ‫ܩ‬ be an arbitrary digital image in gray levels of order ݊, then if ‫ܩ‬ has maximum entropy of
order n, the entropy value ‫ܧ‬ is given by ݈‫݃݋‬ଶ݊.
Proof:
If ‫ܩ‬ is a digital image in gray levels of order ݊ and this has maximum entropy order ݊, then the
probability of occurrence of gray levels are ‫݌‬௜ =
ଵ
௡
, ∀݅ ∈ ‫.ܫ‬ Calculating the value of the entropy,
one has that:

7
‫ܧ‬ଵ = − ෍ ‫݌‬௜݈‫݃݋‬ଶ‫݌‬௜
ଶಳିଵ
௜ୀ଴
= − ෍
1
݊
݈‫݃݋‬ଶ
1
݊
ଶಳିଵ
௜ୀ଴
= −݊ ൬
1
݊
݈‫݃݋‬ଶ
1
݊
൰ = −݈‫݃݋‬ଶ
1
݊
= ݈‫݃݋‬ଶ݊. ∎
The result obtaining from the previous lemma is not new. In what we are interested, it is in the
expression of the formula and their later use in the following sections.
Given the previous lemma it follows that any gray level image of order ݊, its entropy cannot be
greater than ݈‫݃݋‬ଶ݊. Images that have the property of ‫݌‬௜ =
ଵ
௡
, ∀݅ ∈ ‫ܫ‬ will be called “ideal images
of order n” and its entropy by “maximum entropy order ݊”. The last term will be denoted by ‫.݉ܧ‬
In Figure 2(a) is shown a digital image of order 8, (݊ = 8) and 8‫8ݔ‬ dimension, in which the
equivalence classes can be appreciated; namely, ‫ܥ‬଴, ‫ܥ‬ଷଶ, ‫ܥ‬଺ସ, ‫ܥ‬ଽ଺, ‫ܥ‬ଵଶ଼, ‫ܥ‬ଵ଺଴, ‫ܥ‬ଵଽଶ, ‫ܥ‬ଶଶସ and their
corresponding frequencies are ݈଴ = 7, ݈ଷଶ = 4, ݈଺ସ = 8, ݈ଽ଺ = 9 , ݈ଵଶ଼ = 13, ݈ଵ଺଴ = 12, ݈ଵଽଶ = 3,
݈ଶଶସ = 8. In Figure 2(b), 2(c), 2(d) different regions are shown whose entropies are maxima of
order 8. These regions have the gray levels and relative frequencies in common. Observe that
Figure 2(b) and 2(c) have the same dimensions as the image shown in Figure 2(a), but differ in
the spatial distribution of gray levels, while in Figure 2(d) the region has a smaller dimension.
However, the entropy in Figure 2(b) and 2(c) is the same. In the same way that in Figure 2(d) it is
possible to find a region with a much bigger dimension than in Figure 2(a), for example of
dimension 64 ‫,46 ݔ‬ and to have a maximum entropy of order 8. This fact shows that for any
image in gray levels with entropy of order n it is always possible to obtain an image with
maximum entropy of order n and that it presents the same gray levels.
(a) (b)
(a) (b)

8
Figure. 2. Digital images of order 8. 2(a) ‫ܧ‬ = 2.8740. 2(b) Digital image of order 8, size 8x8, maximum
entropy of order 8, ‫݉ܧ‬ = 3. 2(c) Image digital order 8, size 8x8, maximum entropy of order 8, ‫݉ܧ‬ = 3.
2(d) digital image of order 8, size 4x2, maximum entropy of order 8, ‫݉ܧ‬ = 3.
2.8. Applications
The previously obtained result becomes important when an image is compared with another after
running the iterativealgorithm of the mean shift. This will be seen in section 3.
2.8.1 Entropy and its relation with the classes of equivalence
One can observe in expression (5) that in the entropy of a digital image in gray levels, each term
of the same is associated to one and only a gray level. If this way, a one-one relation among the
terms of entropy is established; that is, gray levels and classes of equivalence. With the aim of
revealing other properties in the entropy of image, in the segmentation process this link will be
used.
2.8.2 Entropy as hypersurface
As was seen in the proof of the Theorem 1, when an image is of order ݊, its entropy depends on (݊ − 1)-
variables related by the formula ∑ ‫݌‬௜
௡ିଵ
௜ୀଵ + ൫1 − ∑ ‫݌‬௜
௡ିଵ
௜ୀଵ ൯ = 1 , with 0 < ‫݌‬௜ < 1 . The domain of
definition of the variables ‫݌‬௜, with ݅ = 1, ݊ − 1തതതതതതതതതത, is 0 < ∑ ‫݌‬݅
݊−1
݅=1 < 1, which is convex and open set of
ℝ௡ିଵ
, and its topological properties could be employed for obtaining information about the
evolution of the entropy during the segmentation process. Of this way, entropy can be seen as a
hypersurface in (݊ − 1)-dimensional space and it can be applied tools of the theory of differential
geometry.
Note that if ݊ > 3 it is not possible to obtain the graph of the entropy function for all pi due to
the inability of representing in ܴଷ
sets with dimensions greater than 3. However, in spite of this, it
is possible to plot the entropy function upon the domain by pairs of pi which are of interest to
analyze. For example, those ‫݌‬௜, that have not been annulled during all the segmentation process.
Figure. 3. Graphs of entropy as a function of two variables, ݊ = 3 (rotated for better visualization)
In space R3 the entropy is seen as a surface (see Figure 3) and its graphical view gives an overview of
treatment for the extension to larger spaces. In Figure 3, the graphs of the entropy function for the case that

9
presents three gray levels are shown, the domain of definition is 0 < ‫݌‬ଵ + ‫݌‬ଶ < 1, which is represented by
the shaded region by contour lines in the ܻܺ plane. One can observe that this set is open and convex. It is
also shown the maximum point in the coordinates ൫1
3ൗ , 1
3ൗ ൯ and the maximum value of entropy function
is 1.5850.
2.9 Relative entropy in images
As it was point out in Definition 2 the Relative Entropy express by:
‫݌(ܦ‬ ∥ ‫)ݍ‬ = ෍ ‫݌‬௜
௫∈॒
݈‫݃݋‬
‫݌‬௜
‫ݍ‬௜
= − ෍ ‫݌‬௜
௫∈॒
݈‫ݍ݃݋‬௜ − ൭− ෍ ‫݌‬௜
௫∈॒
݈‫݌݃݋‬௜൱ =
− ෍ ‫݌‬௜
௫∈॒
݈‫ݍ݃݋‬௜ − ‫݌(ܧ‬௜), (6)
Supose that probability ‫ݑ‬ is the uniform probability of occurrence of colors ݅ ∈ ‫,ܫ‬ i.e., ‫ݍ‬௜ =
1
݊
∀݅ =
1, ݊തതതതത, then:
‫)ݑ(ܧ‬ = − ෍ ‫݌‬௜
௫∈॒
݈‫ݍ݃݋‬௜ = − ෍ ‫݌‬௜
௫∈॒
݈‫݃݋‬
1
݊
= − ൬݈‫݃݋‬
1
݊
൰ ෍ ‫݌‬௜
௫∈॒
= −݈‫݃݋‬
1
݊
= ݈‫,݊݃݋‬
which it is an entropy of ݊ order. Combining Definition 2 and the concept of maximum entropy
of ݊ we obtain the following concept:
Definition 7 (Relative entropy in gray level images)
Let ‫ܩ‬ a gray level image with ݊ gray levels, then the relative entropy in image ‫ܩ‬ is defined by
ܴ‫)ܩ(ܧ‬ = ݈‫݊݃݋‬ − ‫ܧ‬൫‫݌‬݅
൯, (7)
where ‫݌‬௜ is the probability of occurrence of color ݅.
Note that Definition 7 is the difference between the maximum entropy of ݊ order and the entropy
of the given image ‫,ܩ‬ for that reason it will denoted by ‫݉ܧ‬ − ‫.ܧ‬
Expression 7 will let us characterize the ‫݅ܵܪܯ‬ algorithm as it is treated in the following section
related to the experimental results.
3. EXPERIMENTS AND ANALYSIS OF RESULTS
With theoretical results it is possible to carry out a study of entropy while the ‫ܵܯ‬ℎ݅ algorithm is
running. The used images in this study are represented in Figure 4.

10
(a) Barbara (b) Cameraman (c) Bird (d) Baboon (e) Montage
Figure. 4. Images used in the experimentation.
The images and graphics that are shown, have been obtained using the system MATLAB. The
images have 256x256 pixels, the used stopping threshold is 0.000001. The parameters chosen
were hs = 3 and hr = 5. For more details on our algorithm, one can refer to [11], [10]. In Figure
5(a) first result is presented.
When comparing the images shown in Figure 5, one can observe that some details were lost (see
arrows). In Figure 5(b) one can see some more homogeneous areas, in which a certain
segmentation level has been achieved.
(a) Original image (b) Segmented image
Figure. 5. Homogeneous regions
The graph in Figure 6 shows that the entropy gradually decreases (black points), while the
maximum entropy of order ݊ has 6 disjoint intervals remaining constant for the iterations
1 − 2, 6 − 7, 9 − 10, 13 − 15, 23 − 26, 29 − 95 (squares). Throughout this section, the sets of
points upon the graphs formed by these intervals will be called groups of iterations, and will be
identified as the first group of iterations, the second group of iterations, last group of iterations.
Figure 6. Graphs of entropy. Em vs. Number of iterations (Barbara)

11
The variation of the order of the image is associated with the appearance or disappearance of gray
levels. However, due to the homogenization that produces the iterative algorithm, certain gray
levels are replaced by others that were not present, in this way it is achieved, that the order of the
image remains constant. The search and analysis of the pi that are annulled in the mentioned
iterations led us to the following conclusion: appearance and disappearance of the gray levels
only in the fourth and fifth groups of iterations, with iterations 13 − 15 and 23 − 26 respectively.
These results appear in Figure 7.
Figure 7 shows the values of ‫݌‬௜ for iterations from 13 to 15. At iterations 13 and 14 the gray level
216 is absent (light gray bar, discontinuous edges) and this appears at iteration 15, while the gray
level 217 decreases until it disappears (dark gray bar, continuous edges). This justifies why gray
levels remains constant. Another way of seeing this issue is analyzing the quantity of “null ‫݌‬௜” per
iteration, which is always the same; i.e., by iteration only one of the ‫݌‬௜ is annulled. In a simple
graphical way, it can be seen that the number of columns per iteration (corresponding to no “null
‫݌‬௜”) is always the same. Similar analysis is carried out with the group of fifth iterations.
(a) 13 − 14 iterations (b) 23 − 26 iterations
Figure. 7. Graph of pi vs. iterations.
In Figure 7(b) we observed that most of the ‫݌‬௜ takes the value 1.526 ∗ 10ିହ
, which is not a casual
number. As we pointed out in the beginning of this section all images are of dimension 256x256,
this makes that the total number of pixels is 65536. If one calculates the minimum probability
(no null) of occurrence of a gray level in the image gives a value of 0. 0000152587890625,
which one can express as 1.526 ∗ 10ିହ
. This indicates that when seeing this value in the graphics,
the gray level that corresponds can be only expressed in a pixel of the 65536. In other words, one
can consider this gray level as noise, and if its value did not vary in the last group of iterations can
be eliminated, which will contribute to improve the homogenization in the segmented image. On
the other hand, this gray level will not affect the value of the entropy since its contribution is
minimum. Similarly, one can check that the value 0.3052 ∗ 10ିହ
corresponds to the probability
that its gray level appears only in two pixels in the whole image.
With regard to the last group of iterations it is important to point out that it is quite high (69 %
bigger than the rest), which shows that the number of gray levels tends to be stabilized when
increasing the iterations.
The appearance or disappearance of gray levels in the image determines the variation of the value
of the maximum entropy of order n, since this depends only on the quantity of gray levels present

12
in the image. Taking the difference between the maximum entropy of order n with the entropy of
the image, difference which will be denoted by ‫݉ܧ‬ − ‫,ܧ‬ an idea of entropy away from its upper
bound can be given (see Figure 8).
The graph in Figure 8, shows that starting from the iteration 29 the entropy of the image tends to
move away from the maximum entropy of order n, without varying the order. The abrupt change
in the graph shows clearly that it varied the quantity of gray levels. Iterations which not belong to
groups of iterations are marked in black color triangles. Observe that several of these iterations
have the same order; for example, the iterations 12, 16, 18, 20 and 22. The difference ‫݉ܧ‬ − ‫ܧ‬
corresponding to the iteration 1 it is bigger than the final one (iteration 95).
Figure. 8. Graph of ‫݉ܧ‬ − ‫ܧ‬ vs. number of iterations (Barbara)
Figure 9 was obtained from Figure 8 by removing black points. The connected symbols by dotted
lines belong to consecutive iterations (groups of iterations), and those with the same shape are
associated to iterations of the same order, a “type of separation by levels”, it is being well-defined.
Figure 9. Graph ‫݉ܧ‬ − ‫ܧ‬ vs number of iterations.

13
It is highlighted with different geometrical figures and the iterations in which the image has the
same order (Barbara).
The points in the first group of iterations have been differentiated with squares from the points of
the last groups of iterations with points. One can appreciate that as the number of iterations
increase the number of groups of iterations with same gray level diminishes until one. In
consecutive the iterations that have the same quantity of gray levels, the values of the difference
‫݉ܧ‬ − ‫ܧ‬ tend to increase, and the slope of the curve tends to diminish its inclination. On the other
hand, the iterations of the first group have higher values in the difference ‫݉ܧ‬ − ‫ܧ‬ than the last
group.
Figure 10 shows another example using the image of Cameraman. In Figure 10(b) the arrows
indicate two regions that have been homogenized.
(a) Original image (b) Segmented image
Figure. 10. Homogenous regions.
Figure 11 shows the plot of ‫݉ܧ‬ (squares) and entropy of the image (points) versus the number of
iterations. Observe that after iteration 10, the number of gray levels did not change, it is constant.
The consecutive iterations that did not change were 1-2, 6-7, 8-13, 14-78, of which the second
group of iterations presented the appearance and disappearance of gray levels. The last group of
iterations (14-78) was 82 % higher than the rest of the iterations in the segmentation process.

14
Figure. 11. Graphics Entropy and Em vs. number of iterations (Cameraman)
Figure 12 shows the values of pi for the iterations 6 and 7. One can note that there is only one
column per iteration, and these have the same height but different colors. One can interpret that
the 244 gray level in the iteration 6 was replaced completely by the 245 gray level, maintaining
the order of the image. The value 0.4578 ∗ 10ିହ
corresponds to the probability that its respective
gray level appears only in three pixels of the entire image.
Figure. 12. Graph of vs pi 6-7 iteration (Cameraman).
In Figure 13 it is possible to observe that the iterations previous to iteration 14 do not go through
as many changes of gray levels in comparison to the image of Barbara. This iteration has been
marked with number 14, which shows the instant in that the last group of iterations begins. The
difference ‫݉ܧ‬ − ‫ܧ‬ for the iteration 1 is smaller than the final iteration (iteration 78), contrary to
the Barbara image.

15
Figure. 13. Graphics ‫݉ܧ‬ − ‫ܧ‬ vs number of iterations (Cameran).
Figure 14 was obtained from Figure 13 by removing only 3 points. In this way, four groups of
iterations are obtained. Abrupt changes are not observed in the groups in which in which the order
of the image remains constant as was in the Barbara image. It can be observed that the value of
‫݉ܧ‬ − ‫ܧ‬ tends to increase inside the groups when the quantity of iterations increases. The last
group of iterations (black points) presents a bigger number of points in comparison with the rest.
Figure. 14. Graph of the difference in ‫݉ܧ‬ − ‫ܧ‬ vs number of iterations.
The iterations in which the image has the same order (Cameran) are highlighted with different
geometrical figures.

16
Figure 15 shows the results of the image of Bird. In Figure 15(b) the arrows indicate two regions
that have been homogenized.
(a) original image (b) segmented image
Figure. 15. (a) original image. (b) segmented image.
The graph in Figure 16 shows the entropy and maximum entropy of order ݊ versus the number of
iterations of the segmented Bird image. It can be appreciated that the iterations whose resulting
images present the same order are separated into disjoint intervals, which elevate their longitude
when increasing the quantity of iterations. Inside the groups of iterations there is no appearance or
disappearance of gray levels. The last group of iterations is 63 % bigger than the rest of the
iterations in the segmentation process.
Figure 16. Graphics of Entropy and Em vs number of iterations (Bird)
In Figure 17 the graph of the value of the difference ‫݉ܧ‬ − ‫ܧ‬ versus the quantity of iterations is
shown, where a triangle marks the iteration that does not belong to the groups of iterations. The
behavior of this image does not suffer very marked oscillations as in the case of Barbara image.

17
Starting from the iteration 29 the quantity of gray levels is stabilized. The value of ‫݉ܧ‬ − ‫ܧ‬ initial
is bigger than the final.
Figure 17. Graph ‫݉ܧ‬ − ‫ܧ‬ vs. number of iterations (Bird)
The graph of Figure 18 presents 6 groups of consecutive iterations that have the same order. The
slopes of the groups of iterations are highlighted with different geometrical figures and one can
see these tend to decrease when the number of iterations increases. This behavior is common for
the previously analyzed images. However, in this example more clarity is evident.
Figure. 18. Graph ‫݉ܧ‬ − ‫ܧ‬ vs. number of iterations.
The iterations in which the image has the same order (Bird) are highlighted with different
geometrical figures.
The range in that the values of the difference ‫݉ܧ‬ − ‫ܧ‬ oscillate in the last group of iterations is
very specific to each image. It is shown clearly in Figure 19. However, these present the common
characteristic such as a slow growth of the value of the difference of ‫݉ܧ‬ − ‫,ܧ‬ where this slow

18
growth is more accentuated with the increase of the number of iterations. The graph includes the
results of the images of Baboon and Montage.
Figure. 19. Graph ‫݉ܧ‬ − ‫ܧ‬ vs number of iterations for the last group of iterations in different images.
4. CONCLUSIONS
In the graphs corresponding to the Entropy vs. the number of iterations can be appreciated that, as
the segmentation was reached, the entropy diminishes from one iteration to another. The order of
the image decreases as the number of iterations increases, this fact can be interpreted that when
the image is moving away from its ideal image of order ݊, it becomes more homogeneous.
However, it was possible to verify that, starting from an instant, the order of the image remains
constant until the final segmentation is reached, according to the selected stopping threshold. The
differences of the values of ‫݉ܧ‬ − ‫ܧ‬ between the maximum entropy of order n and the entropy of
the image are significant and these differences increase in the groups of iterations that present the
same quantity of gray levels. In the graphs of bars (see Figures 12 and 13), one can check that
there are gray levels that disappeared from the image as the algorithm runs, contributing to more
homogenization of the image. It is possible to see that in the last stages of segmentation most
images present gray levels with “no null” probability of occurrence. This issue will be the subject
of future research as well as other aspects that arose in this study.
APPENDIX
MAXIMUM VALUE REACH BY ENTROPY
Definition 7: (Negative Definite Function)
Let ‫ܣ‬ ∈ ‫ܯ‬௡(ℝ) be, such that ‫ܣ‬ = ‫ܣ‬௧
, it is said to be negative definite it satisfies that ∀‫ݕ‬ ≠ 0,
‫ݕ‬ ∈ ℝ௡
⟹ ‫ݕ‬௧
‫ݕܣ‬ < 0, where ‫ܯ‬௡(ℝ), is the set of square matrices of ݊ order with coefficients in
ℝ [6].
Definition 8: (Maximum Strict)

19
Suppose that the function ݂(‫)ݔ‬ is defined over the set ܺ ⊂ ℝ௡
. The point ‫ݔ‬(଴)
∈ ܺ is called strict
maximum point if a neighborhood ܸ (‫ݔ‬(଴)
) of the point ‫ݔ‬(଴)
that is, ∀‫ݔ‬ ∈ V൫‫ݔ‬(଴)
൯⋂ܺ, ‫ݔ‬ ≠ ‫ݔ‬(଴)
,
such that the inequality ݂ (‫)ݔ‬ < ݂൫‫ݔ‬(଴)
൯ is checked [6].
Theorem 3 (Condition of Maximum Entropy of Order ࢔):
Let ‫ܩ‬ be an arbitrary digital image of order n, then has maximal entropy of order ݊, if ‫݌‬௜ =
ଵ
௡
,
where ‫ܫ‬ = ሼ݅ ∈ [0, 2஻
− 1] |݇௜ ≠ 0ሽ.
Proof:
By hypothesis, since ∑ ‫݌‬௜
௡
௜ୀଵ = 1, we construct the vector ‫݌‬ = (‫݌‬ଵ, ⋯ , ‫݌‬௡). Without loss of
generality, we will assume that ‫݌‬௡ = 1 − ∑ ‫݌‬௜
௡
௜ୀଵ , substituting the expression (3) in the entropy
formula, obtain
‫ܧ‬ = − ∑ ‫݌‬௜݈‫݃݋‬ଶ‫݌‬௜
௡ିଵ
௜ୀଵ − ൫1 − ∑ ‫݌‬݅
݊
݅=1 ൯݈‫݃݋‬ଶ൫1 − ∑ ‫݌‬݅
݊
݅=1 ൯ (‫ܧ‬ depends on ݊ − 1 variables).
The partial derivatives of first order are,
߲‫ܧ‬
߲‫݌‬௜
= − ൬݈‫݃݋‬ଶ‫݌‬௜ +
1
ln 2
൰ + ൭݈‫݃݋‬ଶ ൭1 − ෍ ‫݌‬௜
௡ିଵ
௜ୀଵ
൱ +
1
ln 2
൱.
Making
డா
డ௣೔
= 0 with ݅ = 1, ݊തതതതത, one arrives to the expressions
݈‫݃݋‬ଶ‫݌‬௜ = ݈‫݃݋‬ଶ(1 − ∑ ‫݌‬௜
௡ିଵ
௜ୀଵ ) and ݈‫݃݋‬ଶ‫݌‬௜ = ݈‫݃݋‬ଶ‫݌‬௡, ∀݅ ≠ ݊, ݅ = 1, ݊തതതതത.
Because the logarithm function is strictly monotonous, this implies that ‫݌‬௜ = ‫݌‬௡∀݅ ≠ ݊ .
Therefore, we have arrived to that a candidate to strict extreme is obtained when ‫݌‬௜ = ‫݌‬௡∀݅ ≠ ݊,
i.e., when all the probabilities of occurrence of gray levels are equals, so
∑ ‫݌‬݊
௡
௜ୀଵ = 1 ⟹ ݊‫݌‬݊
= 1 ⟹ ‫݌‬݊
=
1
݊
.
Since all ‫݌‬௜ have the same value, then ‫݌‬௜ =
ଵ
௡
∀݅ i∀ , the obtained point ‫݌‬ = ቀ
ଵ
௡
, ⋯ ,
ଵ
௡
ቁ ∈ ℝ௡
.
We shall now determine the type of local extreme (maximum or minimum).
Analyzing the second partial derivatives of the matrix of the entropy function ‫,ܧ‬ one has that
൬
డమா
డ௣೔డ௣ೕ
൰ ∈ ‫ܯ‬௡ିଵ(ℝ), i.e., the entropy being a function of (݊ − 1) variables, then the second
differential of the entropy function will depend only on (݊ − 1) variables and it will be a square
matrix of size (݊ − 1)x(݊ − 1). The elements of this matrix are:
డమா
డ௣೔డ௣ೕ
= ൞
−
ଵ
୪୬ ଶ
൬
ଵ
௣ೕ
+
ଵ
൫1−∑ ‫݌‬݅
݊−1
݅=1 ൯
൰ ∀݅ = ݆
−
ଵ
୪୬ ଶ
൬
ଵ
൫1−∑ ‫݌‬݅
݊−1
݅=1 ൯
൰ ∀݅ ≠ ݆
.
Therefore, the matrix of second partial derivatives has the form

20














+
+
+
=
babb
bbab
bbba
A
mL
MOLM
L
L
2
1
,
where ܽ௜ =
ଵ
௣೔
i∀ and ܾ =
1
൫ଵି∑ ௣೔
೙షభ
೔సభ ൯
.
Now, we shall prove that the matrix ൬
డమா
డ௣೔డ௣ೕ
൰ is negative definite by Definition 7. Let ‫ܣ‬ =
൬
డమா
డ௣೔డ௣ೕ
൰ and ‫ݕ‬ ∈ ℝ௡ be, ‫ݕ‬௧
ቀ−
ଵ
୪୬ ଶ
‫ܣ‬ቁ ‫ݕ‬ = −
ଵ
୪୬ ଶ
(‫ݕ‬௧‫.)ݕܣ‬ Working with the expression ‫ݕ‬௧‫,ݕܣ‬ one
has that:
( ) ( )
2
11
2
1
2
1
1 







+=
























+
+
+
= ∑∑ ==
m
j
j
m
j
jj
m
m
m
t
ybay
y
y
babb
bbab
bbba
yyAyy M
L
MOLM
L
L
L .
Since 0, >bai , ni ,1= , then 0>Ayyt m
y ℜ∈∀ , ni ,1= and therefore
‫ݕ‬௧
൬−
1
ln 2
‫ܣ‬൰ ‫ݕ‬ = −
1
ln 2
(‫ݕ‬௧
‫)ݕܣ‬ < 0
This last expression says that matrix ‫ܣ‬ is negative definite and of this way the entropy of order ݊
has at least a strict maximum ∀݅; that is, ‫݌‬௜ =
ଵ
௡
∀݅. Now, we shall prove that this point is the
global maximum. Suppose that a value of entropy of order ݊ that is, which is bigger or similar to
the maximum than was previously found, but since the point ‫݌‬ = ቀ
ଵ
௡
, ⋯ ,
ଵ
௡
ቁ , it is a strict
maximum in a small neighborhood ܸ , the entropy values decreases from this point over
neighborhood ܸ ; therefore, there exists a value greater or equal to the entropy in another point,
then a saddle point or an absolute minimum, at least, must exist. However, this is not possible
because the previous analysis gives a negative definite matrix ൬
డమா
డ௣೔డ௣ೕ
൰(‫݌‬) for all ‫݌‬ =
(‫݌‬ଵ, ⋯ , ‫݌‬௡ିଵ) that belongs to the domain of the entropy function. Therefore, the non-existence
of saddle or strict minimum point implies non-existence of other strict maximums, which leads to
only a global maximum.
REFERENCES
[1] Cheng~Y., (1995) “Mean Shift, Mode Seeking, and Clustering”, IEEE Trans, Pattern Analysis and
Machine Intelligence Neurocomputing, Vol. 17, No. 8, pp790–799.
[2] Comaniciu, D. I., (2000) “Nonparametric Robust Method for Computer Vision”, Ph.D. Thesis, New
Brunswick, Rutgers, The State University of New Jersey.
[3] Comaniciu, D. & Meer, P., (2002) “Mean Shift: A Robust Approach toward Feature Space Analysis”,
IEEE Transaction on Pattern Analysis and Machine Intelligence, Vol. 24, No. 5.
[4] Domínguez, D. & Rodríguez, R.: Use of the L (infinity) norm for image segmentation through Mean
Shift filtering, International Journal of Imaging, Vol. 2, No. S09, pp81–93, 2009.
[5] Fukunaga, K. & Hosteler, D. (1975) “The Estimation of the Gradient of a Density Function”, IEEE
Trans., Information Theory, No. 21, pp32–40.
[6] Kudriavset, L. D., (1984) “Curso de Anlisis Matemtico”, Mir Editorial, Vol II.

21
[7] Rodriguez R. & Suarez, A. G., (2006) “An Image Segmentation Algorithm Using Iteratively the
Mean Shift”, Image Analysis and Applications, Book Progress in Pattern Recognition, Book Series
Lecture Notes in Computer Science Publisher Springer Berlin/Heidelberg, Vol. 4225/2006 pp326-
335.
[8] Noriega, T. D. & Piero, L. R., (2007) “Algebra”, Flix Varela Editorial, Vol II.
[9] Rodriguez R., (2008) “Binarization of medical images based on the recursive application of mean
shift filtering: Another algorithm”, Journal of Advanced and Applications in Bioinformatics and
Chemistry, , Dove Medical Press Ltd, Vol. I, No. 1:12.
[10] Rodriguez, R., Suarez, A. G., & Sossa J. H., (2011) “A Segmentation Algorithm based on an Iterative
Computation of Mean Shift Filtering”, Journal Intelligent and Robotics System, Vol 63, No. 3-4,
pp447–463.
[11] Rodriguez, R., Torres E. & Sossa J. H., (2012) “Image Segmentation via an Iterative Algorithm of the
Mean Shift Filtering for Different Values of the Stopping Threshold”, International Journal of
Imaging and Robotics, Vol. 7, No. 6, pp1–19..
[12] Shannon C., A, (1948) “Mathematical Theory of Communication”, Bell System Technology Journal,
No. 27, pp370–423.
[13] Shen, C.& Brooks, M. J., (2007) “Fast Global Kernel Density Mode Seeking: Applications to
Localization and Tracking”, IEEE Transactions on Image Processing, Vol. 16, No.5, pp1457-1469.
[14] Thomas, M. C., Thomas, J. A. Elements of Information Theory, John Wiley & Sons, Inc., 1991.

Behavior study of entropy in a digital image through an iterative algorithm

More Related Content

What's hot (20)

Similar to Behavior study of entropy in a digital image through an iterative algorithm (20)

Recently uploaded (20)

Behavior study of entropy in a digital image through an iterative algorithm