Simultaneous Smoothing and Sharpening of Color Images
- 1. Simultaneous sharpening and smoothing of color images C. Pérez-Benito, S. Morillas, C. Jordán, J.A. Conejero Mathematical Modelling in Engineering & Human Behaviour Universitat Politècnica de València. July 2017
- 2. Cristina Pérez Benito Introduction Colour image smoothing and sharpening are two important pre-processing techniques within the Computer Vision field. Smoothing consist on the removal of possible image perturbations resulted from the image acquisition. On the other hand, sharpening is in charge of the improvement of the image visual appearance and the enhancement the details and borders of the image. There exists a lot of smoothing and sharpening methods that are able to improve visual quality of images. However the same does not happen with the simultaneous approaches due to the opposite nature of these two operations. We present a new model based on graph theory that allows us to improve the details of the image at the same time that the noise is removed.
- 3. Cristina Pérez Benito Contents • Sharpening vs smoothing • Graph based model for color image processing • Application of the model to simultaneous sharpening and smoothing • Conclusion
- 4. Cristina Pérez Benito Contents • Sharpening vs smoothing • Graph based model for color image processing • Application of the model to simultaneous sharpening and smoothing • Conclusions
- 5. Cristina Pérez Benito • Colour image smoothing is the set of pre-processing techniques intended for jk removing possible image perturbations without losing image information. ‐ White Gaussian noise is the most common factor which can significantly affect visual quality of images: each pixel of the image will be changed from its original value by some small amount that follows a Gaussian distribution. • Colour image sharpening is the set of techniques whose purpose is the improvement of the image visual appearance and highlight or recover certain details of the image for conducting an suitable analysis by a human or a machine. Sharpening vs smoothing
- 6. Cristina Pérez Benito • Typical spatial filters for colour image smoothing are based on the convolution of the image with different kernels, depending on the intended result. • These kernels could be of any size n x n, but usually 3 x 3 ‒ According to previous works, using n > 3 results in higher noise smoothing capability but much more blurred images, along with the large increase in computational cost that make increasing the window not a good choice in general. Sharpening vs smoothing
- 7. Cristina Pérez Benito Sharpening vs smoothing 𝟏 𝟗 𝟏 𝟗 𝟏 𝟗 𝟏 𝟗 𝟏 𝟗 𝟏 𝟗 𝟏 𝟗 𝟏 𝟗 𝟏 𝟗 • Typical spatial filters for colour image smoothing are based on the convolution of the image with different kernels, depending on the intended result. • These kernels could be of any size n x n, but usually 3 x 3
- 8. Cristina Pérez Benito Sharpening vs smoothing 𝟎 𝟏 𝟎 𝟏 −𝟒 𝟏 𝟎 𝟏 𝟎 • In the same way that in the smoothing case, typical spatial techniques for sharpening images are based on kernels. • The main difference is the sign of the possible values. In the sharpening case, it will appear negative terms. • Most common methods are based on derivatives, such as, Laplacian filter.
- 9. Cristina Pérez Benito Sharpening vs smoothing • These two operations have an opposite nature, both details or edges of the image and the noise present in the image correspond to high frequencies (for this reason, differentiate them is very difficult) ‒ Smoothing intends to remove the high frequencies. ‒ Sharpening intends to increase the high frequencies. • This is the main problem of the approaches for smoothing and sharpening in a simultaneous way.
- 10. Cristina Pérez Benito Sharpening vs smoothing • The initial approach is to consider a two-steps process: first smoothing and then sharpening, or the other way around. • However, this approaches usually leads too many problems. ‒ If we first apply a smoothing technique, then we will be loosing information that cannot be recovered in the succeeding sharpening step. ‒ If we first apply a sharpening method over a noisy image, we will amplify the noise presented in it.
- 11. Cristina Pérez Benito Sharpening vs smoothing • The initial approach is to consider a two-steps process: first smoothing and then sharpening, or the other way around. • However, this approaches usually leads too many problems. • There are any author that proposes approaches for simultaneous sharpening and smoothing. ‒ One of the best approaches is BM3D-SH3D, proposed by Dabov et. al. ‒ However this method are based on non-local means Increased computational cost
- 12. Cristina Pérez Benito Contents • Sharpening vs smoothing • Graph based model for color image processing • Application of the model to simultaneous sharpening and smoothing • Conclusions
- 13. Cristina Pérez Benito Graph based model for colour image processing Each pixel is processed by using a 3x3 sliding window of neighbors with center on it. F1 F2 F3 F4 F5 F6 F7 F8 F9 Each pixel is represented by its RGB coordinates: 𝐹𝑖 = (𝑅𝑖 𝑅 , 𝑅𝑖 𝐺 , 𝑅𝑖 𝐵 )
- 14. Cristina Pérez Benito }8,,0|{ iFV i Graph based model for colour image processing • 𝐺(𝐸, 𝑉) complete weighted graph associated to 𝐹5
- 15. Cristina Pérez Benito }8,,0|{ iFV i }|),({ , jiFFeE jiji 2, ||||)( jiji FFew • 𝐺(𝐸, 𝑉) complete weighted graph associated to 𝐹5 Graph based model for colour image processing
- 16. Cristina Pérez Benito • 𝐺(𝐸, 𝑉) complete weighted graph associated to 𝐹5 }8,,0|{ iFV i }|),({ , jiFFeE jiji 2, ||||)( jiji FFew An edge 𝑒𝑖,𝑗 exists between two pixels, if 𝑤(𝑒𝑖,𝑗) is lower than a certain threshold 𝑈 . Graph based model for colour image processing
- 17. Cristina Pérez Benito • The value of the threshold 𝑈 is one of the keys of the model. • It ensures us that two pixels will be joined if they are similar. Flat region Graph based model for colour image processing
- 18. Cristina Pérez Benito Detail region • The value of the threshold 𝑈 is one of the keys of the model. • It ensures us that two pixels will be joined if they are similar. Graph based model for colour image processing
- 19. Cristina Pérez Benito • The feature that better characterizes whether a pixel belongs to a flat region or a detail region is the cardinal of the nodes set of the connected component that contain the central pixel, that we denote by 𝑐𝑎𝑟𝑑 𝐶𝐶 𝐹5 . • We define a border detection based on this cardinal to segment noise free images ‒ A grayscale image was created from this cardinal: the bigger cardinal, the greater intensity of greys. Adjustment of the threshold
- 20. Cristina Pérez Benito Adjustment of the threshold • We obtain the optimal threshold by maximizing the mutual information between the grayscale image obtained using 𝑐𝑎𝑟𝑑 𝐶𝐶 𝐹5 and the output of a fuzzy edge detector.
- 21. Cristina Pérez Benito • We are interested in a robust method in noisy environments, therefore, we add white Gaussian noise with different standard deviations to all the set of images. • In analogous way that before, the optimal threshold for the noisy images was calculated maximizing the mutual information. Adjustment of the threshold
- 22. Cristina Pérez Benito • The standard deviation of the Gaussian noise of an image is unknown in general. A noise variance estimation was calculated for a set of images. • Using linear regression analysis, we obtain a relation between this estimation and the optimal threshold, that enables us to fix the threshold without prior information of the image. Adjustment of the threshold
- 23. Cristina Pérez Benito Contents • Sharpening vs smoothing • Graph based model for color image processing • Application of the model to simultaneous sharpening and smoothing • Conclusions
- 24. Cristina Pérez Benito Application of the model to simultaneous Smoothing and Sharpening • We have each pixel of the image modelled by a local graph. • This graph allow us to know the local structure of the image, information that permit us to apply smoothing and sharpening in a simultaneous way. • We are going to use
- 25. Cristina Pérez Benito Application of the model to simultaneous Smoothing and Sharpening • We have each pixel of the image modelled by a local graph. • This graph allow us to know the local structure of the image, information that permit us to apply smoothing and sharpening in a simultaneous way. • We are going to use ‒ The connected component of the central pixel for apply smoothing.
- 26. Cristina Pérez Benito Application of the model to simultaneous Smoothing and Sharpening • We have each pixel of the image modelled by a local graph. • This graph allow us to know the local structure of the image, information that permit us to apply smoothing and sharpening in a simultaneous way. • We are going to use ‒ The others components for apply sharpening. ‒ The connected component of the central pixel for apply smoothing.
- 27. Cristina Pérez Benito Application of the model to simultaneous Smoothing and Sharpening • We design a 3 x 3 kernel for each pixel so that 𝑤𝑖 > 0 ⟺ 𝐹𝑖 ∈ 𝐶𝐶5 𝑤𝑖 < 0 ⟺ 𝐹𝑖 ∉ 𝐶𝐶5
- 28. Cristina Pérez Benito Application of the model to simultaneous Smoothing and Sharpening Where 𝛼 is a parameter for controlling the smoothing effect. We calculate the denoised value 𝐹5 D , by 𝐶𝐶 𝐹5 as: 𝐹5 𝐷 = 𝑖∈𝐶𝐶5 𝑖 ≠5 𝑒 1 0.1+∥𝐹5−𝐹 𝑖∥2 𝐹𝑖 + 𝛼 𝐹5 𝑖∈𝐶𝐶5 𝑖 ≠5 𝑒 1 0.1+∥𝐹5−𝐹 𝑖∥2 + 𝛼 • We design a 3 x 3 kernel for each pixel so that
- 29. Cristina Pérez Benito Where 𝜆 is a parameter for controlling the sharpening effect. Application of the model to simultaneous Smoothing and Sharpening • We design a 3 x 3 kernel for each pixel so that 𝐹5 ′ = 1 + 𝜆 𝐹5 𝐷 − 𝜆 𝑖 ∉ 𝐶𝐶5 𝑖 𝑖 ∉𝐶𝐶5 𝐹𝑖 And finally, we calculate the denoised-sharpened value 𝐹5 ′ by 𝑁𝐶𝐶 𝐹5 as : 𝐹5 𝐷 = 𝑖∈𝐶𝐶5 𝑖 ≠5 𝑒 1 0.1+∥𝐹5−𝐹 𝑖∥2 𝐹𝑖 + 𝛼 𝐹5 𝑖∈𝐶𝐶5 𝑖 ≠5 𝑒 1 0.1+∥𝐹5−𝐹 𝑖∥2 + 𝛼
- 30. Cristina Pérez Benito Application of the model to simultaneous Smoothing and Sharpening • In this way, the final proposed kernel is given by: 𝑤5 = 𝛼 𝑖=1 #𝐶𝐶5 𝑒 1 ∥𝐹5−𝐹𝑖∥2 + 𝛼 1 + 𝜆 𝑤𝑖 = 𝑒 1 0.1+ ∥𝐹5−𝐹𝑖∥2 𝑖=1 #𝐶𝐶5 𝑒 1 0.1+∥𝐹5−𝐹𝑖∥2 + 𝛼 1 + 𝜆 ∀ 𝑖 ≠ 5 𝑠𝑢𝑐ℎ 𝑎𝑠 𝐹𝑖 ∈ 𝐶𝐶5 𝑤𝑖𝑡ℎ 𝛼, 𝜆 ∈ ℝ 𝑤𝑖= − 𝜆 𝑖 ∉ 𝐶𝐶5 𝑖 ∀ 𝑖 𝑠𝑢𝑐ℎ 𝑎𝑠 𝐹𝑖 ∉ 𝐶𝐶5
- 31. Cristina Pérez Benito Experimental results Noisy image Flat Region: APPLY METHOD Only Smoothing
- 32. Cristina Pérez Benito Experimental results Noisy image Border Region: APPLY METHOD Smoothing and Sharpening
- 33. Cristina Pérez Benito Experimental results Noisy image Border Region: APPLY METHOD Smoothing and Sharpening
- 34. Cristina Pérez Benito Experimental results 𝛼 = 6 𝜆 = 0 𝛼 = 6 𝜆 = 10 Only smoothingNoisy image Proposed method • Proposed method applied to a noisy image
- 35. Cristina Pérez Benito Contents • Sharpening vs smoothing • Graph based model for color image processing • Application of the model to simultaneous sharpening and smoothing • Conclusions
- 36. Cristina Pérez Benito Conclusions • We have constructed a model based on graph theory able to characterize a color image and differentiate appropriately texture regions and flat regions. • Using this model, a new spatial filter for color image smoothing and sharpening have been designed. • This filter is able to enhance the details of the image at the same time that the noise is removed.
- 37. Cristina Pérez Benito Future Work • Optimize the parameters of the model and obtain the better combination between sharpening and smoothing. • Measure the performance of the proposed method by any non-reference image quality measure. • Join the two parameters in order to have only one parameter in the model that allow us to control the smoothing-sharpening effect.
- 38. Cristina Pérez Benito Thanks for your attention !
- 39. Cristina Pérez Benito Thanks for your attention !