Soft Shadow Rendering based on Real Light Source Estimation in Augmented Reality

Zakiah Noh & Mohd Shahrizal Sunar
Advances in Multimedia - An International Journal (AMIJ), Volume (1): Issue (2) 26
Soft Shadow Rendering based on Real Light Source Estimation
in Augmented Reality
Zakiah Noh zakiahnoh@gmail.com
ViCubeLab,Department of Computer Graphics and Multimedia,
Faculty of Computer Science and Information System,
UniversitiTeknologi Malaysia,
81310 Skudai, Johor, Malaysia
Mohd Shahrizal Sunar shah@cs.utm.my
ViCubeLab, Department of Computer Graphics and Multimedia,
Faculty of Computer Science and Information System,
UniversitiTeknologi Malaysia,
81310 Skudai, Johor, Malaysia
Abstract
The most challenging task in developing Augmented Reality (AR) applications is
to make virtual objects mixed harmoniously with the real scene. To achieve
photorealistic AR environment, three key issues must be emphasized namely
consistency of geometry, illumination and time. Shadow is an essential element
to improve visual perception and realism. Without shadow, virtual objects will
appear like it is floating and thus will make the environment look unrealistic.
However, many shadow algorithms still have drawbacks such as producing sharp
and hard-edged outlines, which make the shadow’s appearance unrealistic.
Thus, this paper will focus on generating soft shadow in AR scene, rendered
base on real light sources position. The reflective sphere is used to create
environment map image that can estimate the light source from the real scene
and generate the soft shadows.
Keywords: Augmented Reality, Shadow, Soft Shadow, Reflective Sphere, Environment Map.
1. INTRODUCTION
Augmented Reality (AR) is part of mixed reality that mixes the physical world with virtual objects.
This technology allows users to interact and control the environment with their actions. Azuma [1]
defined three criteria of an AR system. Firstly, AR is a combination of real and virtual world.
Secondly, AR is interactive in real-time, and finally, AR must be registered in 3D. Currently,
developers of AR technology are working on broad areas of AR applications which are reliable to
be used in real world application such as in cultural heritage [2][3], game, simulation, medical and
education.
To achieve photorealistic rendering in AR, three problems have been identified, geometry
consistency, illumination consistency and time consistency [4]. Consistency of geometry refers to
the correct position of a virtual object in real scene location. Time consistency is the
correspondence between real world and virtual world. Hence, it is important part to make a
possible smooth interaction in real time. Meanwhile, consistency of illumination is to match the
shading of virtual object with other object in real scene, where virtual object must cast a correct

shadow. Shadow is one of the elements that can add realism to an AR environment. To improve
objects presence, it is important to provide more information of the size, position and shape of
virtual objects in the real world.
Related techniques have been done since several years ago, where elements such as light
source, object placement, luminance and the geometry of the environment were given high
consideration onto creating a realistic shadow [5][6]. Without shadows, computer generated
images will look unreasonable even with precise measurements of the light source and material
properties [7]. Thus, this paper will focus on generation of soft shadow that can be rendered
based on real light source estimation to produce a credible soft shadow in AR environment.
The rest of the paper is organized as follows: Section 2 gives brief explanations about related
works in shadow in AR. Section 3 discusses the process of shadow implementation in AR
environment and continue on with experiment in section 4. Experimental result will be discussed
in section 5 and finally, section 6 will conclude about this paper and future work.
2. RELATED WORKS
Recently, computer graphics technologies have seen rapid growth and researchers involving in
AR field have continuously trying to improve the quality of graphics system. Jacobs et al. [8]
present the classification of illumination methods to be applied in mixed reality environment. Two
categories of rendering method were stated, common illumination and relighting. Common
illumination method provides a consistent lighting when virtual objects are added into real world,
and does not allows any modification of the current scene. The method that used common
illumination can be found in [9][10][11][12][13][14][15]. Whilst, relighting method allows the
modification of the original illumination such as in [16][17].
Naemura et al. [18] proposed the concept of virtual light and virtual shadow. The concept of
virtual shadow in this method is divided into four types: i) real to virtual shadow for rigid objects, ii)
real to virtual shadow for non-rigid objects, iii) image-based virtual to virtual shadow, and iv)
virtual to real shadow. These methods will project the shadow of real object onto virtual world and
vice versa. A natural merge between real and virtual worlds will be obtained when the shading
and shadows correspond between these two worlds. Sugano et al. [7] and Madsen et al. [19]
highlight the importance of consistent shadows in AR environment.
Research which is related to performance of shadow can be found in [20][11][15]. These
proposed approaches were designed to run in graphics hardware and offer the way to balance
the performance without sacrificing the visual quality of shadows. Besides that, generating
shadows using shadow maps [7][15] or shadow volume [12] can be developed at a low cost.
Meanwhile, Haller et al. [12] proposed the concept of using shadow volume, which is focusing
more on the shadow problems in AR system. The reality of AR world was improved with projected
real shadows onto virtual objects and vice versa.
The survey of soft shadow classification method was done in [21]. This method can be applied to
generate shadows in the context of AR environment. Jacobs et al. [13] and Madsen and Laursen
[15] point the issues of double shadow in AR environment, where they had solved the overlapping
between real and virtual shadow to produce realistic shadow. A real-time rendering solution to
simulate color-consistent of virtual shadows in a real environment was presented by Jacobs et al.
[13]. The rendering process in their proposed method has three step mechanisms; shadow
detection, shadow protection and shadow generation, in which every step produces consistent
shadows between real and virtual objects in real-time. To accomplish the successful shadow
detection and shadow generation in their method, three requirements are needed: the geometry,
the light source position and only hard or semi-soft shadows are allowed.

The credibility of shadow generation also can be achieved with correct estimation of light source
position. The researches related to the light source position can be found in [5] [17][22], which all
have the same direction, to propose a method that can create lighting for virtual object in AR
environment to be as realistic as possible like in the real world. This paper is based on the
research that estimates light source position to render soft shadows which is associated with the
real world.
3. SHADOW GENERATION IN AR
This section will explain the detail process of generating realistic shadow. The framework was
created to lead the development process of shadow and the method of soft shadow that will be
used in this work will be explained.
Implementation
The problems in terms of consistency of geometric and photometric registration as described in
[23] will be highlighted in the implementation processes. These problems need to be resolved
especially when dealing with unrealistic fake shadows generation. Then, shadow will be rendered
based on estimation of light source position in the real scene and get the correct-perception of
user viewpoint. This result will produce the photorealistic rendering in AR environment. In this
paper, the implementation of the proposed method involves several steps as illustrated in Figure
1.
The cameras will detect markers composed with reflective spheres in the real scene. Then, the
relationship between the 2D marker and camera coordinate system will be determined using
existing technique. This relationship is an important step to complete the geometric registration so
that the virtual object is placed in the right position. After that, the system will detect the reflective
sphere segmentation on the marker, which is painted with glossy black paint to avoid the dynamic
range problem to create the environment map. Environment map will define the incoming light
from all possible directions at some reference point.
The system will use median cut algorithm, which presents a credibly complex lighting
environment. This algorithm will produce a set of light sources [25] and will estimate the light
source position in real scene from the environment map. The light sources are used to generate
realistic shadows using projection shadow. Then, the soft shadow will be applied to get smooth-
edges outline of shadows so that the shadows will look realistic. Finally, the 3D virtual object with
the correct attached shadow is rendered in the real scene.
The setup of the system in this work is inspired from [5]. This setup consists of laptop or personal
computer as a display device, camera to track marker and display virtual object as output video,
2D marker to display virtual object, 3D marker to estimate light source from the real scene and
light as a light source for the system. 3D marker was constructed using the reflective sphere and
2D marker. Figure 2 illustrates the setup of the system prototype.
Median cut algorithm
The algorithm will split the environment map image into 2n regions based on latitude and
longitude format. The regions that are already split have an equal light energy and each of them
has a light source representation. The steps to split the environment map image are as follows:
1. The environment map of light probe is added to the region list as a single region.
2. Every region in the list will subdivide along the longest dimension so that its light energy
is split evenly.
3. Return to step 2 if number of iteration is less than n.
4. The light source is placed at the center of each region and the color of light source is set
to the sum of pixel values within the region [25].

The advantages of using this algorithm are it is efficient, fast and easy to put into practice. The
algorithm is very practical to apply the merging of virtual object into the real scene.
FIGURE 1: Frameworks of System Prototype.
Camera tracking
Marker detection
Render 3D objects
with soft shadows
Create shadow
Detect reflective
sphere segmentation
Create
environment map
Median Cut
Algorithm
Estimate light
source direction
Photometric
Registration
Geometric
Registration
Soft shadow

FIGURE 2: Setup of the System Prototype.
Soft Shadow
The generated shadows are still unrealistic since they have sharp and hard-edges outlines of
shadows, also known as hard shadow. To get a realistic shadow, the soft shadow method must
be applied. The differences in appearance between hard and soft shadow is depicted in Figure 3.
FIGURE 3: Hard Shadow (Left) and Soft Shadow (Right) [21].
The soft shadow method that can be applied in this research is based on the concept of Heckbert
& Herf’s soft shadow [27]. The method will produce the number of hard shadows samples. These
samples consist of the number of different dark color and size of the shadows, where the size
slightly bigger than the size of the original shadow. The numbers of samples influence the quality
of soft shadow. The higher number of sample used, the higher quality soft shadow will be
generated and vice versa.
After few number of hard shadow samples have been produced, the samples will be blended
together with different dark colors and sizes together with the original shadow. This process will
be done by stacking each other starting from the less dark color with ends with original color of
the shadow. Figure 4 shows the overlapping process of samples hard shadow.
+ + + + =
Overlapping
FIGURE 4: Overlapping Process of Samples Hard Shadow.

4. EXPERIMENT
This section discussed the experiment of rendering soft shadow in AR environment. The
experiments were conducted using three samples hard shadow in which every sample has
different number of hard shadow.
Testing Platform
In this experiment, the laptop and camera device with specification specified in Table 1 and Table
2 were used to conduct the experiment processes. The results from this experiment are
discussed based on Frame per Second (FPS) and the appearance of the soft shadow on the
display screen measured the quality of soft shadow.
Processor
Intel® Core ™2 Duo CPU T7250 @
2.00GHz
Memory (RAM) 3.00 GB
Graphics Card NVIDIA GeForce 8400M GS
Operating System Microsoft Windows XP Professional
TABLE 1: Specification of the Laptop for Display Soft Shadow.
Type Aloha Digital PC Camera
Frame rate VGA, 30 Frame/Sec
White Balance Automatic
Megapixels up to 8 megapixels
Over View Glass element lens
Connectivity USB 2.0
System
Requirements
Windows 2000, Windows XP,
Windows Vista
TABLE 2: Specification of the Camera.
Soft shadow process
The soft shadow was generated based on the concept of Heckbert & Herf’s soft shadow [27] as
described in previous section. This technique used two parameters to produce the soft shadow
which is length and gap factors. The length factor determines the length of soft shadow from the
original hard shadow. Meanwhile, the gap factor determines the distance between hard shadows
in the sample. Figure 5 illustrated the concept of Heckbert & Herf’s soft shadow [27] with length
and gap factors.

Length factor
Gap factor
Original hard
shadow
FIGURE 5: Illustration of Soft Shadow with Length and Gap Factors.
5. RESULT
This section will discuss the result from the experiment of rendering soft shadow in AR
environment. The quality of shadow appearance is important to produce realistic AR environment.
In this experiment, three samples of hard shadow are being used which consist of five, seven and
ten shadows in each samples. Every sample will be measured in terms of their performance
based on FPS and the number of the light source used in median cut algorithm.
(a) (b)
FIGURE 6: Shadow of Computer Generated Object Based on Real Light Position.
The median cut algorithm was used to estimate real light source position from the real scene
using the environment map. Figure 6 (a) and 6 (b) show the shadow that render based on
different position of real light source. The object with green color is a computer generate object
and object with white color is the real object. These figures show the computer generated object
produce shadow that are in the same direction of real shadow. Thus, these make the AR
environment more realistic. The object in this experiment also can be applied to more complex
geometry objects and render multiple objects in the one AR environment.

(a) (b)
(c) (d)
FIGURE 7: Comparison of Appearance Hard Shadow (a) and Soft Shadow Generation (b)(c)(d).
Figure 7(a) shows the result of hard shadow that was generated from the system in AR
environment. The appearance of hard shadow has proved that it is not realistic because it has
sharp and hard-edge outlines. Meanwhile, Figure 7(b), (c) and (d) show the result of soft shadow
being generated from Heckbert & Herf’s technique [27] with different samples of hard shadow.
Sample 1 consists of 5 layers of hard shadow, sample 2 consists of 7 layers of hard shadow and
sample 3 consists of 10 layers of hard shadow. From these results, we can see that the number
of hard shadow layers influenced the quality of shadow appearance. The quality of soft shadow
becomes higher with the large number of hard shadow layer and less quality with small number of
hard shadow layer.
The experiment also measures the performance of the system based on FPS. Figure 8 illustrates
the graph of comparison performance between the three samples of hard shadow used in the
experiment. In this graph, the result of FPS depends on the number of light source used in the
median cut algorithm. The graph shows the FPS will be decreased with the increasing number of
light sources. It is because every light source renders the samples of hard shadow which involved
a lot of computation.

FIGURE 8: Comparison of 3 Samples of Hard Shadow Layer.
Apart from that, the samples used in the experiment influenced the performance of the system.
From the graph, sample 1 with five layers of hard shadow produced the highest FPS compared to
sample 3 which contain ten layers of hard shadow. This means with the higher number of hard
shadow layers, the high quality will be produced but the performance becomes low. It is
contradictory with the small number of hard shadow layer with high performance but produce low
quality of soft shadow appearance.
Thus, to achieve realistic soft shadow in AR environment, it involved a large number of hard
shadow layer. Since higher computational will decrease the performance of the system, the
optimization techniques is needed such as Level of Detail (LOD), Culling, Octree and others.
These techniques will increase the performance of the system.
6. CONSLUSION & FUTURE WORK
The method to create soft shadow in AR is presented in this paper based on the concept of
Heckbert & Herf’s soft shadow [27]. Compared to hard shadow, the purpose of soft shadow is to
add realistic on appearance of shadow in AR environment. This is because hard shadow still has
drawbacks such as sharp and hard-edge outlines that are deficient in the appearance of shadow.
In this paper, soft shadow was rendered based on estimated light source from the real scene.
The estimation of real light source is important to create credible shadow. However, the
experimental result shows the high quality of soft shadows will reduce the performance of the
system. In future works, the development of method to improve the performance of the system
without sacrificing the quality of the system must be applied to produce realistic AR environment.
7. ACKNOWLEDGMENTS
We would like to express our appreciation to Universiti Teknologi Malaysia (UTM) and Malaysian
Ministry of Higher Education (MOHE) for providing financial support for this research through
Fundamental Research Grant Scheme (vot no.: 78599).

8. REFERENCES
1. R. Azuma. “A Survey of Augmented Reality”. Presence: Teleoperators and Virtual
Environments, 6:355-385, 1997
2. Z. Noh, M.S. Sunar and Z. Pan. “Review on Augmented Reality for Virtual Heritage System,”
Edutainment 2009 Springer-Verlag Berlin Heidelberg, Lecture Note in Computer Science
LNCS 5670, pp. 50–61, 2009.
3. Z. Noh, A.W. Ismail and M.S. Sunar. “Exploring the Potential of using Augmented Reality
Approach in Cultural Heritage System”, Proceedings of the 2nd International Conference on
Advanced Computer Theory and Engineering (ICACTE 2009), Cairo, Egypt, vol. 1, pp. 711-
718, 2009.
4. I. Sato, Y. Sato and K. Ikeuchi. “Acquiring a Radiance Distribution to Superimpose Virtual
Objects onto a Real Scene”. IEEE Transactions on Visualization and Computer Graphics,
5(1):1-12, 1999
5. B. F. Jensen, J. S. Laursen, J.B. Madsen and T. W. Pedersen. “Simplifying Real Time Light
Source Tracking and Credible Shadow Generation for Augmented Reality”. Institute for
Media Technology, Aalborg University, 2009
6. Z. Noh, M.S. Sunar “A review on shadow techniques in augmented reality”, 2nd International
Conference on Machine Vision, ICMV 2009 , pp. 320-324, 2009
7. N. Sugano, H. Kato and K. Tachibana. “The Effects of Shadow Representation of Virtual
Objects in Augmented Reality”. In IEEE/ACM International Symposium on Mixed and
Augmented Reality (ISMAR 2003), IEEE Computer Society, pp. 76-83, 2003
8. K. Jacobs, C. Loscos. “Classification Of Illumination Methods For Mixed Reality”. Computer
Graphics Forum, 25(1):29-51, 2006
9. O. Bimber, A. Grundheimer, G. Wetzstein and S. Knodel. “Consistent Illumination within
Optical See-Through Augmented Environments”. In Proceedings of the 2nd IEEE and ACM
International Symposium on Mixed and Augmented Reality, pp.198 , 2003
10. K. Agusanto, L. Li, Z. Chuangui and N.W. Sing. “Photorealistic Rendering for Augmented
Reality Using Environment Illumination”. In Proceedings of the 2nd IEEE and ACM
International Symposium on Mixed and Augmented Reality, pp.208, 2003
11. S. Gibson, J. Cook, T. Howard and R. Hubbold. “Rapid Shadow Generation in Real-World
Lighting Environments”. In Proceedings of Eurographics Symposium on Rendering 2003, pp.
219–229, 2003
12. M. Haller, S. Drab and W. Hartmann. “A Real-Time Shadow Approach for an Augmented
Reality Application Using Shadow Volumes”. In Proceedings of VRST 03, pp. 56–65, 2003
13. K. Jacobs, C. Angus and C. Loscos. “Automatic Generation of Consistent Shadows for
Augmented Reality”. In Proceedings Graphics Interface, Vancouver, Canada, 2005
14. C.B. Madsen, M. Nielsen. “Towards Probe-Less Augmented Reality”. A Position Paper,
Computer Vision and Media Technology Lab, Aalborg University, Aalborg, Denmark, 2008
15. C. B. Madsen, R. Laursen. “A Scalable GPU-Based Approach to Shading and Shadowing for
Photo-Realistic Real-Time Augmented Reality”. In Proceedings International Conference on
Graphics Theory and Applications, pp. 252 – 261, Barcelona, Spain, 2007
16. C. Loscos, G. Drettakis and L. Robert. “Interactive Virtual Relighting of Real Scenes”. IEEE
Transactions on Visualization and Computer Graphics, 6(3):289-305, 2000
17. F. Yan. “Estimation of Light Source Environment For Illumination Consistency of Augmented
Reality”. In First International Congress on Image and Signal Processing, 3:771-775, 2008
18. T. Naemura, T. Nitta, A. Mimura and H. Harashima. “Virtual Shadows in Mixed Reality
Environment using Flashlight-Like Devices”. Trans. Virtual Reality Society of Japan, 7(2):
227-237, 2002
19. C.B. Madsen, M. K. D. Sorensen and M. Vittrup. “The Important of Shadows in Augmented
Reality”. In Proceedings 6th Annual International Workshop on Presence, Aalborg, Denmark,
2003
20. S. Gibson, A. Murta. “Interactive Rendering with Real World Illumination”. In Proceedings of
Eurographics Symposium on Rendering 2000, pp. 365–376, 2000

21. J. Hasenfratz, M. Lapierre, N. Holzschuch and F. Sillion. “A Survey of Real-Time Soft
Shadows Algorithms”. Eurographics, 22(4):753-774, 2003
22. M. Kanbara, N. Yokoya. “Real-Time Estimation of Light Source Environment for
Photorealistic Augmented Reality”. In Proceedings of the 17th International Conference on
Pattern Recognition, pp. 911–914, Cambridge,United Kingdom, 2004
23. M. Kanbara, N. Yokoya. “Geometric and Photometric Registration for Real-Time Augmented
Reality”. In IEEE and ACM International Symposium on Mixed and Augmented Reality
(ISMAR), pp. 279, 2002
24. L. Williams. “Casting Curved Shadows on Curved Surfaces”. Computer Graphics. In
Proceedings of ACM Siggraph, 1978
25. P. Debevec. “A Median Cut Algorithm for Light Probe Sampling”. International Conference on
Computer Graphics and Interactive Techniques, Los Angeles, California, 2005
26. J. Konttinen, C.E. Hughes and S.N. Pattanaik. “The Future of Mixed Reality: Issues in
Illumination and Shadows”, J. Defense Modeling and Simulation, 2(1):51-59, 2005
27. M. Herf, P.S. Heckbert. “Fast Soft Shadows”. Visual Proceedings SIGGRAPH 96, pp. 145,
1996

Soft Shadow Rendering based on Real Light Source Estimation in Augmented Reality

More Related Content

What's hot (20)

Similar to Soft Shadow Rendering based on Real Light Source Estimation in Augmented Reality (20)

More from Waqas Tariq (20)

Recently uploaded (20)

Soft Shadow Rendering based on Real Light Source Estimation in Augmented Reality