Wujanz_Error_Projection_2011

VISUALISATION OF DEVIATIONS ON ARBITRARY
SURFACES FOR QUALITY ASSURANCE AND
PROCESS CONTROL
D. Wujanz1
, N. Heuwold2
, S. Weisbrich1
, F. Neitzel1
, J. Collstrup1
, C. Gaitanis1
1
Berlin Institute of Technology
Department of Geodesy and Geoinformation Science
Chair of Geodesy and Adjustment Theory
Straße des 17. Juni, 10623 Berlin, Germany
d.wujanz@campus.tu-berlin.de
www.geodesy.tu-berlin.de
2
GFaI – Society for the Promotion of Applied Computer Science
Department for 3D-Data Processing
Volmerstraße 3, 12489 Berlin, Germany
heuwold@gfai.de
www.gfai.de
KEY WORDS: Terrestrial 3D Laser Scanning, Structured Light, Visualisation, Projector,
Quality Assurance, Deformations, Virtual Targets
ABSTRACT:
The detection of deviations plays a crucial role in quality assurance as well as in manufacturing
processes where for instance differences between a CAD and the current state of an object have to be
determined. Hence a problem that arises is the interpretation of the outcome which is displayed on a
monitor in 2D. By combining a projector and 3D-measurement devices a system is presented that is able
to visualise detected deviations on arbitrary surfaces. Thus this technique can be used in process control
and inspection in order to satisfy demands of quality assurance. The approach is demonstrated by
applying terrestrial laser scanners and structured light scanners. In order to solve the problem of
signalisation which is needed to compute the relative orientation between measurement system and
projector unit different approaches are presented. One of them is a novel method referred to as virtual
targets where no physical tie points have to be distributed within the region of interest.
1 INTRODUCTION
Terrestrial Laser Scanners (TLS), structured light scanners and other contactless surveying
instruments are widely used for instance in alignment, as-built documentation, positioning,
deformation analysis or quality assurance. The outcome of these processes is usually
presented in numeric form, through a raw point cloud or, for the last two cases, by a colour
coded representation of the deviations respectively geometric alterations. The interpretation
of colour coded 3D information that is displayed in 2D (inspection map) can be quite
difficult depending on the geometric characteristics or extent of the surveyed object. In order
to simplify this task approaches from augmented reality (AR) have been taken into
consideration where the inspection map is projected on to the object itself.

Therefore several problems have to be solved:
 Transformation of reference dataset (e.g. CAD) into the current sensor coordinate
system.
 In order to receive the parameters of the interior orientation a calibration has to be
carried out before usage of the projector unit. Alternatively they can be calculated
by using homologues points as described in section 2.4.
 Determination of the relative orientation between projector unit and current sensor
coordinate system.
 Conversion of the inspection map into a 2D projector image.
Note that the projector image has to be resampled in a way that geometric properties of the
object have to be taken into consideration. The problem of tie point signalisation for the use
of TLS is presented in two ways. Amongst them is a novel approach called virtual targets
which uses the internal camera of the applied TLS and the projector unit in order to
arbitrarily distribute simulated tie points around an object or within a scene of interest.
1.1 Related work
HEILIG (1962) probably took the first steps towards mixed reality, which can be understood
as a mixture between the real world and virtual components, by developing a multimodal
arcade game called Sensorama based on a stereoscopic video. A classification of mixed
reality categories and their distinctive features can be found in MILGRIM & KISHINO (1994)
while MILGRIM et al. (1994) give a theoretical overview specifically on augmented reality
(AR). RASKAR et al. (2001) apply several projectors in order to generate a virtual texture by
projecting images onto a neutrally coloured object with arbitrary geometric properties. Note
that the current implementation of our system does not apply a camera in order to correct the
computed imagery against the given texture of an object as shown for instance by BIMBER et
al. (2005). A technical limitation of such a system is possibly given by insufficient
resolution of the projector unit e.g. when the object is far away or its surface is comparably
large. This problem could be solved by sequentially projecting sub matrices of the
inspection map by applying a motor controlled projector as described by PINHANEZ (2001)
or by using multiple projectors as proposed by CHEN et al. (2000) in a simultaneous
solution. RASKAR et al. (2003) apply a handheld projector that uses a tilt sensor and a
camera in order to compensate for keystone distortion and also present a self-configuring
technique for several projector units. LEE et al. (2004) solve the problem of determining the
relative orientation between projector and desired scanner coordinate system by embedding
optical fibres into the object whose position can be captured by a structured light scanner.
The proposed technique in this contribution does not apply active sensors for the mentioned
purpose within the scene in order to being applicable in unreachable or untouchable areas,
for instance in museums or on unstable surfaces.
2 DESCRIPTION OF THE SYSTEM
One of the goals of this new approach was to make it adaptable and applicable to many
different problem domains by using standard equipment that is usually available in every
office. This section describes which components have been applied in order to establish the

system. Furthermore a variation by applying another 3D measurement instrument and finally
the mathematical background are presented.
The workflow of the proposed system can be divided into three major steps:
 Data preparation, which includes acquisition, transformation and comparison, is
described in section 2.2,
 details concerning the relative orientation between the applied sensor systems are
presented in section 2.3,
 and finally the transformation of the 3D content itself into a planar form is
demonstrated in section 2.4.
2.1 Applied components
Two types of 3D measurement systems have been applied during the implementation of the
approach. One being a Leica C10 terrestrial laser scanner with a maximum reach of 300 m
the other being a GFaI structured light scanner (GFAI 2011a) which can be adapted to
different scenarios in terms of reach and spatial resolution. The off-the-shelf projection unit
is fabricated by projectiondesign (PD 2011) whereas the F1 model with a maximum sensor
resolution of 1280 x 1024 pixels has been used.
2.2 Data preparation
The first step of this procedure is the data acquisition process in which the object of interest
is captured. Subsequently this dataset has to be compared to a reference which can be a
CAD or known geometric properties, for instance that an object is assumed to be planar.
Figure 1 depicts how an inspection or deviation map is processed. In this figure both
datasets are already correctly aligned, so that distances between the two geometries can be
calculated. Afterwards these distances are converted into colours according to settings that
can be adjusted by the user such as the acceptable spectrum of deviations. Various solutions
are available on the market to fulfil this step such as Raindrop Geomagic Qualify or GFaI
Final Surface (GFAI 2011b).
Figure 1: Determination of an inspection map between measured values (left) and a CAD (middle).
The outcome is shown on the right.

In order to calculate correct deviations between reference dataset and measured values
transformation parameters have to be determined which can e.g. be done by using BESL &
MCKAY’S (1992) iterative closest point algorithm (ICP). This procedure is also referred to
as registration or matching. Note that the superior coordinate system has got to be the one of
the measurement instrument in which the reference dataset has to be transformed as the
inspection map should be projected on to the measured object. It has to be mentioned that
this transformation is a crucial step, which is of course also subject to falsification by the
geometric quality of the measurements, as it directly influences the outcome of the deviation
or inspection map which is computed afterwards. Note that the ICP is falsified by
deviations, which is why they have to be excluded from the registration process otherwise
erroneous transformation parameters would be involving. Current research focuses on this
subject.
2.3 Relative orientation
The aim of this step is to determine the relative orientation between the coordinate system of
the measurement instrument and the projector unit which will be described in detail in the
following section. Therefore tie- or homologous points have to be determined where six
points are the minimal configuration as 11 parameters have to be solved. Six unknowns
describe the exterior orientation while the remaining five represent the interior orientation.
These points are measured in three dimensions in order to represent the scanner’s location
while two-dimensional coordinates are determined within the projector unit system. While
the registration process influences the outcome of the inspection map the determination of
tie points has its impact on how accurate the visual information can be projected onto the
object. Therefore an established approach as well as a completely new solution for TLS
have been analysed.
The first option for TLS usage applies physical targets that have been distributed around the
area of interest. While the determination of the 3D coordinates within the measurement
instrument system can be achieved automatically the 2D coordinates required human
interaction. Therefore the centre had to be detected by manually pointing the mouse cursor
into the middle of a target whose display coordinates were then stored in a coordinate file.
Note that Leica’s Cyclone software, which has been applied to determine the centre
coordinates of all targets, works only on intensity data that describes a quotient between
emitted and received energy of the laser beam.
As physical targets such as spheres, paper print-outs or coded tilt and turn versions can be
unpractical to distribute especially within larger scenes or hardly accessible areas it was
opted for a more convenient solution. Motivated by these drawbacks a new approach called
virtual targets has been developed where the projector and the internal camera of the TLS,
which is calibrated to the scanner’s coordinate system, have been applied. Another vital
component of this system is software where coded targets can be placed interactively within
the projectable region by moving the cursor into the designated position. In order to ensure
sufficient local resolution, placed targets can be altered in size to adapt for decreasing
resolution of optical sensors and TLS with preset point spacing.

In order to test the approach a scene including physical and virtual targets has been captured
in terms of geometry and RGB imagery. As expected no distinguishable intensity
information has been detected of the projected targets by the TLS so that a workaround had
to be developed in order to solve this problem. Therefore the RGB imagery from the
scanner’s internal camera has been converted into a one channelled grey scale image.
Subsequently these 8-bit values are then recomputed into a range between 0 and 1 and hence
can be used to replace all captured intensity values for each according 3D point. Note that
this approach assumes nearly planar projection surfaces as free form shapes would interfere
with the automatic determination of the target centre.
Figure 2 illustrates several products captured from TLS that finally lead to usable targets.
The left part shows a RGB image captured by the internal scanner camera. Six physical
print-out targets are attached to the wall while the four slightly smaller targets on the outer
left side are projected targets and hence virtual ones. These projected targets aren’t visible in
the intensity image which can be found in the middle section. After conversion from RGB
values into intensity values and its integration into the point cloud, the determination of
target centres can be carried out as shown in the right part of the figure below.
Figure 2: RGB image captured by the internal camera of the scanner (left), measured intensity
information (middle), virtual targets derived from RGB imagery and physical targets (right)
It has to be mentioned that the problem simplifies significantly when structured light
scanners are applied as the relative orientation between measurement system and projector
has already been determined during calibration. Thus this case applies the projection unit as
an integral component of the measurement system itself. Another advantage of this system is
that there is no parallax between measurement system and the projected information as they
share the same optical path.
2.4 Transformation
Perspective mapping from an Euclidian three-dimensional space to a two-dimensional
space can be, according to HARTLEY & ZISSERMAN (2010), solved by introducing
homogenous coordinates for object points ( ) as well as image points ( ). A major
advantage of using homogenous coordinates is that rotation, translation and scaling can be
applied to a set of points quite convenient as all necessary information is stored in one
matrix. Homogeneous coordinates are composed by extending the Cartesian coordinates of
the Euclidian space by an additional dimension. The relationship between Cartesian and
homogeneous coordinates can be described by

(1)
where denotes an arbitrary scaling factor. Using homogeneous coordinates comprehends
the possibility to describe a transformation between object and image points via projection
matrix P
[ ] [ ] [ ] [ ] (2)
where R denotes the rotation matrix and C the translation vector between object and image
coordinate system. Furthermore calibration matrix K represents the interior orientation of the
used projector and only has to be determined once. In order to derive projection matrix P
from given tie points, object as well as image coordinates for each point have to
be normalised and reduced to the centroid by determining ̃ respectively
̃ with
[
⁄ ⁄
⁄ ⁄
⁄ ⁄
] (3)
and
[
⁄ ⁄
⁄ ⁄ ] (4)
where and represent the absolute maximum value for object
respectively image coordinates. For each one of the n tie points the following two rows of
matrix have to be set up
[
̃ ̃ ̃ ̃
̃ ̃ ̃ ̃
] (5)
and finally combined to design matrix which is containing elements. An elegant
way to solve the equation system
(6)
with is to determine the right singular vector related to the
smallest eigenvalue which can be achieved via Singular Value Decomposition (SVD) of .
This vector contains the 12 elements of the projection matrix ̃ for normalised and
reduced coordinates. Finally the projection matrix can be derived from
̃ . (7)
Due to the fact that only depends on eleven parameters all elements have to be divided by
its last entry . For further information see FÖRSTNER (2000), RODEHORST (2004) or
HARTLEY & ZISSERMAN (2010). Figure 3 illustrates the workflow of the proposed system
including all previously mentioned components.

Figure 3: Workflow of the presented approach
3 SAMPLE APPLICATIONS
This section focuses on potential sample applications such as alignment and quality
assurance. Before these examples are presented the workflow has been tested in order to
verify its operational reliability. Therefore a checkerboard pattern has been mapped onto the
three dimensional mesh of a surveyed bust by applying a parallel projection. After
application of the technique described in section 2.4 the left part of Figure 4 emerges. The
silhouette of the bust is caused by the fact that projection direction and viewing angle of the
projector differ. The same reason leads to effects that are visible in the right part of the
illustration where a shade right next to and above the bust can be noticed. A close look at the
image reveals imperfections in transition sections between bust and wall which can be
simply explained by a parallax between projection direction and location in relation to the
camera standpoint and its alignment. In summary the quality of the outcome fulfils all
expectations so that further applications have been explored.
Figure 4: Verification of the approach. Calculated image (left) and projected outcome (right).
3.1 Quality assurance and inspection
The main motivation of this project was to develop a more interactive and at the same time
simpler way of how to present and communicate products that have been derived via 3D
survey. The first idea for potential fields of applications was quality assurance, inspection of
produced goods or of products within the manufacturing process. Hence the first example

that has been tackled was a comparison between a CAD and a surveyed bust as depicted in
Figure 1 on the left. Note that the CAD has been derived from surveyed data by carrying out
reverse engineering as well as removal of obvious dents.
After the survey has been carried out by TLS six tie-point coordinates have been determined
within the coordinate system of the measurement instrument and of the projector unit.
Subsequently the CAD has been registered into the coordinate system of the surveyed data
whereas an average deviation of 2.3 mm between the two datasets resulted from ICP. Based
on these datasets an inspection map has been computed which was then processed by the
implemented software and projected onto a white coloured version of the bust as illustrated
in Figure 5. On closer inspection it can be seen that darker shades of blue, for instance
located on the shin and on the nose of the bust, which are clearly noticeable in the left part
of the image below, aren’t visible on the right. This effect can be explained by the quality of
the projector. Concluding the achieved geometric quality of the outcome can be rated quite
positive as no apparent visible offsets occurred apart from the lack of contrast by the
projector.
Figure 5: Computed image (left) and projected picture (right).
3.2 Object alignment
Another application where the proposed technique can be applied would be for alignment of
laminar objects. To this day most alignment processes are carried out with discrete sensors
such as total stations which are quite time consuming to operate. Modern 3D scanners are
able of capturing point clouds within very short periods of time, e.g. GFaI ScanMobile
which takes about 1 s to 1.5 s. This capability opens possibilities for online systems where
coordinates are captured continuously while deviations to a designated position are
projected through colour coding onto the object of interest.
A simple demonstration has been conducted by using two arbitrarily positioned cardboard
boxes on which white sheets of paper have been attached in order to increase contrast. The
goal of this test was to bring both boxes into a desired alignment. Therefore one of the boxes
was declared to be the reference which has been established by approximating a plane
through the according points. Subsequently the remaining points have been compared to the
reference plane where colours were assigned to each point. After projection of the resulting
pattern onto the box the position of it has been corrected, a new survey was conducted and a
new projectable pattern has been determined. During this test a coarse to fine approach has

been chosen so that the colour spectrum was tuned to the expected maximum deviation of
the current position of the box.
Figure 6 illustrates how an alignment based on the proposed technique works. The left part
of the image shows both boxes in their initial position whereas the left one has to be aligned
parallel to the right green coloured box. A circular grey area visible on the last mentioned
box is caused by a target that shielded the box against it. As this was the first position of the
demonstration large deviations of up to +/- 30 cm have been set as acceptable while the
colour spectrum was segmented in 5 cm steps. As the projected image shows step-like
offsets a rotational misalignment apart from the translational offset can be spotted. The
figure in the middle depicts the corrected position and alignment based on the previously
mentioned pattern. Note that the colour spectrum has been refined in this step where
different colours separate by 1 cm. After two iterations the image on the right emerges
where both boxes are aligned to a satisfactory degree. The yellow spot in the left box is
caused by a bulge of the attached paper which becomes visible as the tolerance between two
colours is set to +/- 0.5 cm.
Figure 6: Processing steps of the alignment: initial position of the boxes (left), corrected location
according to projected colour coding (middle) and final alignment (right).
4 CONCLUSIONS AND OUTLOOK
A new visualisation technique for projecting inspection maps on to arbitrary surfaces with
off-the-shelf projection units and application of different 3D measurement devices is
presented. Furthermore the use of virtual targets for TLS is proposed, a convenient approach
to interactively place simulated targets within a region of interest. After demonstration of the
capabilities of the implemented system two primary fields of application, quality assurance
and alignment are revealed on practical examples.
Apart from the demonstrated use of virtual targets they can be applied for connecting point
clouds in inaccessible areas or where the ICP-Algorithm might fail such as planar surfaces.
The presented projection technique can not only be useful within the geometric domain but
also in other fields where problem specific information can be linked to spatial information
for instance multispectral, thermal or acoustic information of objects that have been
captured e.g. by sensor fusion or new developments.
Possible scenarios for the application of the presented system can be found in various fields
of application for instance on construction sites where concrete slabs could be checked for
planarity during casting. Free formed metal sheets, as used on shipyards, could be
interactively aligned before welding. A variety of computer designed work pieces such as
moulds from a technical perspective or products for instance manufactured by stone cutters
to cover the artistic point of view are still handcrafted goods due to the low quantity of
batches. By applying the proposed technique the manufacturing process could be controlled
to by a much higher degree and could provide assistance during fabrication. Car rentals
could introduce the system in combination with a camera to correct against object colour in
order to locate damages after vehicles have been returned.

ACKNOWLEDGEMENTS
The first author would like to express his gratitude to his colleagues Ben Jastram (3D
Laboratory, TU Berlin) and Tian Yumiao for their help during this project.
REFERENCES
Besl, P. & McKay, N., 1992. A method for registration of 3D shapes. IEEE Trans. Pattern Anal. Mach.
Intell. 14, pp. 239–256.
Bimber, O., Emmerling, A., Klemmer, T, 2005. Embedded Entertainment with Smart Projectors. In:
IEEE Computer, pp. 48–55.
Chen, Y., Gotz, D., Hensley, J., Towles, H., Brown, M., 2000. Automatic Alignment of High
Resolution Multi-Projector Displays Using an uncalibrated Camera. In: Proceedings of IEEE
Visualization, IEEE CS Press, pp. 125-130.
Förstner, W., 2000. Moderne Orientierungsverfahren. Photogrammetrie – Fernerkundung –
Geoinformation, Heft 3/2000, pp. 163-176.
GFAI 2011a. GFaI ScanMobile Datenblatt. http://www.gfai.de/forschungsbereiche/3d/3ddv/sites/
default/files/GFaI_ScanMobile_Datenblatt.pdf (accessed 27.10.2011).
GFAI 2011b. Final Surface® Produktseite. www.final-surface.de (accessed 27.10.2011).
Hartley, R., Zisserman, A., 2010. Multiple View Geometry in Computer Vision, Cambridge University
Press.
Heilig, M. L., 1962. Sensorama Simulator. U.S. Patent 3,050,870.
Lee , J., Dietz, P., Maynes-Aminzade, D., Raskar, R., Hudson, S., 2004. Automatic projector calibration
with embedded light sensors. In: Proceedings of the 17th annual ACM symposium on user interface
software and technology, United States of America.
Milgram, P., Kishino, F., 1994. A Taxonomy of Mixed Reality Visual Displays. In: Transactions on
Information and Systems, Special issue on Networked Reality.
Milgram, P., Takemura, H., Utsumi, A., Kishino, F., 1994. Augmented Reality: A class of displays on
the reality-virtuality continuum. In: Telemanipulator and Telepresence Technologies, Vol. 2351,
Boston, United States of America, pp. 282-292.
PD 2011. Projectiondesign F1 brochure. http://www.projectiondesign.com/applications/3d-stereoscopic-
display/_attachment/1866?_ts=1302756843e&download=true. (accessed 27.10.2011).
Pinhanez, C., 2001. The Everywhere Displays Projector: A Device to Create Ubiquitous Graphical
Interfaces. In: Proceedings of the 3rd international conference on Ubiquitous Computing, Atlanta,
United States of America, pp. 315-331.
Raskar R., Welch, G., Low K., Bandyopadhyay, D., 2001. Shader Lamps: Animating real objects with
image-based illumination. In: Proceedings of the 12th Eurographics Workshop on Rendering
Techniques, pp. 89-102.
Raskar, R., van Baar, J., Beardsley, P., Willwacher, T., Rao, S., Forlines, C., 2003. iLamps:
geometrically aware and self-configuring projectors. In: ACM Transactions on Graphics (TOG).
Rodehorst, V., 2004. Photogrammetrische 3D-Rekonstruktion im Nahbereich durch Auto-Kalibrierung
mit projektiver Geometrie. wvb Wissenschaftlicher Verlag, Berlin, Germany.

Wujanz_Error_Projection_2011

More Related Content

What's hot (20)

Similar to Wujanz_Error_Projection_2011 (20)

Wujanz_Error_Projection_2011