Gamification with virtual characters at the borders of mixed realities and algebras

CASA, Geneva, 2016 Slide 1University of Crete & Foundation for Research and Technology - Hellas
Gamification with virtual characters at the
borders of mixed realities and algebras
Prof. George Papagiannakis
University of Crete
&
Foundation for Reserach and Technology Hellas
Heraklion, Crete, Greece

From Augmenta
http://whatisaugmenta.com

To Magic Leap
http://www.magicleap.com/#/home

From J.A.R.V.I.S.
http://marvel-movies.wikia.com/wiki/J.A.R.V.I.S.

To Hololens
https://www.microsoft.com/microsoft-hololens/en-us

& Asobo
http://www.asobostudio.com

Overview
• A journey in Mixed Reality
(MR) and Algebras
• How many Realities?
• Gamification
• Presence?
• Different algebras for
character simulation
• Dual quaternions
• Geometric algebra?
• Current motivation and MR
research questions
• MR character enabling
technologies
• Conclusions and future
work

How many Realities? Mixed Reality = AR + VR+ AV+ CR+ IR+ HR+ SR…
MR Reality-Virtuality Continuum
Extent of World Knowledge
World
Unmodeled
World Fully
Modeled
Real
Environment
Virtual
Environment
AR VR
AR-VR: Papagiannakis, G., Schertenleib, S., O'Kennedy, B., Poizat, M., Magnenat-Thalmann, N., Stoddart, A., Thalmann, D., 2005. Mixing Virtual and
Real scenes in the site of ancient Pompeii. Computer Animation and Virtual Worlds, John Wiley and Sons Ltd 16, 1, 11–24.
Cross-Reality (CR): Davies, C.J., Miller, A., and Allison, C. 2012. Virtual Time Windows: Applying cross reality to cultural heritage. Proceedings of the
Postgraduate Conference on the Convergence of Networking and Telecomunications, ISBN: 978-1-902560-26-7.
Hybrid-Reality (HR): Reda, K., Febretti, A., Knoll, A., et al. 2013. Visualizing Large, Heterogeneous Data in Hybrid-Reality Environments. Computer
Graphics and Applications, IEEE 33, 4, 38–48.
Indirect Reality (IR): Wither, J., Tsai, Y.-T., and Azuma, R. 2011. Indirect augmented reality. Computers & Graphics 35, 4, 810–822.
Substitutional Reality (SR): K. Suzuki, S. Wakisaka, and N. Fujii, “Substitutional Reality System: A Novel Experimental Platform for Experiencing
Alternative Reality,” Sci. Rep., vol. 2, pp. 1–9, Jun. 2012.
CR-IR
HRAV
The Mixed Reality (MR) Continuum [Milgram99] [Azuma01]
SR

Gamification, Serious Games
• “gamification”: as the use of game design
elements in non-game contexts [Deterding
et al 11]
• Use of game dynamics (e.g. plot), game
mechanics (e.g. rules) and game components
(e.g. points, avatars) in order to engage more
people
• “Serious games”: computer games that
are not limited to the aim of providing
entertainment [Macedonia 2000],
[Macedonia 2002]
• that allow for collaborative use of 3D spaces
that are used for learning and educational
purposes in a number of application domains
[Macedonia 2002]
S. Deterding, D. Dixon, R. Khaled, and L. Nacke,“From game design elements to gamefulness,” presented at the the 15th International Academic
MindTrek Conference,NewYork,NewYork,USA,2011, p. 9.
Macedonia M (2000) Using technology and innovation to simulate daily life.IEEE Computer 33(4):110–112
Macedonia M (2002) Games soldiers play.IEEE Spectrum 39(3):32–37

Presence
• The concept of ‘presence’ refers to the phenomenon of behaving and feeling as if we
are in the virtual world: ‘being there’ created by computer displays [Sanchez-Vives05]
• Another approach is that the sense of ‘being there’ or ‘being someone’ in a VE is
grounded on the ability ‘to do there’ in VR [Sanchez-Vives05] , AR
[Torre00][Egges07]and MR [Wagner09]
• …and many more definitions…
-Torre R, Fua P,Balcisoy S, Ponder M,Thalmann D. Interaction between real and virtual humans: Playing checkers. Proc. Eurographics Workshop OnVirtual
Environments (2000)
- Sanchez-Vives M., and Slater, M.,“From presence to consciousness through virtual reality”. Nature Reviews Neuroscience (2005) vol. 6 (4) pp.pp.332-
339
- Egges,A., Papagiannakis,G.,Magnenat-Thalmann, N.,“Presence and Interaction in Mixed Realities”,The Visual Computer, Springer-Verlag,Volume 23,
Number 5, May 2007
-Wagner I, Broll W, Jacucci G, Kuutii K, McCall R, Morrison A, Schmalstieg D,Terrin J. On the Role of Presence in Mixed Reality. PRESENCE:Teleoperators
and Virtual Environments (2009) vol.18 (4) pp.249-276

The VHD++ framework
§M. Ponder, G. Papagiannakis, T. Molet, N. Magnenat-Thalmann, D. Thalmann. VHD++ Development Framework:
Towards Extendible, Component Based VR/AR Simulation Engine Featuring Advanced Virtual
Character Technologies. IEEE Computer Society Press, Proceedings of Computer Graphics International (CGI), pp. 96-104. 2003
components
•Animation blending
• Idle motion
• skeleton animation
• skin deformation
• cloth simulation
• face simulation
• speech simulation
• AR image blending
• PRT Illumination
• hair animatipn
• Crowd simulation
• Python scripting
• motion capture
• VR devices
• 3D sound
• stereo rendering

MOBILE AUGMENTED
REALITY SYSTEM
Real image for tracking
3D Rendering
VR Runtime Engine
Skeleton Animation
Skin Deformation
VRML97 Loader
XML Loader
Real-time, Marker-less
Camera Tracking
3D Static Models
Virtual Humans
Occluder Models
VR Content Repository
Binocular HMD
Augmented Scene
FireWire WebCam
Local connectivity
Camera Matrix
&
Background
image
Cloth Simulation
Face & Speech Simulation
Sound Rendering
Python Script Engine
sounds
Facial expressions
3D clothes
Python Scripts
Animations
User
Mobile Workstation
Scene Tracking features
Papagiannakis, G., Schertenleib,S., O’Kennedy, B., Arevalo-Poizat, M., Magnenat-Thalmann, N., Stoddart,A.,Thalmann, D.,“Mixing Virtual and Real scenes in the site of
ancient Pompeii”, Computer Animation and VirtualWorlds, p 11-24,Volume 16, Issue 1, John Wiley and SonsLtd, February 2005.

The importance of AR Tracking
• Tracking is the basic enabling technology for Augmented Reality
• Without accurate tracking you can’t generate the merged real-virtual
environment
• Tracking is significantly more difficult in AR than in Virtual Environments
generate
character
VR
track
camera
G. Papagiannakis and N. Magnenat-Thalmann,“Mobile Augmented Heritage:Enabling Human Life in ancient Pompeii,” The International Journal of
Architectural Computing, Multi-Science Publishing, vol.5, no. 2, pp. 395–415, 2007.

MOBILE AR CULTURAL HERITAGE GUIDE: LIFEPLUS EU PROJECT
Papagiannakis,G., Magnenat-Thalmann,N.,“Mobile Augmented Heritage: Enabling Human Life in ancient Pompeii”,International Journal
of Architectural Computing, Multi-Science Publishing,July 2007,issue 02, volume 05, pp.395-415, 2007.
G. Papagiannakis, S.Schertenleib,B. O'Kennedy, M.Arevalo-Poizat,N. Magnenat-Thalmann,A. Stoddart, and D.Thalmann,“MixingVirtual and
Real scenes in the site of ancient Pompeii,” Computer Animation and Virtual Worlds, John Wiley and Sons Ltd,vol. 16, no.1, pp. 11–24, Feb. 2005.

MOBILE AR CULTURAL HERITAGE GUIDE: LIFEPLUS EU PROJECT
G. Papagiannakis, S.Schertenleib,B. O'Kennedy, M.Arevalo-Poizat,N. Magnenat-Thalmann,A. Stoddart, and D.Thalmann,“MixingVirtual and
Real scenes in the site of ancient Pompeii,” Computer Animation and Virtual Worlds, John Wiley and Sons Ltd,vol. 16, no.1, pp. 11–24, Feb. 2005.

Dialogue-based interaction and Presence in AR
A. Egges, G. Papagiannakis, and N. Magnenat-Thalmann, “Presence and interaction in mixed reality environments,” The
Visual Computer, Springer, vol. 23, no. 5, pp. 317–333, May 2007.

Self Adaptive Animation in AR
• Dual Quaternions 1,2:
• Rigid transformation
• Blending
• Linear Blending(DLB)
• Iterative Blending(DIB)
(1+
ε
2
(t0i + t1 j + t2k)).q0
1. Clifford,W.: Mathematical Papers.Macmillan (1882).
2. Kavan, L., Collins,S., Zara, J., O’Sullivan,C.: Skinning with
dual quaternions.In: 2007 ACM SIGGRAPH Symposium on
Interactive 3D Graphics and Games, pp. 39–46.ACM Press
(2007).
Chaudhuri, P., Papagiannakis, G., Magnenat-Thalmann, N., "Self adaptive animation based on user perspective", The Visual
Computer, Springer-Verlag, 24(7-9), pp. 525- 533, July 2008, presented also in CGI08

Our current MR Research Questions
• Does a single mathematical geometric framework exist, to
combine a virtual human's skinning and animation in MR?
• How can we enhance presence with gamification?
• What about modern game engines and 3D platforms?

Back to the drawing board…

glGA
Utilities
glGARigMesh
PlatformWrapper
glGAMesh
glGAHelper Application
Init
Display
Windows Mac Linux IOS
Platforms
ImageMagick
AntTweakBar
GLFW
GLEW
Assimp
GLM
External Libraries
Textures 3D ModelsShader Programs
Resources
SmartBody
“Keep it simple”
character rendering
framework
G. Papagiannakis, P. Papanikolaou,E. Greassidou,and P. E.Trahanias,“glGA: an OpenGL Geometric Application Framework for a Modern,
Shader-based Computer Graphics Curriculum.,” Eurographics’14 Education Track,2014.

Algebras for Computer Graphics (CG)
• Limitations of CG linear algebra
representations
• Points, lines, areas, vector products
• E.g. cross product not defined in 3D
• CG scientists apply several fixes:
• Separate data structures (e.g. quaternions,
dual-quaternions)
• Different combinational programming procedures
(line-sphere-triangle-plane intersections)
• Is there an alternative?

Some history
William Hamilton Hermann Grassmann William Clifford
Chris Doran, Introduction to GA, http://geometry.mrao.cam.ac.uk/home/introduction-to-ga/

Quaternions
Generalises complex
numbers, introduced the
cross product and some
notation still in use today.
Confusion over status
of vectors, but
quaternions are very
powerful for
describing rotations.

Grassmann algebra
German schoolteacher (1809-
1877) who struggled for
recognition in his own lifetime.
Published the Lineale
Ausdehnungslehre in 1844.
Introduced a new, outer
product that is antisymmetric.

Properties of the outer product
The result of the outer
product is a new object: a
BIVECTOR
Bivectors form a linear
space
We can visualise bivector
addition in 3D
a
b
b+c
c
a

Geometric algebra
W.K. Clifford (1845 –
1879) introduced the
geometric product of
two vectors.
The product of two
vectors is the sum of
a scalar and a bivector.
Think of the sum as
like the real and
imaginary parts of a
complex number.

Two dimensions
2D sufficient to understand basic
results. Construct an orthonormal
basis.
Parallel vectors commute.
Orthogonal vectors
anticommute.

The bivector
The unit bivector has negative square.
Follows purely form the axioms of
geometric algebra.
We have not said anything about
complex numbers, or solving polynomial
equations.
We have invented complex numbers!

From dual-quaternions to GA
• In [Kavan et al. 2008] dual quaternions were employed for geometrical
skinning (but not for animation).
• In [Chaudhuri et al. 2008] we employed them for animation blending
• Dual quaternions:
• where qε and q0 are real quaternions and ε is the dual number where ε2 = 0 . This looks
just like Clifford’s biquaternionq + ωr where ω2 = 0 [Clifford1873].
• In our recent geometric algebra (GA) [Papagiannakis 2013] framework we
replaced quaternions with GA:
• Character animationblending
• Drop-in, efficient replacement
of existingquaternionspherical interpolationmethods
• Can this GA framework be employed in modern ARM architectures for
mobile AR?
ˆq = q1,q2,q3,q4( )+ε q01,q02,q03,q04( )= q0 +εqε
[Clifford 1873] CLIFFORD, W.K. 1873. A preliminary sketch of biquaternions. Proc. London Math. Soc. 4:381–395
[Kavan 2008] KAVAN, L., COLLINS, S., ZARA, J., AND CO'SULLIVAN. 2008. Geometric skinning with approximate dual quaternion blending. ACM Transactions on Graphics.
[Chaudhuri 2008] CHAUDHURI, P., PAPAGIANNAKIS, G., AND MAGNENAT-THALMANN, N. 2008. Self Adaptive Animation based on User Perspective. The Visual Computer, Springer-Verlag
24, 7-9, 525–533.
[Papagiannakis 2013] PAPAGIANNAKIS, G. 2013. Geometric algebra rotors for skinned character animation blending. Technical Brief, ACM SIGGRAPH ASIA 2013, Hong Kong, November
2013, 1–6.

Contribution: Main Algorithm for GA animation interpolation
• employment of Euclidean GA rotors
• as fast, drop-in replacements for quaternion algebra, during animation
orientation interpolation
• We provide two different approaches to express quaternions
as GA rotors
• fast interpolate between them using the rotor exponential
formula:
RARn
= RB ⇒ R = e
−I3a
φ
(2n)

• Input: quaternion orientation of existing animation &
skinning frameworks
• Output: express with different ways input and output
quaternions as GA rotors and members of the algebra:
i = e1I3= e2 ^ e3
j = e2I3= −e1^ e3
k = e3I3= e1^ e2
ijk = −1
q =1+i+ j + k
R = ba = b⋅a + b ^ a = cosφ − I sinφ ⇒
R = e−Iφ
⇒ R = e
−I3u
φ
2
Contribution: Main Algorithm for GA animation interpolation II
Papagiannakis,G. 2013. Geometric algebra rotors for skinned character animation blending. Technical Brief,
ACM SIGGRAPH ASIA 2013, Hong Kong, November 2013

GA for character animation on mobile ARM
• Use of GA for encoding
orientations, for mobile AR
devices (ARM architecture)
• The GA rotors e(φ I*α/2n) can
convert any axis α (not only at
the origin) into a rotational
operator.
• Quaternions can be used only on
other quaternions.
• GA rotors are universal operators
capable of rotating other subspaces:
lines, planes, and volumes as first
class operators.
G. Papagiannakis, G. Elissavet,P. E.Trahanias,and M.Tsioumas,“A
Geometric Algebra Animation Method for Mobile Augmented
Reality Simulations in Digital Heritage Sites.,” EuroMed, vol.8740,
no.25, pp. 258–267, 2014.

AR Presence and Gamification
How to augment indoors and outdoors Cultural
Heritage sites with real time Virtual characters
Propose a complete methodology for robust
authoring of AR virtual characters using only modern
mobile devices
Propose an alternative method for smooth rotation of
AR scene: Geometric Algebra (GA) rotors

The Smartbody character animation system
Animation system with a wide range of capabilities
Harness its advanced character behavioral capabilities.
SHAPIRO, A. 2011. Building a character animation system, MIG’ 11,Proceedings of the 4th
International Conference on Motion in Games
Locomotion Object manipulation
Gazing
Speech synthesis and
Lip Sync
Blending Steering
Eye movementsGestures Head movement

A fast and robust pipeline for populating mobile AR
scenes with gamified virtual characters
[Papaefthymiou et al. 2015a] Papaefthymiou, M., Feng, A., Shapiro, A., and Papagiannakis, G. (2015) 'A fast and robust
pipeline for populating mobile AR scenes with gamified virtual characters'. In SIGGRAPH Asia 2015 Mobile Graphics
and Interactive Applications, SA ’15, pages 22:1–22:8, New York, NY, USA.

Life-sized crowd simulation in mobile AR
• Creation of AR environments with real life-sized crowd and group simulation,
on current mobile with cardboard style mobile VR displays [Zikas et al. 2016a]
• Virtual characters can support many real life behaviors like locomotion and
performing gestures
• Extend [Papaefthymiou et al. 2015a] to to include group and crowd
simulation in mobile AR.
[Zikas et al. 2016a] Zikas Paul, Papaefthymiou Margarita, Mpaxlitzanakis Vasilis, and Papagiannakis George. 2016.
Life-sized Group and Crowd simulation in Mobile AR. In Proceedings of the 29th International Conference on
Computer Animation and Social Agents (CASA '16). ACM, New York, NY, USA, 79-82.

A Conformal Geometric Algebra code generator
comparison for Virtual Character Simulation in MR
• Efficient method for robust authoring (rotation) of Augmented reality
scene using Euclidean Geometric Algebra rotors [Papaefthymiou et al.
2016a, Papaefthymiou et al.2015b]
• Propose two fast animation blending methods using GA and CGA and
compare them with:
• Quaternions
• Dual Quaternions
• Compare efficiency of GA code generators:
• Gaigen library
• Versor library
• Gaalop C++ code generator
[Papaefthymiouet al.2015a] Papaefthymiou, M., Papagiannakis, G., Aristidou, A., Ioannides, M., “A conformal geometric algebra framework for mixed
reality and mobile display”, Applied Geometric Algebra in Computer Science and Engineering 2015, AGACSE2015, Barcelona, July 2015
[Papaefthymiouet al. 2016b] Papaefthymiou M., Hildenbrand D. and Papagiannakis G., “A Conformal Geometric Algebra code generator comparison
for Virtual Character Simulation in Mixed Reality”, Advances in Applied Clifford Algebras Journal (accepted), also presented in GACSE workshop, CGI
2016, Heraklion, Greece, June 2016

An inclusive Conformal Geometric Algebra GPU animation
interpolation and deformation algorithm
• An inclusive, single mathematical framework
[Papaefthymiou et al. 2016c] replacing previous linear
algebra, matrices, euler angles, single and dual-
quaternions used for:
• character animationinterpolationand
• skinning (rotation,translationanddilation)
• GA Motor to represent (translationand rotation)
• GA Dilator to represent uniform scaling
• Final transformation is computed by the geometric
product of Motor and Dilator
[Papaefthymiou et al. 2016c] Papaefthymiou M., Hildenbrand D. and Papagiannakis G., “An inclusive Conformal Geometric Algebra GPU animation interpolation
and deformation algorithm”, The Visual computer Journal (accepted), Springer-Nature, also presented in CGI 2016, Heraklion, Greece, June 2016

An inclusive Conformal Geometric Algebra GPU animation
interpolation and deformation algorithm
[Papaefthymiou et al. 2016c] Papaefthymiou M., Hildenbrand D. and Papagiannakis G., “An inclusive Conformal Geometric Algebra GPU animation interpolation
and deformation algorithm”, The Visual computer Journal (accepted), Springer-Nature, also presented in CGI 2016, Heraklion, Greece, June 2016

Mixed Reality Serious Games (MRSGs) for smart
education
• Creation of two MRSGs [Zikas et al. 2016b]
with similar educational content for both VR
and AR technologies in order to compare:
• gamification elements
• learning experience
• gameplay
• Dynamics: are dependent on the used AR or
VR display technology (META AR glasses
or Google cardboard)
• Mechanics and Components: can absorb
minimal display changes and not affect the
game’s core.
[Zikas et al. 2016b] Paul Zikas, Vasileios Bachlitzanakis, Margarita Papaefthymiou, Steve Kateros, Stylianos
Georgiou, Nikos Lydatakis, George Papagiannakis, “Mixed Reality Serious Games for smart education” European
Conference on Games Based Learning, , Paisley, Scotland, October 2016 (submitted)

Summary &
Future Work
• Our work is formulated as a human-centric, technology-fuelled
mobile, mixed reality R&D. Its main idea is summarized as:
• Mixed Reality Character Enabling Technologies:
• Employing Gamification research,
• for immersive, integrated simulation employing geometric algebra
• In the future we aim to employ GA for
• real-time camera tracking
• area light spherical harmonic rotation and rendering

Acknowledgements
Funded by:
• Marie Curie Initial TrainingNetwork, FP7-PEOPLE-2013-ITN,
under grant agreement n° 608013
• Initial Training Network for Digital Cultural Heritage: Projecting
our Past to the Future: ITN-DCH
• Marie-Curie Intra-EuropeanFellowshipFP7-PEOPLE-2010-
IEF, under grant agreement n° 274669
• “hifi-PRINTER”: High fidelity Presence and Interaction:
convergence of computer graphics, vision and robotics for
improving human-robot and human-computer interaction
• nVIDIA Academic Partnershiph/w grant 2012, 2016

and one last thing…

http://www.ics.forth.gr/CGI2016

Sponsored by
Thank you for your
attention!
Special thanks to:
Prof. Nadia Magnenat-Thalmann
Prof. Daniel Thalmann
CASA’16 conference committee
M. Papaefthymiou, S. Kateros, S. Georgiou, N. Lydatakis, P. Zikas, V. Mpahlitzanakis
FORTH & University of Crete
MIRALab

Gamification with virtual characters at the borders of mixed realities and algebras

More Related Content

What's hot (12)

Viewers also liked (20)

Similar to Gamification with virtual characters at the borders of mixed realities and algebras (20)

Recently uploaded (20)

Gamification with virtual characters at the borders of mixed realities and algebras