Agence de Doublures Numériques : the Digital Doubles Agency

ADN Agence de Doublures Numériques Creation, Management, Exploitation of Digital Doubles www.adnda.com

Facial modeling and animation The human face : « a technical and artistic challenge » First revealing the nature and personality of the considered character, Extreme anatomic, morphologic and physiologic complexity , Special human ability and sensitivity in the perception and interpretation of facial images

Constant increasing use in the entertainment industry : Anonymous CG character in interactive applications: - Makes identification and personification easier, - Natural and familiar communication scheme, - Makes interaction positive and efficient. VIPs reproduction in movie productions: - Conveys the charisma & fame of celebrities for narrative, artistic, logistic, economic,… reasons Fields of application

CG character in SFX feature films A common practice for figurative roles and crowd simulation purposes: Establishing shots << ¼ screen Anonymous character

Digital Doubles for the cinema and advertising Commonly accepted for stunt actions: Framing/notoriety drives new usages evolution:

CG characters in video-games Sports and IPs based on prominent persons: Quality increase on trailers & in-games cinematics:

In the present production mode, the indistinguishable CG reproduction of identifiable personality is mostly the prerogative of major studios while it carries the most significant potential Photo-realism / Fidelity Fidelity Realism Virtual Representation Reproduction of the reality Non Photo-Realistic rendering Common N.A. Photo-realistic rendering Emergent Current stake ! Framing Character Large Close-up Anonymous Walk-on player N.A. Identifiable Secondary character Main actor !

Standard production workflow These representations are mostly processed during the film post-production / video-game’s development: - on a case by case basis - with significant costs and time constraints over the modeling and set-up - variable quality and usability - dependence towards technical providers - without the control of the actor on his representation - without legal grants over these data and results

Agence de Doublures Numériques ADN project: let’s try to move the camera closer identification/classification of actual use-cases & requirements methodological, technical, legal stakes & problematics linked to the photo-realistic and accurate CG reproduction of the appearance and performance of an actor ADN introduces a new process to: control the creation, management and exploitation of give independent productions and studios access to Digital Doubles of prominent persons

Mission Natural extension of the traditional principle and framework of « doubles » applied to CG representation Objectives: supervise and optimize the production share costs related to the modeling/set-up establish a legal framework and jurisprudence ensure the control of the actor over his representation guaranty the photo-realism certify the results conformity insure exploitability of the studios

Process Restructuration of production activities into 3 stages: Creation:  creation of the Digital Double Implementation:  implementation of the Digital Double Integration:  integration of the Digital Double

Stages Creation Implementation Integration Who ADN ADN VFX / Video-game studios When On the first production For the making of a sequence During the cinematics production What Morphology, Reflectance, Expressions Performance capture and retargeting of the acting (potential adaptation to the character appearance) Lighting & Rendering Legal Agreement with the celebrity Contract with the executive production deriving from the actor’s commitments No impact on Studios - Producers contractual relations References Reference image/video of the actor Reference image/video of the character Global illumination conditions of the scene Results Digital Double of the actor ready for animation/lighting Adapted & Animated Digital Double of the character ready for render/compo Final images of the sequence

Services Creation:  Costs sharing of the Digital Double creation Implementation:  Adaptation of the appearance  Performance capture/retargetting Integration: (  Supervision of the integration)  Models management and maintenance   Legal and moral framework governing   Administration of portfolio exploitation 

Productions assets Actor: image rights governing conformity of his appearance on the screen control over the digital double performance availability (free-time, additional commitments, notoriety….) Talent manager: appropriation of a technological evolution Production studio: compliancy with standard animation and rendering techniques technical garanties on the photo-realism and conformity Producer / Editor: costs/time sharing on the modeling/set-up legal garanties on the results exploitation

Technical requirements To be implemented, this approach requires the model to be: Sustainable: i.e. not tailored for a single use but built to last Versatile in terms of appearance: i.e. the appearance can be adapted upon the production requests ( mock-up, aging, weighting, alteration, … ) Generic in terms of implementation: i.e. fit to respond to various animation and lighting conditions Exploitable: i.e. results integration follows traditional tools and technics (granularity, nature, format of the interfaces)

Technical approach Analysis/Synthesis approach founded on: Measures acquisition/selection, Data analysis/reconstruction Separation between Appearance and Performance geometry / material / deformation signal processing Positioned as technical integrator: Data acquisitions are externalized to benefit from technical progress and market trends Technological asset: - Data/Model registration and information transfer/retargeting - Characterization of adapted search and parametrization spaces Concerns: - Error estimation on data which directly/indirectly impact the results - Interoperability with traditional animation/rendering approaches

Creation of the Digital Double Production of a model of the actor ready to be animated and rendered under any conditions: Appearance and expressions space characterization based on morphologic, reflectance, expressions acquisition/analysis/reconstruction Qualitative & Quantitative validation of the conformity The creation and validation of the Digital Double follows an agreement with the celebrity. This contract can be settled independently of any production since this model is to be shared among/between different titles.

Structured light acquisition Static scan of the neutral face

Dynamic scan of the facial ROM: - 90 seconds: emotional expressions and locution facial gym - Statistical de-correlation close to muscular decomposition - Set-up of 103 FAUs Structured light acquisition

Detailed geometry and reflectance fields acquisition: - Polarized shots - Unpolarized shots under 4 lighting conditions Monocular camera - 5 structured light patterns Binocular cameras Polarized gradient light acquisition

Resulting HD Maps ( before conformation upon the model’s UV): Polarized gradient light acquisition Diffuse Specular Displacment Spec. Normal Diff. Normal R Diff. Normal G Diff. Normal B

SSS diffusion radius estimation: Polarized gradient light acquisition Detailed geometry:

Neutral Surfacing & Shading Diffuse Specular Displacement + Normal*Bump + HDRI lighting Hybrid normal shader / Diffuse bent normal + specular / SSS simulation + specular

Stretched/Relaxed extreme expressions: Dynamic Surfacing & Shading uncompressed maps compressed maps bump displace bump displace

Deformation set-up Deformation & Shading Set-up Shading set-up FAUs influence mask

Eyes, eyelids (modeling/shading) : Complementary elements

Inner mouth, teeth, tongue (modeling/shading) : Complementary elements

Implementation of the Digital Double Reproduction of the appearance and performance of the character considered on the different sequences: Performance capture and retargeting of the acting (potential adaptation to the character appearance) Qualitative & Quantitative validation of the conformity These services are contracted by the executive production in agreement with the actor’s commitment. Each production is only charged for its specific use of the Digital Double.

Performance capture Characterization of the pose in the expression space drives the animation of the considered appearance Motion characterization, editing and retargeting: This scheme is also considered to assess the acting’s conformity

Integration of the Digital Double The integration of the animated Digital Double for the realization of the final shots follows standard lighting/rendering techniques ADN delivers the animated Digital Double to the production studio contracted by the producer This process has no impact on Studios-Producers contractual relations. In any case, the use of the Digital Double remains limited to the program contracted with the celebrity.

Conclusions & Perspectives The treatment of this problematic highly benefits from: - Progress and democratization of data acquisitions systems - Methods and technics issued from computer vision Still, Data/Model registration and legal framework definition are the most significant barriers we tackled Research & development towards in-game derivation: - declension of Digital Doubles for real-time game engine - coupling visual+audio reproduction of actors while maintaining personality’s control over animation possibilities Domain, technical and media convergence lead to extend this approach to: - other celebrities: athletes, singers, fashion models, peoples… - other sectors: internet applications, live TV broadcast, events…

Conclusions & Perspectives Technological evolution and markets trends will rapidly permit to develop flexible approaches to capture and reproduce visual representations of real world objects, persons, scenes, events hardly distinguishable from the image reality … We better get ready to manage this evolution …

Question / Answer Agence de Doublures Numériques Digital Doubles Agency www.adnda.com [email_address] [email_address]

Agence de Doublures Numériques : the Digital Doubles Agency

More Related Content

Similar to Agence de Doublures Numériques : the Digital Doubles Agency (20)

Recently uploaded (20)

Agence de Doublures Numériques : the Digital Doubles Agency