SlideShare a Scribd company logo

Tutorial MPEG 3D Graphics

Marius Preda PhD, profile picture
Marius Preda PhD

The document discusses the need for standardized formats for 3D synthetic content to improve interoperability across diverse multimedia platforms, likening it to existing standards for audio and video. It highlights the complexity of 3D data representation, including various geometry types, compression technologies, and encoding techniques necessary for efficient representation and transmission. The text also outlines the evolution of MPEG standards and technologies designed to support 3D graphics applications, focusing on compression strategies, quality of service, and encoding interoperability.

TechnologyEntertainment & Humor
1 of 133
Downloaded 122 times
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
Web3D 2011, June, 20, Paris Marius Preda, PhD Chairman of MPEG 3DG Institut TELECOM
  It’s a crazy multimedia world! –  Network is everywhere… but very heterogeneous –  Terminals are “same same”… but different! –  Content must be designed with a priori knowledge of its future use –  Applications are platform-centric instead of user-centric   Fragmented value chain Production Transmission Consumption Capturing HW/SW Playback Networks devices providers devices INTEROPERABILITY NEEDED!!! Authoring IPR Terminal Providers tools holders manufacturers Service End users scenarios
  No common representation –  Heterogeneous kind of data •  Different types of geometry, appearance and animation models –  Always easier to specify a new data representation format than learning an existent one   Very different application domains
Tutorial MPEG 3D Graphics
Tutorial MPEG 3D Graphics
FAST TRANSPORT TROUGHT THE NETWORK Size is of main Size doesn’t matter Size doesn’t matter importance
FAST TRANSPORT TROUGHT THE NETWORK Size doesn’t matter Size doesn’t matter Size is of main importance
To define a standard format for compressed 3D synthetic content. In other words to be for graphics what MP3 and AAC are for audio, MPEG-2 and MPEG-4 are for video and JPEG is for still images. Additionally "MPEG 3D Graphics" aims at providing mechanisms such as APIs to enable easy integration and development of applications using its standard representation tools.
MPEG 3D Graphics within the MPEG standards family Core technologies in MPEG 3D Graphics Compressing other standards (COLLADA, X3D, …) with MPEG Virtual Worlds Interoperability for avatars with MPEG
MPEG 3D Graphics within the MPEG standards family Core technologies in MPEG 3D Graphics Compressing other standards (COLLADA, X3D, …) with MPEG Virtual Worlds Interoperability for avatars with MPEG
MPEG has several names •  Common: MPEG = Moving Picture Experts Group •  Official: MPEG = ISO/IEC JTC1/SC29/WG11 ISO Over 224 technical committees JTC1 Joint Technical Committee with IEC SC24 SC29 Computer graphics and image processing Coding of audio, picture, multimedia and hypermedia information GKS, PHIGS WG1 JPEG CGM, VRML, X3D Coding of still pictures WG11 MPEG Coding of moving pictures and audio
MPEG has several subgroups Video ... and produced several Requirements Audio Systems successful standards 3D Graphics MPEG MPEG-1 MPEG-2 MPEG-4 MPEG-7 MPEG-21 ISO/IEC 11172 ’92 ISO/IEC 13818 ’94 ISO/IEC 14496 ’99 ISO/IEC 15938 ’01 ISO/IEC 21000 ’02 Video & audio Video & audio Multimedia Metadata Terminal (CD-ROM) (DVD & DVB) & interactive (description & network applications of content) specification MPEG-A MPEG-B MPEG-C MPEG-D MPEG-E MPEG-M MPEG-U MPEG-V
MPEG-4 Features MPEG-4 Objects Naturals – still images, audio, 2D/3D video Synthetic - audio, 2D/3D objects and scenes Compression Compression MPEG-4 Objectives Compression Provide technologies for efficient compression and transmission MPEG-4 Terminal Composition, at end-user side, of natural and synthetic objects, into hybrid and interactive scenes
MPEG-4 Features System architecture Interactive Scene Description Scene layer Object descriptor layer MPEG-4 stream Media data layer
MPEG-4 3D chronology
MPEG-4 3D chronology
MPEG-4 3D chronology
MPEG 3D Graphics within the MPEG standards family Core technologies in MPEG 3D Graphics Compressing other standards (COLLADA, X3D, …) with MPEG Virtual Worlds Interoperability for avatars with MPEG
  IFS surfaces
Approximation of target surface •  Method: tesselation with planar facets •  Quality: first order (linear) ⇒ no smoothness (C0 continuity) Mesh definition: IFS (Indexed Face Set) •  Connectivity: list of faces {…, Pn = {in0, in1, in2, …}, …} ⇒ arbitrary topology (non-manifold, open, higher genus, etc.) •  Geometry: list of vertices {…, Vk = (xk, yk, zk), …} Mesh coding •  Connectivity (lossless): triangle strips, triangle+vertex trees, etc. •  Geometry (lossy): coordinate quantisation + prediction from conn.
LOD concept •  1976: Clark introduced the idea •  Main interest: rendering efficiency Taxonomy of LOD extraction techniques •  Static vs. dynamic •  Global vs. local •  Progressive vs. hierarchical LODs Successful simplification techniques •  1996: Hoppe’s edge collapses •  1997: Garland’s quadrics ⇒ qslim Progressive 3D mesh coding •  1996: Hoppe’s PM (Progressive Mesh) •  1998: IBM’s PFS (Progressive Forest Split)
Triangles: 69451 vs. 29519
Triangles: 69451 vs. 9575
Triangles: 69451 vs. 1627
Triangles: 69451 vs. 238
MPEG-4 (1999): IFSs •  Based on VRML97 •  Arbitrary topology meshes •  “Properties” (normals, colours and textures) MPEG-4 Amd.1 (2000): 3DMC (3D Mesh Coding) •  40-50:1 compression of IFSs by IBM’s TS (Topological Surgery) •  Incremental transmission and rendering •  Progressive coding by IBM’s PFS •  Error resilience by SAIT
MPEG-4 (1999): IFSs •  Based on VRML97 •  Arbitrary topology meshes •  “Properties” (normals, colours and textures) MPEG-4 Amd.1 (2000): 3DMC (3D Mesh Coding) •  40-50:1 compression of IFSs by IBM’s TS (Topological Surgery) •  Incremental transmission and rendering •  Progressive coding by IBM’s PFS •  Error resilience by SAIT
  IFS surfaces
  IFS surfaces   Patches
Approximation of target surface •  Method: tesselation with predefined curved patches •  Quality: higher order (polynomic/rational) ⇒ Cn continuity Mesh definition •  Connectivity: regular grid of quads. or triangles ⇒ planar topology •  Geometry: list of control points {…, Pk = (xk, yk, zk), …} Mesh coding •  Connectivity (lossless): implicit •  Geometry (lossy): coordinate quantisation + prediction from conn.
Tensor product of cubic Bézier curves Single patch (4x4 control points) vs. two patches (4x7 control points) Compression  (3547 polygons; 1215 vertices) vs. (86 patches; 212 control points)
MPEG-4 Part 16 (2003): NURBS •  Based on VRML97 Amd., originally proposed by blaxxun •  Support for NURBS curves and patches   Specific nodes for Bézier’s curves and patches (for increased efficiency) •  Support for free-form deformations
  IFS surfaces   Patches   Subdivision surfaces
SS = limit of recursive refinement of base control mesh NB: refinement affects both Geometry smoothing achieved with connectivity (of abstract graph) and stencils particular to each scheme geometry (of 3D mapping) 1/16 1/16 ½ 1/16 9/16 9/16 1/16 1/8 1/8 Border/sharp vs. z interior edge stencils ½ of “butterfly” scheme x y 1/16 1/16 SS inherently define hierarchically nested LODs
Approximation of target surface •  Method: tesselation with curved patches •  Quality: higher order ⇒ Cn continuity Mesh definition •  Connectivity: list of triangles/quads., e.g., {…, Tn = {in0, in1, in2}, …} ⇒ arbitrary (manifold) topology •  Geometry: list of control points {…, Pk = (xk, yk, zk), …} Mesh coding •  Connectivity (lossless): as for polygonal (manifold) mesh •  Geometry (lossy): as for polygonal (manifold) mesh
Polygons + Are the simplest approach (linear approximation) + Can resolve fine details and handle arbitrary topologies – Lead to unstructured, huge meshes Patches + Are a more powerful approach (higher order approximation) + Are convenient for coarse and smooth models – Need cumbersome trimming and stitching mechanisms SSs + Connect and unify the two extremes above ++ Provide multi-resolution handles for hierarchical coding/editing
Catmull-Clark’s (1978): quadrilateral, primal, approximating, C2 Loop’s (1987): triangular, primal, approximating, C2 Dyn++’s “butterfly” (1990): triangular, primal, interpolating, C1
  IFS surfaces   Patches   Subdivision surfaces   Wavelet subdivision surfaces
Target surface Base mesh Subdivide Add details
Target surface Base mesh Subdivide Add details Price (requirements) •  Base mesh extraction •  Subdivision scheme = predictor •  Details (3D vectors) = prediction errors ⇒ remeshing Prize (advantages) •  Predictive coding ⇒ immediate (if smooth target mesh)
Spatial partitioning Adding details in appearing parts Removing details in disappearing parts ZT ZT ZT … ZT
by courtesy of FranceTelecom
MPEG-4 Part 16 (2003): “plain” + wavelet SSs •  “Plain” SSs for mesh smoothing   Considered schemes: Catmull-Clark, [extended] Loop, butterfly   No details are added but…   normal control achievable through edge/vertex tagging of initial control mesh •  Wavelet/detailed SSs for surface approximation   Possibly tagged base mesh   Details are added after each subdivision step, which are…   wavelet-transformed according to one of several possible schemes   Most suited for multi-resolution editing/animation   Most suited for view-dependent transmission
  IFS surfaces   Patches   Subdivision surfaces   Wavelet subdivision surfaces   Mesh Grid
CW (Connectivity-Wireframe) RG (Reference-Grid)
Vertex offset is a relative value Update vertex position when grid is deformed or animated
Tutorial MPEG 3D Graphics
  IFS surfaces   Patches   Subdivision surfaces   Wavelet subdivision surfaces   Mesh Grid   Solids
Solid primitives Solid models: the “arithmetic of forms”
Implicit equation: Quadrics (2nd order): Quartics (4th order):
A cube = multiplication of 3 degenerated quadrics Multiplication of two forms Examples of solid operations
Architecture Mechanics Biotechnology Virtual models
Exact geometry Compactness 21 Kb 37 Kb 407 Kb 1.1 Mb
  IFS surfaces   Patches   Subdivision surf.   Wavelet SS   Mesh Grid   Solids   SC-3DMC
MPEG 3DGC Scalable Complexity 3D Mesh Compression •  Not all 3DG applications have the same needs in compression •  Not all the 3DG applications can afford spending extra CPU/GPU for compression TFAN SVA SC3DMC QBCR Same Quantization and Binarization blocks Towards the continuum model: enlarge the application domain where MPEG-4 3DG can be used
MPEG 3DGC Scalable Complexity 3D Mesh Compression x, y, z (floats) + normals (floats) x, y, z (floats) + colors (floats) x, y, z (floats) + … (floats) i, j, k (integers) Attributes i, j, k (integers) i, j, k (integers) Connectivity
SC-3DMC General Schema Connectivity TFAN BP CABAC Delta (lossless) SVA FLB AC BAC Connectivity Prediction Binarization Entropy i, j, k Analysis Encoding Attributes Prediction Entropy Binarization (lossy) Encoding Parallelogram Delta BP CABAC x, y, z Quantization Barycentre FLB AC 5 different paths with different performances
SC-3DMC Connectivity Analysis: Empty Do Nothing 3 1 4 2 123234
SC-3DMC Connectivity Analysis: SVA Previous Face 4 modes for encoding how consecutive Current Face faces share vertices Shared Vertex 1 5 3 1, 2, 3, 1, 2, 3, 1 4 3 2, 3, 4 3, 4, 5 12304 2 2 4 123245 Mode 0 Mode 2 1 4 1 1 1, 2, 3, 1, 2, 3, 4, 5, 6 1, 2, 3 2 3 2 3 2 3 5 6 OR 1231 5 Mode 1 Mode 3 1233
SC-3DMC Connectivity Analysis: TFAN Split the mesh in set of triangle fans and encode each fan 1675 62937 2984 Transformed in local indices
SC-3DMC What we measure? Encoding performances but also complexity for decoder (and encoder) TFAN Performances SVA-AC 3DMC SVA-BAC SVA-BP NCA BIFS Complexity
  IFS surfaces   Patches   Subdivision surf.   Wavelet SS   Mesh Grid   Solids   SC-3DMC
•  Visual Texture Coding Generic compression tool •  Synthesized Textures Very Compact representation for •  Procedural Textures specific applications •  Depth Image-based Representation Less calculation intensive
•  MPEG-4 3DMC •  MPEG-4 MeshGrid •  MPEG-4 Subdivision Surfaces 3D Mesh •  MPEG-4 VTC •  JPEG 2000 3D object 2D Texture
Zero-Tree WT 3D mipmaps
Multiple In Place mapping = multi-resolution textures Without mipmapping With mipmapping
Zero-Tree WT Computational Graceful Degradation: Quality => processing power
Region of Interest with resolution/quality selection Visual importance depends on viewing angle Low Resolution High Resolution
Packet selection by using error-resilience markers
  IFS surfaces   VTC   Patches   Synthesized texture   Subdivision surf.   Wavelet SS   Mesh Grid   Solids   SC-3DMC
Vector Graphics Primitives Color profile Line Color Profile ( LC ) Area Color Point ( A C ) Line ( LN ), bounded by 2 Terminal Points ( TP ) The image cannot be displayed. Color Patch ( PA ) Line Segment ( LS ), bounded by 2 Line Points ( LP ) Sub -Texture ( ST ) Line marked as control line (Skeleton) The image cannot be displayed.
65x96 pixels 10 seconds animation 1.35 kB
  IFS surfaces   VTC   Patches   Synthesized texture   Subdivision surf.   Procedural texture   Wavelet SS   Mesh Grid   Solids   SC-3DMC
DEF Fabric ProceduralTexture { type 2 width 256 height 256 cellWidth 4 cellHeight 4 roughness 1 distortion 0.05 seed 114300 color [ 0.898 0.89418 0.95294, 0.34118 0.29418 0.70196, 0 0 0, 0 0 0 ] aWarpmap [ 0 0, 0.03 1, 0.88 1, 1 0 ] bWarpmap [ 0 0, 0.48 1, 1 0 ] aWeights [ 0, 0, 0, 0, 0, 0, 0.56, 0, 0, 0, 0, 0, 0, 0, 0.20, 0.24 ] bWeights [ 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0 ] } DEF Marble ProceduralTexture { width 256 height 256 roughness 1 seed 22209 color [ 0.8 0.7098 0.6902, 0.95686 0.8902 0.87451, 0.87451 0.37255 0.23529, 0.95686 0.8902 0.87451 ] aWarpmap [ 0 1, 0.33 0, 1 1 ] bWarpmap [ 0 0, 0.55 0, 0.6 1, 0.65 0, 1 0 ] bWeights [ 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0 ] }
  IFS surfaces   VTC   Patches   Synthesized texture   Subdivision surf.   Procedural texture   Wavelet SS   DIBR   Mesh Grid   Solids   SC-3DMC
Reconstruct 3D representation from projections Depth image Projection
  IFS surfaces   VTC   Patches   Synthesized texture   Subdivision surf.   Procedural texture   Wavelet SS   DIBR   Mesh Grid   Point Texture   Solids   SC-3DMC
  IFS surfaces   VTC   Interpolators   Patches   Synthesized texture   Subdivision surf.   Procedural texture   Wavelet SS   DIBR   Mesh Grid   Point Texture   Solids   SC-3DMC
Piece-wise linear functions defined by a set of pairs time-value Standardized by MPEG-4 for: - position - orientation - scale - coordinate in IFS - normals in IFS Straightforward animation based on key-frames
Large amount of data for high quality, smooth animation
Re-sampling, sub-sampling: two methods supported by MPEG-4 Path preserving mode Key preserving mode Three schemes supported by MPEG-4 Orientation Interpolators Coordinate Interpolators Position Interpolators Exploit temporal redundancy for the most common interpolators
Re-sampling, sub-sampling: two methods supported by MPEG-4 Key value Key value Path preserving mode Key(time) Key(time) Key preserving mode N key values N-1 key values Three schemes supported by MPEG-4 Key value Key value Orientation Interpolators Coordinate Interpolators Key(time) Key(time) Position Interpolators N-2 key values N-3 key values Sub-sampling or re-sampling based on minimal distortion
A dedicated elementary stream for IC, multiplexed into an AFX stream
  IFS surfaces   VTC   Interpolators   Patches   Synthesized texture   Bone-Based   Subdivision surf.   Procedural texture   Wavelet SS   DIBR   Mesh Grid   Point Texture   Solids   SC-3DMC
  Face, MPEG-4 V1, 1999   Body, MPEG-4 Amd1, 2000   Skinned Model, MPEG-4 Part 16, 2003 Two frameworks: human-like (FBA) and generic skeleton (BBA)
  Geometry: Seamless mesh: shapes sharing the same vertices list   Texture: Image Mapping on vertices sub-set   Hierarchy: Skeleton layer Muscle layer Seamless mesh affected by a hierarchical skeleton
  1D controllers: bone & muscle   for each bone and each muscle - a list of affected vertices - a measure of affectedness are provided Right balance between control parameters and influence volume
Uncompressed Uncompressed Segment #n Frame #n BAP/FAP/BBA BAP/FAP/BBA Frame P DCT Prediction DC Coeff. AC Coeff. Frame I Segment P Segment Prediction I Quantization Quantization DC Q DC Q AC Q Arithmetic Arithmetic coding coding Huffman coding Huffman coding Binary file Binary file Basic comp.: prediction, freq.transform, quantization and entropy encoding
  Face, MP4 V1, 2kbps   Body, MP4 Amd1, 5-30 kbps   Skinned Model, MP4 Part 16, 5-30 kbps for a human like skeleton Very low bit-rate
  IFS surfaces   VTC   Interpolators   Patches   Synthesized texture   Bone-Based   Subdivision surf.   Procedural texture   Morphing   Wavelet SS   DIBR   Mesh Grid   Point Texture   Solids   SC-3DMC
  Defined as a base mesh and a collection of target meshes   Animation obtained by updating the weights of the target meshes   BBA stream updated to include morph data   Usable for any kind of 3D object Local and precise control for shape deformation
  IFS surfaces   VTC   Interpolators   Patches   Synthesized texture   Bone-Based   Subdivision surf.   Procedural texture   Morphing   Wavelet SS   DIBR   FAMC   Mesh Grid   Point Texture   Solids   SC-3DMC
  Cluster the vertices with respect to their motion   Encode a cluster motion by an affine transform   Encode the residual error at vertex level by traditional approach (DCT/W, quantization, entropy encoding)
  IFS surfaces   VTC   Interpolators   Patches   Synthesized texture   Bone-Based   Subdivision surf.   Procedural texture   Morphing   Wavelet SS   DIBR   FAMC   Remote and   Mesh Grid   Point Texture programmatic   Solids   SC-3DMC
Bifs-Anim   Remote animation BIFS-Anim & BIFS Command Bifs-Command scene   Programmatic animation Script Node - ECMA Script scene part of the scene description - Java Code standardized API for accessing MPEG-J Stream the scene graph scene Complex application scenario can be built
  IFS surfaces   VTC   Interpolators   Patches   Synthesized texture   Bone-Based   Subdivision surf.   Procedural texture   Morphing   Wavelet SS   DIBR   FAMC   Mesh Grid   Remote and   Point Texture programmatic   Solids   SC-3DMC
MPEG 3DG How measuring compression performances? On-line benchmarking platform: www.MyMultimediaWorld.com        
MPEG 3DG How measuring compression performances? On-line benchmarking : www.MyMultimediaWorld.com 3D compression Web site benchmark Benchmark 2 Benchmark 3 Filter and Presentation Engine Benchmark manager 2 Benchmark manager 1 Indexed MDB 3D Compression benchmark manager API MP4 Extended MP7 Algorithm 1 Algorithm 2 … Algorithm n
MPEG 3DG How measuring compression performances? On-line benchmarking : www.MyMultimediaWorld.com GetNbBitstream n Vertex Buffer Encoder Proprietary Coder library DumpBitSream BitStream (1,n) DumpCompressedVB Proprietary Decoder Decoder library .Compressed_VB (1,n)
MPEG 3DG How measuring compression performances? Benchmarking platform, per-object visualization - object properties (number of vertices/ triangles, number of components and files size, - distortion graph (linear or logarithmic), - compression gain with respect to 3DMC. -  encoding time -  decoding times.
MPEG 3DG How measuring compression performances? Benchmarking platform, global visualization Filters - semantic category, - average distortion, - number of vertices, - number of connected components in the object, - database subset.
MPEG 3DG How measuring compression performances? Benchmarking platform, global visualization
MPEG 3D Graphics within the MPEG standards family Core technologies in MPEG 3D Graphics Compressing other standards (COLLADA, X3D, …) with MPEG Virtual Worlds Interoperability for avatars with MPEG
Tutorial MPEG 3D Graphics
Tutorial MPEG 3D Graphics
Geometry compression: -compression ratio 40:1 Animation compression: -compression ratio 100:1
Any XML Scene Representation XML XML (Binary) XML Binarisation 3D Graphics Compression
Layer 1: Any XML Scene Representation XML Textual XML representation of the XML scene (Binary) XML Binarisation 3D Graphics Compression
Layer 2: Any XML Scene Representation XML Binarized XML layer XML Contains the (Binary) XML Binarisation unclassified elements of the scene graph 3D Graphics Compression
Layer 3: Any XML Scene Representation XML Compressed layer XML Contains specific (Binary) XML Binarisation types of media information 3D Graphics (geometry, Compression animation, ...)
Result: Any XML Scene Representation XML Multiplexed XML layers 2 and 3 (Binary) XML Binarisation 3D Graphics Compression
MPEG-4 P25 Encoder side: possible implementation (informative) Encoder Side MP4 XMT, COLLADA, X3D Comp ratio: 40-70:1
MPEG-4 P25 Decoder side (normative) Decoder Side MP4 XMT, COLLADA, X3D Lossless for the data structure, lossless or lossy for graphics primitives
MPEG 3D Graphics within the MPEG standards family Core technologies in MPEG 3D Graphics Compressing other standards (COLLADA, X3D, …) with MPEG Virtual Worlds Interoperability for avatars with MPEG
Standards for Avatars as visualization support Standard Generic Features Avatar representation COLLADA 3D objects/ scenes Generic Object VRML/ X3D 3D objects /scenes H- Anim Application behavior MPEG-4 2D/3D objects/ scenes A set of dedicated nodes Application behavior A dedicated compressed stream Compression
Standards for Avatars as interaction support Representation Features HumanML human physical description, emotion EmotionML emotion, facial expressions, gestures BML speech, gesture, gaze MPML speech VHML facial and body animation, emotional representation CML character attribute and animation definition
Why none of the existing standards is solving the issue of avatar interoperability in Virtual Worlds? –  The Virtual Worlds are proprietary applications and the 3D assets including the avatars have economical value –  The Virtual Worlds and in general 3D applications have specific data format allowing rendering optimization However, while maintaining a strict control for economic and technical reasons, Virtual Worlds allow users to personalize the avatars
Avatar Template Attributes that can be modified by the user Specify the set of Personalization Parameters (PP) that transforms a template of an arbitrary Virtual World into the user designed avatar
What avatar feature can be personalized? Mainly the appearance Analysis of SecondLife, IMVU, Entropia Universe, SonyPlaystation and HumanML
Very heterogeneous set of personalization parameters Entropia Nintendo Universe Wii HumanML Second Life PlayStation
“Appearance” element <Appearance> <Body> <BodyHeight value=165/> <BodyFat value=15/> </Body> <Head> <HeadShape value="oval"/> <EggHead value="true"/> </Head> </Appearance>
Interoperability at the Animation level <Animation> <Greeting> <Salute>salut</Salute> <Cheer>cheer</Cheer> </Greeting> <Fighting> <shoot>pousse</shoot> <throw>throw</throw> </Fighting> </Animation>
Motion retargeting No “Walk” Avatar Avatar animation “Walk” animation Template in VW1 Template in VW2 defined in VW2 In VW1
“Control” element Body Face Control Control <Control> <BodyFeaturesControl > <UpperBodyBones> <LCalvicle>my_LCalvicle</LCalvicle> Control <RClavicle>my_RCalvicle</RClavicle> </UpperBodyBones> </BodyFeaturesControl> <FaceFeaturesControl> <HeadOutline> <Left X=0.23 Y=1.25 Z=7.26/> <Right X=0.25 Y=1.25 Z=7.21/> </HeadOutline> </FaceFeaturesControl> </Control>
MPEG-V (Personalization Parameters) MPEG-V + MPEG-4 The Avatar in The Avatar in VW1 VW2 The Avatar in an external player
MPEG 3DGC Ongoing work
MPEG 3DGC Multi-Resolution 3DMC Current 3DG content in VW © Samsung © Second Life Tomorrow’s 3DG content in VW (100 times denser) © OTOY © OTOY
MPEG 3DGC Reconfigurable Graphics Coding A framework that allows to set up the decoder at run time
Thank you! www.MyMultimediaWorld.com www-artemis.it-sudparis.eu marius.preda@it-sudparis.eu
MPEG 3DGC Multi-resolution 3DMC Progressive mesh [Hoppe’96] Progressive Forest Split (PFS) [Taubin’98] Patch coloring [Cohen-Or’99] Valence-based decimation approach [Alliez’01] Octree-based compression [Peng’05] Spectral coding [Karni’01]
MPEG 3DGC Multi-resolution 3DMC : two proposed methods 1. Progressive TFAN Original Interpolated with 1% of the vertices
MPEG 3DGC Multi-resolution 3DMC : two proposed methods 2. KLT encoder 2 2 2 1 1 1
MPEG 3DGC Multi-resolution 3DMC : two proposed methods 1. Work on -  document the use case scenarios (object examination, navigation, …) -  database -  comparison with other codecs -  comparison method (PSNR, Resolution) -  platform for testing 2. You are invited to contribute

More Related Content

PPTX
MPEG 3D graphics compression offer
Marius Preda PhD
 
PDF
Point Cloud Compression in MPEG
Marius Preda PhD
 
PPTX
MPEG Augmented Reality Tutorial
Marius Preda PhD
 
PDF
Basics of Mpeg 4 3D Graphics Compression
Marius Preda PhD
 
PPTX
Mpeg ARAF tutorial @ ISMAR 2014
Marius Preda PhD
 
PPTX
3DgraphicsAndAR
Marius Preda PhD
 
PPTX
XLcloud 3-d remote rendering
Marius Preda PhD
 
PPTX
Augmented Reality: Connecting physical and digital worlds
Marius Preda PhD
 
MPEG 3D graphics compression offer
Marius Preda PhD
 
Point Cloud Compression in MPEG
Marius Preda PhD
 
MPEG Augmented Reality Tutorial
Marius Preda PhD
 
Basics of Mpeg 4 3D Graphics Compression
Marius Preda PhD
 
Mpeg ARAF tutorial @ ISMAR 2014
Marius Preda PhD
 
3DgraphicsAndAR
Marius Preda PhD
 
XLcloud 3-d remote rendering
Marius Preda PhD
 
Augmented Reality: Connecting physical and digital worlds
Marius Preda PhD
 

What's hot (20)

PPTX
Tutorial on Point Cloud Compression and standardisation
Rufael Mekuria
 
PPTX
Bridging the gap between web and television
Marius Preda PhD
 
PPTX
MPEG Technologies and roadmap for Augmented Reality
Marius Preda PhD
 
PPTX
Mmsys slideshare-intel-nokia
Rufael Mekuria
 
PPT
MPEG4 vs H.264
Utsav Dholakia
 
PPT
Introduction to Video Compression Techniques - Anurag Jain
Videoguy
 
ODP
MPEG-1 Part 2 Video Encoding
Christian Kehl
 
PDF
MPEG Compression Standards
Ajay
 
PDF
Mpeg v-awareness event
Marius Preda PhD
 
PPT
Games related research at Institut TELECOM
Marius Preda PhD
 
PPT
Jpeg and mpeg ppt
siddharth rathore
 
PPTX
simple video compression
LaLit DuBey
 
PPT
Video Compression Basics - MPEG2
VijayKumarArya
 
PDF
Introduction To Video Compression
guestdd7ccca
 
PPTX
Multimedia presentation video compression
LaLit DuBey
 
PDF
Video Compression Standards - History & Introduction
Champ Yen
 
PPTX
Video Compression Basics by sahil jain
Sahil Jain
 
PDF
Video Compression Techniques
cnssources
 
PDF
video_compression_2004
aniruddh Tyagi
 
PDF
A0540106
IOSR Journals
 
Tutorial on Point Cloud Compression and standardisation
Rufael Mekuria
 
Bridging the gap between web and television
Marius Preda PhD
 
MPEG Technologies and roadmap for Augmented Reality
Marius Preda PhD
 
Mmsys slideshare-intel-nokia
Rufael Mekuria
 
MPEG4 vs H.264
Utsav Dholakia
 
Introduction to Video Compression Techniques - Anurag Jain
Videoguy
 
MPEG-1 Part 2 Video Encoding
Christian Kehl
 
MPEG Compression Standards
Ajay
 
Mpeg v-awareness event
Marius Preda PhD
 
Games related research at Institut TELECOM
Marius Preda PhD
 
Jpeg and mpeg ppt
siddharth rathore
 
simple video compression
LaLit DuBey
 
Video Compression Basics - MPEG2
VijayKumarArya
 
Introduction To Video Compression
guestdd7ccca
 
Multimedia presentation video compression
LaLit DuBey
 
Video Compression Standards - History & Introduction
Champ Yen
 
Video Compression Basics by sahil jain
Sahil Jain
 
Video Compression Techniques
cnssources
 
video_compression_2004
aniruddh Tyagi
 
A0540106
IOSR Journals
 
Ad

Viewers also liked (13)

PPT
IPTV Codec & Packeting
Kuncoro Wastuwibowo
 
PPT
Mp3
Shirley Aranjo
 
PDF
intro_dgital_TV
aniruddh Tyagi
 
PPT
MPEG-DASH Conformance and Reference Software
Alpen-Adria-Universität
 
PDF
IPTV lecture
Adrian Hornsby
 
PDF
Digital TV
Vijayananda Mohire
 
PDF
mpeg2ts1_es_pes_ps_ts_psi
hexiay
 
PDF
IPTV, Internet Video and Adaptive Streaming Technologies
Cisco Canada
 
PPTX
video compression techique
Ashish Kumar
 
PDF
IPEG- Program2010
Luis Fernando Gonzalez-Llinás
 
PDF
Compression: Video Compression (MPEG and others)
danishrafiq
 
PPTX
Graphics card ppt
Sowrabh Tandle
 
PDF
Video Compression Basics
Sanjiv Malik
 
IPTV Codec & Packeting
Kuncoro Wastuwibowo
 
Mp3
Shirley Aranjo
 
intro_dgital_TV
aniruddh Tyagi
 
MPEG-DASH Conformance and Reference Software
Alpen-Adria-Universität
 
IPTV lecture
Adrian Hornsby
 
Digital TV
Vijayananda Mohire
 
mpeg2ts1_es_pes_ps_ts_psi
hexiay
 
IPTV, Internet Video and Adaptive Streaming Technologies
Cisco Canada
 
video compression techique
Ashish Kumar
 
IPEG- Program2010
Luis Fernando Gonzalez-Llinás
 
Compression: Video Compression (MPEG and others)
danishrafiq
 
Graphics card ppt
Sowrabh Tandle
 
Video Compression Basics
Sanjiv Malik
 
Ad

Similar to Tutorial MPEG 3D Graphics (20)

PPT
Mpeg 7-21
Higher Education Department KPK, Pakistan
 
PPT
The MPEG Extensible Middleware API
Alpen-Adria-Universität
 
PDF
m4-out-20027
Aniruddh Tyagi
 
PDF
m4-out-20027
aniruddh Tyagi
 
PDF
m4-out-20027
aniruddh Tyagi
 
PDF
MPEG-4 Developments
Martin Uren
 
PPTX
Presentation NBMP and PCC
Rufael Mekuria
 
PPT
Compression
Ashish Kumar
 
PDF
mpeg4
Aniruddh Tyagi
 
PDF
mpeg4
aniruddh Tyagi
 
PDF
mpeg4
aniruddh Tyagi
 
PDF
Wireless PC2TV
Deepak Malani
 
PPT
JPEG2000 Alliance IBC 2009
Hal J. Reisiger
 
PPT
Including MPEG-4 3D Graphics in your Application
Alpen-Adria-Universität
 
DOC
Report
Videoguy
 
ODP
Hw2
guest8244b5f
 
PDF
Introduction to MPEG21
Giordano Beretta
 
PDF
Performance evaluation of mpeg 4 video transmission over ip-networks
Alexander Decker
 
PDF
11.performance evaluation of mpeg 0004www.iiste.org call for-paper video tran...
Alexander Decker
 
PPTX
Mpeg 7 slides
rahul kundu
 
Mpeg 7-21
Higher Education Department KPK, Pakistan
 
The MPEG Extensible Middleware API
Alpen-Adria-Universität
 
m4-out-20027
Aniruddh Tyagi
 
m4-out-20027
aniruddh Tyagi
 
m4-out-20027
aniruddh Tyagi
 
MPEG-4 Developments
Martin Uren
 
Presentation NBMP and PCC
Rufael Mekuria
 
Compression
Ashish Kumar
 
mpeg4
Aniruddh Tyagi
 
mpeg4
aniruddh Tyagi
 
mpeg4
aniruddh Tyagi
 
Wireless PC2TV
Deepak Malani
 
JPEG2000 Alliance IBC 2009
Hal J. Reisiger
 
Including MPEG-4 3D Graphics in your Application
Alpen-Adria-Universität
 
Report
Videoguy
 
Hw2
guest8244b5f
 
Introduction to MPEG21
Giordano Beretta
 
Performance evaluation of mpeg 4 video transmission over ip-networks
Alexander Decker
 
11.performance evaluation of mpeg 0004www.iiste.org call for-paper video tran...
Alexander Decker
 
Mpeg 7 slides
rahul kundu
 

Recently uploaded (20)

PDF
Encapsulation_ Review paper, used for researhc scholars
gurumoop
 
PDF
Build a system with the filesystem maintained by OSTree @ COSCUP 2025
Jian-Hong Pan
 
PDF
Approach and Philosophy of On baking technology
30anthuong17
 
DOCX
The AUB Centre for AI in Media Proposal.docx
GAONE9
 
PPTX
Programs and apps: productivity, graphics, security and other tools
4mqw9zch22
 
PPTX
KOM of Painting work and Equipment Insulation REV00 update 25-dec.pptx
muhammadmuneebnaeem7
 
PDF
cuic standard and advanced reporting.pdf
SimoneTitaniumTomase
 
PDF
Optimiser vos workloads AI/ML sur Amazon EC2 et AWS Graviton
Julien SIMON
 
PDF
7 ChatGPT Prompts to Help You Define Your Ideal Customer Profile.pdf
SOFTTECHHUB
 
PPTX
Spectroscopy.pptx food analysis technology
nawahassan45
 
PDF
Network Security Unit 5.pdf for BCA BBA.
Serpent6
 
PDF
Machine learning based COVID-19 study performance prediction
IAESIJAI
 
PPTX
MYSQL Presentation for SQL database connectivity
Swati270511
 
PPTX
Digital-Transformation-Roadmap-for-Companies.pptx
David Malick Dieng
 
PDF
NewMind AI Weekly Chronicles - August'25 Week I
NewMind AI
 
PPTX
Big Data Technologies - Introduction.pptx
kuthubussaman1
 
PDF
Unlocking AI with Model Context Protocol (MCP)
Brian McKeiver
 
PDF
Profit Center Accounting in SAP S/4HANA, S4F28 Col11
Libreria ERP
 
PDF
KodekX | Application Modernization Development
KodekX
 
PDF
Reach Out and Touch Someone: Haptics and Empathic Computing
Mark Billinghurst
 
Encapsulation_ Review paper, used for researhc scholars
gurumoop
 
Build a system with the filesystem maintained by OSTree @ COSCUP 2025
Jian-Hong Pan
 
Approach and Philosophy of On baking technology
30anthuong17
 
The AUB Centre for AI in Media Proposal.docx
GAONE9
 
Programs and apps: productivity, graphics, security and other tools
4mqw9zch22
 
KOM of Painting work and Equipment Insulation REV00 update 25-dec.pptx
muhammadmuneebnaeem7
 
cuic standard and advanced reporting.pdf
SimoneTitaniumTomase
 
Optimiser vos workloads AI/ML sur Amazon EC2 et AWS Graviton
Julien SIMON
 
7 ChatGPT Prompts to Help You Define Your Ideal Customer Profile.pdf
SOFTTECHHUB
 
Spectroscopy.pptx food analysis technology
nawahassan45
 
Network Security Unit 5.pdf for BCA BBA.
Serpent6
 
Machine learning based COVID-19 study performance prediction
IAESIJAI
 
MYSQL Presentation for SQL database connectivity
Swati270511
 
Digital-Transformation-Roadmap-for-Companies.pptx
David Malick Dieng
 
NewMind AI Weekly Chronicles - August'25 Week I
NewMind AI
 
Big Data Technologies - Introduction.pptx
kuthubussaman1
 
Unlocking AI with Model Context Protocol (MCP)
Brian McKeiver
 
Profit Center Accounting in SAP S/4HANA, S4F28 Col11
Libreria ERP
 
KodekX | Application Modernization Development
KodekX
 
Reach Out and Touch Someone: Haptics and Empathic Computing
Mark Billinghurst
 

Tutorial MPEG 3D Graphics

  • 1. Web3D 2011, June, 20, Paris Marius Preda, PhD Chairman of MPEG 3DG Institut TELECOM
  • 2.   It’s a crazy multimedia world! –  Network is everywhere… but very heterogeneous –  Terminals are “same same”… but different! –  Content must be designed with a priori knowledge of its future use –  Applications are platform-centric instead of user-centric   Fragmented value chain Production Transmission Consumption Capturing HW/SW Playback Networks devices providers devices INTEROPERABILITY NEEDED!!! Authoring IPR Terminal Providers tools holders manufacturers Service End users scenarios
  • 3.   No common representation –  Heterogeneous kind of data •  Different types of geometry, appearance and animation models –  Always easier to specify a new data representation format than learning an existent one   Very different application domains
  • 6. FAST TRANSPORT TROUGHT THE NETWORK Size is of main Size doesn’t matter Size doesn’t matter importance
  • 7. FAST TRANSPORT TROUGHT THE NETWORK Size doesn’t matter Size doesn’t matter Size is of main importance
  • 8. To define a standard format for compressed 3D synthetic content. In other words to be for graphics what MP3 and AAC are for audio, MPEG-2 and MPEG-4 are for video and JPEG is for still images. Additionally "MPEG 3D Graphics" aims at providing mechanisms such as APIs to enable easy integration and development of applications using its standard representation tools.
  • 9. MPEG 3D Graphics within the MPEG standards family Core technologies in MPEG 3D Graphics Compressing other standards (COLLADA, X3D, …) with MPEG Virtual Worlds Interoperability for avatars with MPEG
  • 10. MPEG 3D Graphics within the MPEG standards family Core technologies in MPEG 3D Graphics Compressing other standards (COLLADA, X3D, …) with MPEG Virtual Worlds Interoperability for avatars with MPEG
  • 11. MPEG has several names •  Common: MPEG = Moving Picture Experts Group •  Official: MPEG = ISO/IEC JTC1/SC29/WG11 ISO Over 224 technical committees JTC1 Joint Technical Committee with IEC SC24 SC29 Computer graphics and image processing Coding of audio, picture, multimedia and hypermedia information GKS, PHIGS WG1 JPEG CGM, VRML, X3D Coding of still pictures WG11 MPEG Coding of moving pictures and audio
  • 12. MPEG has several subgroups Video ... and produced several Requirements Audio Systems successful standards 3D Graphics MPEG MPEG-1 MPEG-2 MPEG-4 MPEG-7 MPEG-21 ISO/IEC 11172 ’92 ISO/IEC 13818 ’94 ISO/IEC 14496 ’99 ISO/IEC 15938 ’01 ISO/IEC 21000 ’02 Video & audio Video & audio Multimedia Metadata Terminal (CD-ROM) (DVD & DVB) & interactive (description & network applications of content) specification MPEG-A MPEG-B MPEG-C MPEG-D MPEG-E MPEG-M MPEG-U MPEG-V
  • 13. MPEG-4 Features MPEG-4 Objects Naturals – still images, audio, 2D/3D video Synthetic - audio, 2D/3D objects and scenes Compression Compression MPEG-4 Objectives Compression Provide technologies for efficient compression and transmission MPEG-4 Terminal Composition, at end-user side, of natural and synthetic objects, into hybrid and interactive scenes
  • 14. MPEG-4 Features System architecture Interactive Scene Description Scene layer Object descriptor layer MPEG-4 stream Media data layer
  • 18. MPEG 3D Graphics within the MPEG standards family Core technologies in MPEG 3D Graphics Compressing other standards (COLLADA, X3D, …) with MPEG Virtual Worlds Interoperability for avatars with MPEG
  • 20. Approximation of target surface •  Method: tesselation with planar facets •  Quality: first order (linear) ⇒ no smoothness (C0 continuity) Mesh definition: IFS (Indexed Face Set) •  Connectivity: list of faces {…, Pn = {in0, in1, in2, …}, …} ⇒ arbitrary topology (non-manifold, open, higher genus, etc.) •  Geometry: list of vertices {…, Vk = (xk, yk, zk), …} Mesh coding •  Connectivity (lossless): triangle strips, triangle+vertex trees, etc. •  Geometry (lossy): coordinate quantisation + prediction from conn.
  • 21. LOD concept •  1976: Clark introduced the idea •  Main interest: rendering efficiency Taxonomy of LOD extraction techniques •  Static vs. dynamic •  Global vs. local •  Progressive vs. hierarchical LODs Successful simplification techniques •  1996: Hoppe’s edge collapses •  1997: Garland’s quadrics ⇒ qslim Progressive 3D mesh coding •  1996: Hoppe’s PM (Progressive Mesh) •  1998: IBM’s PFS (Progressive Forest Split)
  • 22. Triangles: 69451 vs. 29519
  • 23. Triangles: 69451 vs. 9575
  • 24. Triangles: 69451 vs. 1627
  • 25. Triangles: 69451 vs. 238
  • 26. MPEG-4 (1999): IFSs •  Based on VRML97 •  Arbitrary topology meshes •  “Properties” (normals, colours and textures) MPEG-4 Amd.1 (2000): 3DMC (3D Mesh Coding) •  40-50:1 compression of IFSs by IBM’s TS (Topological Surgery) •  Incremental transmission and rendering •  Progressive coding by IBM’s PFS •  Error resilience by SAIT
  • 27. MPEG-4 (1999): IFSs •  Based on VRML97 •  Arbitrary topology meshes •  “Properties” (normals, colours and textures) MPEG-4 Amd.1 (2000): 3DMC (3D Mesh Coding) •  40-50:1 compression of IFSs by IBM’s TS (Topological Surgery) •  Incremental transmission and rendering •  Progressive coding by IBM’s PFS •  Error resilience by SAIT
  • 30. Approximation of target surface •  Method: tesselation with predefined curved patches •  Quality: higher order (polynomic/rational) ⇒ Cn continuity Mesh definition •  Connectivity: regular grid of quads. or triangles ⇒ planar topology •  Geometry: list of control points {…, Pk = (xk, yk, zk), …} Mesh coding •  Connectivity (lossless): implicit •  Geometry (lossy): coordinate quantisation + prediction from conn.
  • 31. Tensor product of cubic Bézier curves Single patch (4x4 control points) vs. two patches (4x7 control points) Compression  (3547 polygons; 1215 vertices) vs. (86 patches; 212 control points)
  • 32. MPEG-4 Part 16 (2003): NURBS •  Based on VRML97 Amd., originally proposed by blaxxun •  Support for NURBS curves and patches   Specific nodes for Bézier’s curves and patches (for increased efficiency) •  Support for free-form deformations
  • 33.   IFS surfaces   Patches   Subdivision surfaces
  • 34. SS = limit of recursive refinement of base control mesh NB: refinement affects both Geometry smoothing achieved with connectivity (of abstract graph) and stencils particular to each scheme geometry (of 3D mapping) 1/16 1/16 ½ 1/16 9/16 9/16 1/16 1/8 1/8 Border/sharp vs. z interior edge stencils ½ of “butterfly” scheme x y 1/16 1/16 SS inherently define hierarchically nested LODs
  • 35. Approximation of target surface •  Method: tesselation with curved patches •  Quality: higher order ⇒ Cn continuity Mesh definition •  Connectivity: list of triangles/quads., e.g., {…, Tn = {in0, in1, in2}, …} ⇒ arbitrary (manifold) topology •  Geometry: list of control points {…, Pk = (xk, yk, zk), …} Mesh coding •  Connectivity (lossless): as for polygonal (manifold) mesh •  Geometry (lossy): as for polygonal (manifold) mesh
  • 36. Polygons + Are the simplest approach (linear approximation) + Can resolve fine details and handle arbitrary topologies – Lead to unstructured, huge meshes Patches + Are a more powerful approach (higher order approximation) + Are convenient for coarse and smooth models – Need cumbersome trimming and stitching mechanisms SSs + Connect and unify the two extremes above ++ Provide multi-resolution handles for hierarchical coding/editing
  • 37. Catmull-Clark’s (1978): quadrilateral, primal, approximating, C2 Loop’s (1987): triangular, primal, approximating, C2 Dyn++’s “butterfly” (1990): triangular, primal, interpolating, C1
  • 38.   IFS surfaces   Patches   Subdivision surfaces   Wavelet subdivision surfaces
  • 39. Target surface Base mesh Subdivide Add details
  • 40. Target surface Base mesh Subdivide Add details Price (requirements) •  Base mesh extraction •  Subdivision scheme = predictor •  Details (3D vectors) = prediction errors ⇒ remeshing Prize (advantages) •  Predictive coding ⇒ immediate (if smooth target mesh)
  • 41. Spatial partitioning Adding details in appearing parts Removing details in disappearing parts ZT ZT ZT … ZT
  • 42. by courtesy of FranceTelecom
  • 43. MPEG-4 Part 16 (2003): “plain” + wavelet SSs •  “Plain” SSs for mesh smoothing   Considered schemes: Catmull-Clark, [extended] Loop, butterfly   No details are added but…   normal control achievable through edge/vertex tagging of initial control mesh •  Wavelet/detailed SSs for surface approximation   Possibly tagged base mesh   Details are added after each subdivision step, which are…   wavelet-transformed according to one of several possible schemes   Most suited for multi-resolution editing/animation   Most suited for view-dependent transmission
  • 44.   IFS surfaces   Patches   Subdivision surfaces   Wavelet subdivision surfaces   Mesh Grid
  • 46. Vertex offset is a relative value Update vertex position when grid is deformed or animated
  • 48.   IFS surfaces   Patches   Subdivision surfaces   Wavelet subdivision surfaces   Mesh Grid   Solids
  • 49. Solid primitives Solid models: the “arithmetic of forms”
  • 50. Implicit equation: Quadrics (2nd order): Quartics (4th order):
  • 51. A cube = multiplication of 3 degenerated quadrics Multiplication of two forms Examples of solid operations
  • 52. Architecture Mechanics Biotechnology Virtual models
  • 53. Exact geometry Compactness 21 Kb 37 Kb 407 Kb 1.1 Mb
  • 54.   IFS surfaces   Patches   Subdivision surf.   Wavelet SS   Mesh Grid   Solids   SC-3DMC
  • 55. MPEG 3DGC Scalable Complexity 3D Mesh Compression •  Not all 3DG applications have the same needs in compression •  Not all the 3DG applications can afford spending extra CPU/GPU for compression TFAN SVA SC3DMC QBCR Same Quantization and Binarization blocks Towards the continuum model: enlarge the application domain where MPEG-4 3DG can be used
  • 56. MPEG 3DGC Scalable Complexity 3D Mesh Compression x, y, z (floats) + normals (floats) x, y, z (floats) + colors (floats) x, y, z (floats) + … (floats) i, j, k (integers) Attributes i, j, k (integers) i, j, k (integers) Connectivity
  • 57. SC-3DMC General Schema Connectivity TFAN BP CABAC Delta (lossless) SVA FLB AC BAC Connectivity Prediction Binarization Entropy i, j, k Analysis Encoding Attributes Prediction Entropy Binarization (lossy) Encoding Parallelogram Delta BP CABAC x, y, z Quantization Barycentre FLB AC 5 different paths with different performances
  • 58. SC-3DMC Connectivity Analysis: Empty Do Nothing 3 1 4 2 123234
  • 59. SC-3DMC Connectivity Analysis: SVA Previous Face 4 modes for encoding how consecutive Current Face faces share vertices Shared Vertex 1 5 3 1, 2, 3, 1, 2, 3, 1 4 3 2, 3, 4 3, 4, 5 12304 2 2 4 123245 Mode 0 Mode 2 1 4 1 1 1, 2, 3, 1, 2, 3, 4, 5, 6 1, 2, 3 2 3 2 3 2 3 5 6 OR 1231 5 Mode 1 Mode 3 1233
  • 60. SC-3DMC Connectivity Analysis: TFAN Split the mesh in set of triangle fans and encode each fan 1675 62937 2984 Transformed in local indices
  • 61. SC-3DMC What we measure? Encoding performances but also complexity for decoder (and encoder) TFAN Performances SVA-AC 3DMC SVA-BAC SVA-BP NCA BIFS Complexity
  • 62.   IFS surfaces   Patches   Subdivision surf.   Wavelet SS   Mesh Grid   Solids   SC-3DMC
  • 63. •  Visual Texture Coding Generic compression tool •  Synthesized Textures Very Compact representation for •  Procedural Textures specific applications •  Depth Image-based Representation Less calculation intensive
  • 64. •  MPEG-4 3DMC •  MPEG-4 MeshGrid •  MPEG-4 Subdivision Surfaces 3D Mesh •  MPEG-4 VTC •  JPEG 2000 3D object 2D Texture
  • 65. Zero-Tree WT 3D mipmaps
  • 66. Multiple In Place mapping = multi-resolution textures Without mipmapping With mipmapping
  • 68. Region of Interest with resolution/quality selection Visual importance depends on viewing angle Low Resolution High Resolution
  • 69. Packet selection by using error-resilience markers
  • 70.   IFS surfaces   VTC   Patches   Synthesized texture   Subdivision surf.   Wavelet SS   Mesh Grid   Solids   SC-3DMC
  • 71. Vector Graphics Primitives Color profile Line Color Profile ( LC ) Area Color Point ( A C ) Line ( LN ), bounded by 2 Terminal Points ( TP ) The image cannot be displayed. Color Patch ( PA ) Line Segment ( LS ), bounded by 2 Line Points ( LP ) Sub -Texture ( ST ) Line marked as control line (Skeleton) The image cannot be displayed.
  • 72. 65x96 pixels 10 seconds animation 1.35 kB
  • 73.   IFS surfaces   VTC   Patches   Synthesized texture   Subdivision surf.   Procedural texture   Wavelet SS   Mesh Grid   Solids   SC-3DMC
  • 74. DEF Fabric ProceduralTexture { type 2 width 256 height 256 cellWidth 4 cellHeight 4 roughness 1 distortion 0.05 seed 114300 color [ 0.898 0.89418 0.95294, 0.34118 0.29418 0.70196, 0 0 0, 0 0 0 ] aWarpmap [ 0 0, 0.03 1, 0.88 1, 1 0 ] bWarpmap [ 0 0, 0.48 1, 1 0 ] aWeights [ 0, 0, 0, 0, 0, 0, 0.56, 0, 0, 0, 0, 0, 0, 0, 0.20, 0.24 ] bWeights [ 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0 ] } DEF Marble ProceduralTexture { width 256 height 256 roughness 1 seed 22209 color [ 0.8 0.7098 0.6902, 0.95686 0.8902 0.87451, 0.87451 0.37255 0.23529, 0.95686 0.8902 0.87451 ] aWarpmap [ 0 1, 0.33 0, 1 1 ] bWarpmap [ 0 0, 0.55 0, 0.6 1, 0.65 0, 1 0 ] bWeights [ 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0 ] }
  • 75.   IFS surfaces   VTC   Patches   Synthesized texture   Subdivision surf.   Procedural texture   Wavelet SS   DIBR   Mesh Grid   Solids   SC-3DMC
  • 76. Reconstruct 3D representation from projections Depth image Projection
  • 77.   IFS surfaces   VTC   Patches   Synthesized texture   Subdivision surf.   Procedural texture   Wavelet SS   DIBR   Mesh Grid   Point Texture   Solids   SC-3DMC
  • 78.   IFS surfaces   VTC   Interpolators   Patches   Synthesized texture   Subdivision surf.   Procedural texture   Wavelet SS   DIBR   Mesh Grid   Point Texture   Solids   SC-3DMC
  • 79. Piece-wise linear functions defined by a set of pairs time-value Standardized by MPEG-4 for: - position - orientation - scale - coordinate in IFS - normals in IFS Straightforward animation based on key-frames
  • 80. Large amount of data for high quality, smooth animation
  • 81. Re-sampling, sub-sampling: two methods supported by MPEG-4 Path preserving mode Key preserving mode Three schemes supported by MPEG-4 Orientation Interpolators Coordinate Interpolators Position Interpolators Exploit temporal redundancy for the most common interpolators
  • 82. Re-sampling, sub-sampling: two methods supported by MPEG-4 Key value Key value Path preserving mode Key(time) Key(time) Key preserving mode N key values N-1 key values Three schemes supported by MPEG-4 Key value Key value Orientation Interpolators Coordinate Interpolators Key(time) Key(time) Position Interpolators N-2 key values N-3 key values Sub-sampling or re-sampling based on minimal distortion
  • 83. A dedicated elementary stream for IC, multiplexed into an AFX stream
  • 84.   IFS surfaces   VTC   Interpolators   Patches   Synthesized texture   Bone-Based   Subdivision surf.   Procedural texture   Wavelet SS   DIBR   Mesh Grid   Point Texture   Solids   SC-3DMC
  • 85.   Face, MPEG-4 V1, 1999   Body, MPEG-4 Amd1, 2000   Skinned Model, MPEG-4 Part 16, 2003 Two frameworks: human-like (FBA) and generic skeleton (BBA)
  • 86.   Geometry: Seamless mesh: shapes sharing the same vertices list   Texture: Image Mapping on vertices sub-set   Hierarchy: Skeleton layer Muscle layer Seamless mesh affected by a hierarchical skeleton
  • 87.   1D controllers: bone & muscle   for each bone and each muscle - a list of affected vertices - a measure of affectedness are provided Right balance between control parameters and influence volume
  • 88. Uncompressed Uncompressed Segment #n Frame #n BAP/FAP/BBA BAP/FAP/BBA Frame P DCT Prediction DC Coeff. AC Coeff. Frame I Segment P Segment Prediction I Quantization Quantization DC Q DC Q AC Q Arithmetic Arithmetic coding coding Huffman coding Huffman coding Binary file Binary file Basic comp.: prediction, freq.transform, quantization and entropy encoding
  • 89.   Face, MP4 V1, 2kbps   Body, MP4 Amd1, 5-30 kbps   Skinned Model, MP4 Part 16, 5-30 kbps for a human like skeleton Very low bit-rate
  • 90.   IFS surfaces   VTC   Interpolators   Patches   Synthesized texture   Bone-Based   Subdivision surf.   Procedural texture   Morphing   Wavelet SS   DIBR   Mesh Grid   Point Texture   Solids   SC-3DMC
  • 91.   Defined as a base mesh and a collection of target meshes   Animation obtained by updating the weights of the target meshes   BBA stream updated to include morph data   Usable for any kind of 3D object Local and precise control for shape deformation
  • 92.   IFS surfaces   VTC   Interpolators   Patches   Synthesized texture   Bone-Based   Subdivision surf.   Procedural texture   Morphing   Wavelet SS   DIBR   FAMC   Mesh Grid   Point Texture   Solids   SC-3DMC
  • 93.   Cluster the vertices with respect to their motion   Encode a cluster motion by an affine transform   Encode the residual error at vertex level by traditional approach (DCT/W, quantization, entropy encoding)
  • 94.   IFS surfaces   VTC   Interpolators   Patches   Synthesized texture   Bone-Based   Subdivision surf.   Procedural texture   Morphing   Wavelet SS   DIBR   FAMC   Remote and   Mesh Grid   Point Texture programmatic   Solids   SC-3DMC
  • 95. Bifs-Anim   Remote animation BIFS-Anim & BIFS Command Bifs-Command scene   Programmatic animation Script Node - ECMA Script scene part of the scene description - Java Code standardized API for accessing MPEG-J Stream the scene graph scene Complex application scenario can be built
  • 96.   IFS surfaces   VTC   Interpolators   Patches   Synthesized texture   Bone-Based   Subdivision surf.   Procedural texture   Morphing   Wavelet SS   DIBR   FAMC   Mesh Grid   Remote and   Point Texture programmatic   Solids   SC-3DMC
  • 97. MPEG 3DG How measuring compression performances? On-line benchmarking platform: www.MyMultimediaWorld.com        
  • 98. MPEG 3DG How measuring compression performances? On-line benchmarking : www.MyMultimediaWorld.com 3D compression Web site benchmark Benchmark 2 Benchmark 3 Filter and Presentation Engine Benchmark manager 2 Benchmark manager 1 Indexed MDB 3D Compression benchmark manager API MP4 Extended MP7 Algorithm 1 Algorithm 2 … Algorithm n
  • 99. MPEG 3DG How measuring compression performances? On-line benchmarking : www.MyMultimediaWorld.com GetNbBitstream n Vertex Buffer Encoder Proprietary Coder library DumpBitSream BitStream (1,n) DumpCompressedVB Proprietary Decoder Decoder library .Compressed_VB (1,n)
  • 100. MPEG 3DG How measuring compression performances? Benchmarking platform, per-object visualization - object properties (number of vertices/ triangles, number of components and files size, - distortion graph (linear or logarithmic), - compression gain with respect to 3DMC. -  encoding time -  decoding times.
  • 101. MPEG 3DG How measuring compression performances? Benchmarking platform, global visualization Filters - semantic category, - average distortion, - number of vertices, - number of connected components in the object, - database subset.
  • 102. MPEG 3DG How measuring compression performances? Benchmarking platform, global visualization
  • 103. MPEG 3D Graphics within the MPEG standards family Core technologies in MPEG 3D Graphics Compressing other standards (COLLADA, X3D, …) with MPEG Virtual Worlds Interoperability for avatars with MPEG
  • 106. Geometry compression: -compression ratio 40:1 Animation compression: -compression ratio 100:1
  • 107. Any XML Scene Representation XML XML (Binary) XML Binarisation 3D Graphics Compression
  • 108. Layer 1: Any XML Scene Representation XML Textual XML representation of the XML scene (Binary) XML Binarisation 3D Graphics Compression
  • 109. Layer 2: Any XML Scene Representation XML Binarized XML layer XML Contains the (Binary) XML Binarisation unclassified elements of the scene graph 3D Graphics Compression
  • 110. Layer 3: Any XML Scene Representation XML Compressed layer XML Contains specific (Binary) XML Binarisation types of media information 3D Graphics (geometry, Compression animation, ...)
  • 111. Result: Any XML Scene Representation XML Multiplexed XML layers 2 and 3 (Binary) XML Binarisation 3D Graphics Compression
  • 112. MPEG-4 P25 Encoder side: possible implementation (informative) Encoder Side MP4 XMT, COLLADA, X3D Comp ratio: 40-70:1
  • 113. MPEG-4 P25 Decoder side (normative) Decoder Side MP4 XMT, COLLADA, X3D Lossless for the data structure, lossless or lossy for graphics primitives
  • 114. MPEG 3D Graphics within the MPEG standards family Core technologies in MPEG 3D Graphics Compressing other standards (COLLADA, X3D, …) with MPEG Virtual Worlds Interoperability for avatars with MPEG
  • 115. Standards for Avatars as visualization support Standard Generic Features Avatar representation COLLADA 3D objects/ scenes Generic Object VRML/ X3D 3D objects /scenes H- Anim Application behavior MPEG-4 2D/3D objects/ scenes A set of dedicated nodes Application behavior A dedicated compressed stream Compression
  • 116. Standards for Avatars as interaction support Representation Features HumanML human physical description, emotion EmotionML emotion, facial expressions, gestures BML speech, gesture, gaze MPML speech VHML facial and body animation, emotional representation CML character attribute and animation definition
  • 117. Why none of the existing standards is solving the issue of avatar interoperability in Virtual Worlds? –  The Virtual Worlds are proprietary applications and the 3D assets including the avatars have economical value –  The Virtual Worlds and in general 3D applications have specific data format allowing rendering optimization However, while maintaining a strict control for economic and technical reasons, Virtual Worlds allow users to personalize the avatars
  • 118. Avatar Template Attributes that can be modified by the user Specify the set of Personalization Parameters (PP) that transforms a template of an arbitrary Virtual World into the user designed avatar
  • 119. What avatar feature can be personalized? Mainly the appearance Analysis of SecondLife, IMVU, Entropia Universe, SonyPlaystation and HumanML
  • 120. Very heterogeneous set of personalization parameters Entropia Nintendo Universe Wii HumanML Second Life PlayStation
  • 121. “Appearance” element <Appearance> <Body> <BodyHeight value=165/> <BodyFat value=15/> </Body> <Head> <HeadShape value="oval"/> <EggHead value="true"/> </Head> </Appearance>
  • 122. Interoperability at the Animation level <Animation> <Greeting> <Salute>salut</Salute> <Cheer>cheer</Cheer> </Greeting> <Fighting> <shoot>pousse</shoot> <throw>throw</throw> </Fighting> </Animation>
  • 123. Motion retargeting No “Walk” Avatar Avatar animation “Walk” animation Template in VW1 Template in VW2 defined in VW2 In VW1
  • 124. “Control” element Body Face Control Control <Control> <BodyFeaturesControl > <UpperBodyBones> <LCalvicle>my_LCalvicle</LCalvicle> Control <RClavicle>my_RCalvicle</RClavicle> </UpperBodyBones> </BodyFeaturesControl> <FaceFeaturesControl> <HeadOutline> <Left X=0.23 Y=1.25 Z=7.26/> <Right X=0.25 Y=1.25 Z=7.21/> </HeadOutline> </FaceFeaturesControl> </Control>
  • 125. MPEG-V (Personalization Parameters) MPEG-V + MPEG-4 The Avatar in The Avatar in VW1 VW2 The Avatar in an external player
  • 127. MPEG 3DGC Multi-Resolution 3DMC Current 3DG content in VW © Samsung © Second Life Tomorrow’s 3DG content in VW (100 times denser) © OTOY © OTOY
  • 128. MPEG 3DGC Reconfigurable Graphics Coding A framework that allows to set up the decoder at run time
  • 129. Thank you! www.MyMultimediaWorld.com www-artemis.it-sudparis.eu marius.preda@it-sudparis.eu
  • 130. MPEG 3DGC Multi-resolution 3DMC Progressive mesh [Hoppe’96] Progressive Forest Split (PFS) [Taubin’98] Patch coloring [Cohen-Or’99] Valence-based decimation approach [Alliez’01] Octree-based compression [Peng’05] Spectral coding [Karni’01]
  • 131. MPEG 3DGC Multi-resolution 3DMC : two proposed methods 1. Progressive TFAN Original Interpolated with 1% of the vertices
  • 132. MPEG 3DGC Multi-resolution 3DMC : two proposed methods 2. KLT encoder 2 2 2 1 1 1
  • 133. MPEG 3DGC Multi-resolution 3DMC : two proposed methods 1. Work on -  document the use case scenarios (object examination, navigation, …) -  database -  comparison with other codecs -  comparison method (PSNR, Resolution) -  platform for testing 2. You are invited to contribute