SlideShare a Scribd company logo

Tutorial on Point Cloud Compression and standardisation

Download as PPTX, PDF
R
Rufael Mekuria

This document provides a comprehensive overview of point cloud compression, its significance for virtual reality (VR) applications, and the complexities involved in capturing and compressing 3D data. It discusses various techniques for capturing point clouds, the architectural considerations for real-time coding, and the challenges in achieving efficient compression while maintaining quality. Additionally, it outlines the author's expertise and contributions to the field through various projects and publications.

Science
1 of 86
Downloaded 223 times
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
Point Cloud Compression Tutorial Rufael Mekuria (PhD), Unified Streaming and point cloud compression IEEE Visual Communications and Image Processing 2017 St. Petersburg Florida Invited Tutorial rufael@unified-streaming.com Point Cloud Compression
About Unified Streaming Unified Streaming Creator of video streaming software behind many of the large scale early deployments of HTTP Streaming (BBC iplayer etc.., NPO) www.unified-streaming.com key implementer of technologies like DASH, HLS, CMAF et… used by CDN, Telco, cloud, broadcaster etc. MPEG affiliate, DASH-IF Member DASH, CMAF, Network based media processing 5G Video streaming pioneer Involved in EU H2020 project Superfluidity with companies like Intel, RedHat and Nokia. Helped many telco/CDN on advanced streaming architectures Point Cloud Compression
About Me Academic Background MSEE, Delft 2011, PhD CWI/Vrije Universiteit Amsterdam jan’17 25+ peered reviewed papers, best paper in ACM MMSys’13, 5+ invited papers in conferences like SPIE, ICASSP, etc…. Invited talks: UIUC, Huawei, Google etc… Professional Background TNO (2010-2011), CWI 2011-2016, Unified Streaming 2016-date, PCC (2017-) Active in large scale EU Funded projects: Superfluidity H2020, Reverie FP7, developed one of the first large scale tele-immersive system combining real and CGI content Standardization Work Chaired and pioneered PCC in MPEG (since jan.2014) CfP was responded by 9 companies in oct. 2017 including all major mobile device vendors (CfP response referred to as “Historic” by MPEG convenor). Point Cloud Compression
About this Tutorial Before the break: Introduction to this (re-) emerging research area, point clouds for VR, point cloud compression architecture After the break: Standardization activity on point cloud compression in MPEG Applications and requirements, quality assessments State of art and research challenges Copyrights & Acknowledgement: Some of the images and parts of this tutorial fall under copyright of respective contributors/authors as noted to MPEG CfP. I do not name the contributors explicitly for now. Many opinions in this work are derived from standardization meetings representing rough industry consensus. I acknowledge 8i, Microsoft, Technicolor, Ford, Mitshubishi, et al. for contribution of test data. I thank Technicolor for large contributions to this body of work. Point Cloud Compression
 A collection of points  Not related to each other  Typically no order  Typically no local topology (no mesh!)  Each is point is the given of  a position (X,Y,Z)  a color (R,G,B) or (Y,U,V)  possibly other things like transparency, time of acquisition, etc. Point Cloud Compression Point cloud content from Microsoft research laboratory Donated to MPEG Point Cloud Format
.ply files = an example raw data format for point cloud  This is an example raw point coud file format (compare to YUV for video coding)  How many points?  Static case up to several tens of millions, depending on the application  Dynamic case ~1 million per frame, 30 fps  Probably more is needed for good VR  Format  Geometry XYZ  Fixed precision for VR applications  Float still often used  Colors RGB  As usual, integer 8/10 bits.  Possibly other attributes  (not present in the ply file here on the left) plyformat ascii 1.0 element vertex 764940 property float x property float y property float z property uchar red property uchar green property uchar blue end_header 211 63 63 127 98 73 213 63 61 134 109 87 212 62 63 122 97 75 212 63 62 129 102 79 212 63 63 124 98 76 213 62 63 122 98 76 213 63 62 128 104 81 213 63 63 124 99 78 215 61 63 120 97 76 214 63 60 141 117 95 214 63 61 135 111 89 215 63 60 144 120 97 215 63 61 133 109 87 214 62 62 126 102 80 214 62 63 122 98 77 214 63 62 128 104 82 etc. one point X Y Z R G B no order! Swapping points does not change the data Point Cloud Compression
An application of point cloud: free-view point (6DoF) for sport  Scene model  360°/omnidirectional background  reshaping depending on viewpoint  3D object  occlusion, parallax (in HMD)  position relatively to the background  Free-view path  viewer body position freely chosen on the free-view path  + free head movement (in HMD) 360° background 3D objects free- view path https://www.youtube.com/watch?v=Q-LNA9KlHhw Point Cloud Compression
Why Point Cloud? (for VR)  No occlusions  all angle of views are acceptable => parallax  free-view point (=6 DoF VR) is natively supported  Fine topology  volumes, hairs, fur can be represented by a point cloud  naturally captured by sensors without heavy processing  usually deduced from depth and/or disparity from multi-view capture
Point Cloud Representation in Microsoft holoportation
Real and Virtual Engagement in Realistic Immersive Environments Point Cloud Compression/transmission: Immersive Communications (2014) in Reverie FP7 Highly realistic representation for immersive communications reveriefp7.eu Human is reconstructed as a photo realistic 3D Cloud (or mesh) of Points in a 3D space! Challenges: Low bit rate, real-time encoding, color coding, inter frame coding, scalability
Point Clouds for high-end AR/VR  Key Requirements for a 3DoF+/6DoF VR/AR format  Support of stereo imaging with view dependent parallax  360 video ruled out  Universal applicability  Effective handling of occlusions  2D plus depth ruled out  Easy acquisition & Rendering  Candidates: (Super-) Multi-view, Point Cloud, (Mesh)  Comparing MV and PC  Multi-view + Easy acquisition and existing compression technology - Number of views constraints occlusions and viewing angle  Combining with CGI models difficult – requires methods similar to point cloud creation  Point Cloud + Most versatile – works for live-action acquisitions as well as CGI. Composition of scenes is easy (just “cat” point clouds) + Occlusions only depend on acquisition technology o Known but comparably complex approaches for acquisition - No efficient compression technology yet
How to capture point clouds?  Multi camera setups  Depth estimation/measurement  Color/feature detection  3D reconstruction  3D Modelling
High Quality (studio level?) content
Low Quality Setup (3D skype ?)
Algorithms for point cloud reconstruction - From multiple depth images (Reverie project) - From rigs with stereo cameras (microsoft) - From multiple images (3D culture cloud) - Mobile devices (new sony experia for example) - Systems like 8i or owlii - Systems for point cloud capture will be key if pcc will be an impotrant medium For delivery
How to render point clouds?  Giving size to points  Splats, rectangles, cubes (=3D pixels)  Trade-off size vs. texture high frequency  Meshing, and using illumation techniques  A demo using PCC contents (and renderer)  8i content  Technicolor based rendering
Rendering Demonstration MPEG Data and rendering, data by 8i
How to compress point clouds: examples of known technologies  How to compress PC geometry ?  octree-based  occupancy data to be entropy coded using prediction  intra local prediction (local plane, etc.)  inter-prediction (motion)  image based  depth coding but unable to handle occlusions  global/local  global envelope for surfacic objects, then geometrical residual  similar to mesh + height  How to compress PC colors ?  octree-based  palette in an octree, prediction and residual coding  wavelets on trees  image-based  projection on planes/surfaces, compression using a 2D video codec  local projection and tiling/packing in a unique image to avoid loss of occlusions due to global projections  block-based (mimicking video compression)  3D blocks, push points in a corner, prediction, 3D-DCT, quantization and entropy coding  inter prediction with 3D motion vectors
Comparison with other format compression  Performance comparison with other formats  in bit per pixel/points to be displayed (bpp) Compression format (lossy, good quality) bpp 2D flat UHD, intra ~0.25-0.5 2D flat UHD, inter ~0.025-0.1 2D + smooth depth 2D flat + 25% (for parallax) Multi-View 2D*nb views*75% Point Cloud geo ~1-3, texture ~0.5-2 Mesh geo ~0.25, texture ~0.5 Light-field unknown yet  Challenges ahead  improve PC compression efficiency  in particular for inter coding  find a robust method to assess compressed geometry and texture quality  Industry wants fast adoption, re-use of existing hardware infrastructure (e.g. HEVC, AVC) will be important
Questions (round 1) ?? Point Cloud Compression
Design of PCC Anchor Point Cloud Compression - Design for Reverie Immersive online platform - Replicant (3D Point Cloud Reconstruction) by CERTH-ITI - Real-time requirements - Different quality representations (for rendering at different distances) - Publication: R. Mekuria, K. Blom and P. Cesar, "Design, Implementation, and Evaluation of a Point Cloud Codec for Tele-Immersive Video," in IEEE Transactions on Circuits and Systems for Video Technology, vol. 27, no. 4, pp. 828-842, April 2017. doi: 10.1109/TCSVT.2016.2543039 - Experimental work, codec used as anchor for MPEG PCC
Reverie Immersive Framwork Media Router Media Router Large Scale Tele-Immersive Architecture User Analysis Scene structuring & NavigationScene structuring & NavigationServer Composition Renderers Content Decoding Stream Synchronization 3D Capturing 3D Reconstruction Scalable Content Coding Streaming User Generated Objects Network Monitoring Session Manager Session Set-up Social Network Media Coordinator Avatar Embodiment Media Router Stream Selector & Media Objects Audio Streams Perception Cognition Animation Avatar Reasoning Media Stream Demux Visual Streams Client Media Router Network Monitoring Spatial & Audio Composition
Real and Virtual Engagement in Realistic Immersive Environments Point Cloud Compression: Immersive Communications Replicant (immersive communications) reveriefp7.eu Human is reconstructed as a photo realistic 3D Cloud (or mesh) of Points in a 3D space! Challenges: Low bit rate, real-time encoding, color coding, inter frame coding, scalability
Architecture Hybrid Octree point cloud codec Point Cloud Compression
Point Cloud Geometry Compression recursive sub-divsions: complexity O(2^(N)) Limit level N and differentially code of points in larger leafs for real-time coding [Kammerl12] Prediction of subdivisions based on the previous level [Schnabel06, Huang06] Context Adaptive Entropy encoding [Schnabel06, Huang06] range coding [Kammerl12] 10000000 10000000 [Kammerl12] Kammerl, J.; Blodow, N.; Rusu, R.B.; Gedikli, S.; Beetz, M.; Steinbach, E., "Real- time compression of point cloud streams," Robotics and Automation (ICRA), 2012 IEEE International Conference on , vol., no., pp.778,785, 14-18 May 2012 [Schnabel06] Ruwen Schnabel and Reinhard Klein. 2006. Octree-based point-cloud compression. In Proceedings of the 3rd Eurographics / IEEE VGTC conference on Point-Based Graphics (SPBG'06), [Huang06]Huang Yan Huang, Jingliang Peng, C.-C. Jay Kuo, and M. Gopi. 2006. Octree-based progressive geometry coding of point clouds. In Proceedings of the 3rd Eurographics / IEEE VGTC conference on Point-Based Graphics (SPBG'06) Point Cloud Compression
Real and Virtual Engagement in Realistic Immersive Environments Point Cloud (Color) Attribute Compression DPCM [Kammerl12] Colorization [Huang06] Octree Based [schnabel06] Based on Graph Transform [Zhang14] Mapping to jpeg image grid ?? Table 1 scan order for mapping octree centroid colours to image grid (8x10 sample block) 0 1 2 3 4 5 6 7 64 … 15 14 13 12 11 10 9 8 79 … 16 17 18 19 20 21 22 23 80 … 31 30 29 28 27 26 25 24 95 … 32 33 34 35 36 37 38 39 … … 47 46 45 44 43 42 41 40 … … 48 49 50 51 52 53 54 55 … … 63 62 61 60 59 58 57 56 … … 0 0.5 1 1.5 2 2.5 3 dense octree (11) sparse octree (9) bitrate(bytesperoutput vertex/voxel) Achieved Compression Ratio original (5 bits colors) proposed (jpeg mapping) Overall coding gain compared to legacy point cloud codec available in PCL (at comparable objective quality) Traverse octree Write color attributes to an image grid JPEG Encode
Real and Virtual Engagement in Realistic Immersive Environments Inter Predictive Coding of Octree Compressed Point Clouds Basic Algorithm Overview 1. Input Cloud I and Input Cloud P 2. Align bounding box Cloud I and P 3. Compute octrees of I and P at level N – M 4. Find common leafs at level N-M occupied in I and P 5. Try to predict the vertices in leafs in P from leafs in I 6. If P can be predicted, code vertices as a rigid transform on the input vertices 7. Code all other vertices in P via an intra coding scheme Algorithm Details 1. Color variance and point count used to decide wether or not to do the prediction 2. Compute prediction via ICP procedure (iterative closest points) 3. Compress the rigid transform as a quaternion or 2 vectors 4. Use an octree coding upto level N to code the points that cannot be predicted
Real and Virtual Engagement in Realistic Immersive Environments Bounding Box Alignment I input cloud compute BB compute BB P input cloud expand BB I normalize I on BB_IE normalize P on BB_IE P Coding of P I Coding of I expand BB P normalize P on BB_PE I Coding of P BB_P fits BB_IE BB_I BB_P BB_IE BB_IE N BB_IE Y BB_PE
Real and Virtual Engagement in Realistic Immersive Environments Inter prediction Algorithm normalized I Cloud normalized P Cloud Generate Macroblocks of I and P For each Macroblock in P corr macroblock in I ? color_var < TRESH icp converged ? encode rigid transform fittness < thresh ? Intra Encoding of Macro Block Intra Encoding of Macro Block Intra Encoding of Macro Block store key, rigid tf, color_offset I coded part p coded part Compute rigid transform via ICP between corr Macroblocks I and P N Y Y N Y N N Y # of points ok ? 1. 2. 3. 4. 5. 6. 7. 8. 15 bytes N Y N-M Octree !!
Real and Virtual Engagement in Realistic Immersive Environments Results: Bounding Box Alignment Alex at 12 fps, Alex at 24 fps Bounding box factor Alignment % (12 fps) Alignment % (24 fps) 5 % 44 65 10 % 68 76 15 % 72 83 20 % 74 83 25 % 82 88
Real and Virtual Engagement in Realistic Immersive Environments Shared Macroblocks, convergence percentage Alex at 24 fps (100 frames) Alex at 12 fps (50 frames), bounding box factor 20 % 11 level octree Dataset % shared macroblocks % ICP converged % points p coded (apr.) Alex (12 fps) 65 % 32 % ~ 25 % Alex (24 fps) 75 % 35 % ~ 30 % Christos (12 fps) 66 % 34 % ~ 30 % Christos (24 fps) 78 % 40 % ~ 35 % Dimitrios (12 fps) 69,9% 39 % ~ 35 % Dimitrios (24 fps) 78 % 42 % ~ 40 %
Real and Virtual Engagement in Realistic Immersive Environments Point Cloud Compression, Quality Assessment (m36529) Table 2 – MOS on the perceived quality d_sym_rms Symmetric rms distance betw. Clouds (5) d_sym_haussdorf Symmetric Haussdorf dist. betw. clouds (7) psnr_geom PSNR(vertexpositions) (8) psnr_y PSNR(colorsY) (10) psnr_u PSNR (colors U) (as 10 for u) psnr_v PSNR (colors V) (as 10 for v) 𝑑 𝑟𝑚𝑠 𝑉𝑜𝑟 , 𝑉𝑑𝑒𝑔 = 1 √𝐾 ||𝑣𝑙 − 𝑣 𝑛𝑛 _𝑑𝑒𝑔 ||2𝑣 𝑙 ∈𝑉𝑜𝑟 , (4) 𝑑 𝑠𝑦𝑚 _𝑟𝑚𝑠 𝑉𝑜𝑟 , 𝑉𝑑𝑒𝑔 = 𝑚𝑎𝑥⁡( 𝑑 𝑟𝑚𝑠 𝑉𝑜𝑟 , 𝑉𝑑𝑒𝑔 , 𝑑 𝑟𝑚𝑠 𝑉𝑑𝑒𝑔 , 𝑉𝑜𝑟 ) (5) 𝑑ℎ 𝑎𝑢𝑠𝑠 𝑉𝑜𝑟 , 𝑉𝑑𝑒𝑔 = 𝑚𝑎𝑥 𝑣 𝑙 ∈𝑉𝑜𝑟 , ( ||𝑣𝑙 − 𝑣 𝑛𝑛 _𝑑𝑒𝑔 ||2 ) (6) 𝑑 𝑠𝑦𝑚 _ℎ 𝑎𝑢𝑠𝑠 𝑉𝑜𝑟 , 𝑉𝑑𝑒𝑔 = 𝑚𝑎 𝑥 𝑑ℎ 𝑎𝑢𝑠𝑠 𝑉𝑜𝑟 , 𝑉𝑑𝑒𝑔 , 𝑑ℎ 𝑎𝑢𝑠𝑠 𝑉𝑑𝑒𝑔 , 𝑉𝑜𝑟 (7) 𝑝𝑠𝑛𝑟𝑔𝑒𝑜𝑚 = 10𝑙𝑜𝑔10 ( | 𝑚𝑎𝑥 𝑥,𝑦,𝑧 𝑉𝑑𝑒𝑔 |2 2 / (𝑑 𝑠𝑦𝑚 𝑟𝑚𝑠 𝑉𝑜𝑟 , 𝑉𝑑𝑒𝑔 )2 ) (8) 𝑑 𝑦 𝑉𝑜𝑟 , 𝑉𝑑𝑒𝑔 = 1 √𝐾 ||𝑦(𝑣𝑙) − 𝑦(𝑣 𝑛𝑛 _𝑑𝑒𝑔 )||2𝑣 𝑙 ∈𝑉𝑜𝑟 , (9) 𝑝𝑠𝑛𝑟𝑦 = 10𝑙𝑜𝑔10 ( |𝑚𝑎𝑥 𝑦 𝑉𝑑𝑒𝑔 |2 2 / (𝑑 𝑦 𝑉𝑜𝑟 , 𝑉𝑑𝑒𝑔 )2 ) (10) We compare the output point to the nearest input point and vice versa!
Real and Virtual Engagement in Realistic Immersive Environments Results Inter Prediction For Alex, Chris and Dimitrios > 65 % of blocks are shared and ~ 30 percent can be predicted Jog and ski don’t work well because outlier points mess up the bounding box algorithm 0 1 2 3 4 5 alex chris dim Bytesperoutputvertex Dataset (ITI/CERTH, 24 fps) bit rate gain predictive coding intra pred 60 65 70 75 80 alex chris dim Quality[dB] Dataset (ITI/CERTH 24 fps) Quality Geometry [dB] intra pred 0 200 400 600 800 alex chris dim Encodertime[ms] Dataset Encoder Time intra pred
Real and Virtual Engagement in Realistic Immersive Environments PCC Reference Software (m36527) http://wg11.sc29.org/svn/repos/MPEG-04/Part16- Animation_Framework_eXtension_(AFX)/trunk/3Dgraphics/3DG- PCC/trunk/pcc-mp3dg (official mpeg version) https://github.com/RufaelDev/pcc-mp3dg (open source) Tested: Win7, Mac, Ubuntu Linux Cloud Codec V2 compression_eval Quality
Real and Virtual Engagement in Realistic Immersive Environments Octree Based Point Cloud Table 2 – MOS on the perceived quality
Real and Virtual Engagement in Realistic Immersive Environments Coding Color Attributes with JPEG Table 2 – MOS on the perceived quality
Real and Virtual Engagement in Realistic Immersive Environments Temporal Prediction via ICP Table 2 – MOS on the perceived quality
Questions (round 2) ?? Point Cloud Compression
Standardization of PCC Outline Point Cloud Compression - PCC Exploration in MPEG - Use Cases / Requirements - Test Datasets - Quality Assessment - Common Test Conditions - Rendering - Anchor - Call for Proposals (10 responses, october 2017)
MPEG Standardization Point Cloud Compression - Industry consortium for developing standardized specification under ISO/IEC - Meets 3-4 times per year - Joint work with ITU on video coding (AVC, HEVC) - Much used standards in the media industry - Still relatively strong attendance (around 300 per meeting) - PCC AhG group under 3D Graphics groups (MPEG only) - Next Meeting in Gwanju, Korea, in January
Exploration on PCC/3DGC Point Cloud Compression - Started in 2014 in San Diego Meeting - Naturalistic content - Virtual Reality use case (not yet popular back then) - Techniques for point clouds in addition to meshes - Evaluation methodology (objective, subjective metrics) - Anchor codec framework - Work done as PhD student at CWI - Contributions from Reverie EU project - Discussion with major industries in 2016 on the requirements and methodology (e.g. Apple, Nokia, Huawei, Technicolor, 8i, owlii, Orange, Samsung, Orange, Canon, Sony, Panasonic etc..) - Call for Proposals in Jan. 2017, (10 responses, october 2017) - Industry standard specification expected by mid 2019
Table 2 – MOS on the perceived quality Point cloudImages Point Cloud Compression in VR Ecosystem (1) Point Cloud Compression
Use Cases (1) Point Cloud Compression - High Quality Broadcast of content (volumetric video) - Tele-immersive mixed reality (real-time volumetric video), both AR/VR like - Cultural Heritage, 3D photography, CAD scans - Mobile mapping data - Large scale geographic information - https://mpeg.chiariglione.org/standards/exploration/point-cloud- compression/use-cases-point-cloud-compression
Use cases (2) Gis: Lidar: http://www.cadalyst.com/cadalyst/gis-tech-news-99-13394 Robotics: lidar, kinect: www.pointclouds.org GIS: Aereal Imaging http://www.flightlinegeographics.com/3D_Point_Cloud.html Cultural heritage (images): Culture3D Cloud Mixed Reality
Use Cases (3) Point Cloud Compression 3D Recon Struction Software Reconstructed 3D Human 3D Point Cloud Representation 3D Source Encoding IP Network Multi Depth Camera Capture Or other 3D Capture Real-Time 3D Rendering Composition In Virtual World Packetization & Transmission 3D Source Decoding N RGB + Depth Images Or other sensor data Reception & Synchronization Figure 1: Transmission pipeline for conferencing with 3D geometry Key requirements for this application are: Lossy compression with bit-rate control is needed low complexity and/or support for real-time encoding/ decoding is needed error resilience to transmission errors is needed color attributes coding is needed for realistic rendering material/appearance related attributes coding is needed to code additional attributes to support the rendering view dependence: for streaming optimization view dependence can be used to optimize the transmission process Typical point clouds in this use case have the following characteristics: Between 100,000 and 10,000,000 points to represent a reconstructed human color attributes with 8-10 bits per color component Normals and or material properties to support the rendering using a shader
Use Cases (4) Point Cloud Compression Key requirements for this application are: Lossy compression with bit-rate control is needed error resilience to transmission errors is needed color attributes coding is needed for realistic rendering material/appearance related attributes coding is needed to code additional attributes to support the rendering view dependence: the view dependence can be used to optimize the encoding process Typical point clouds in this use case have the following characteristics: Between 100,000 and 10,000,000 points to represent closeby objects in the scene, Color attributes with 8-10 bits per color component Global parameters defining the spatial constraints of the rendering viewport
Use Cases (5) Point Cloud Compression Figure: Cultural heritage in the 3D Cloud project Key requirements for this application are: Progressive coding to enable increasing quality. Color attributes coding is needed, preferable 8-12 bits per component Generic attributes coding such as for material properties. Lossless is important to enable the best representation when possible Typical point clouds in this use case have the following characteristics: 1 Million upto billions of points (e.g. [13]) Color attributes of 8-12 bits per color component Can contain multiple clusters/groups of points
Use Cases (6) Point Cloud Compression Key requirements for this application are: a) High precision is needed to support localization needs b) Low complexity and/or support for real-time encoding/decoding is needed c) Low delay is needed for real-time communication of dynamic parts of the map d) Region selectivity is important to maintain and access the map data e) Color attributes coding is needed for realistic rendering and visualization f) Additional attributes coding for reflectance and other scene properties Typical point clouds in this use case have the following characteristics: a) Millions to billions of 3D points with up to 1cm precision b) Color attributes with 8-12 bits per color component c) Normals and/or reflectance properties as additional attributes
Use Cases (7) Point Cloud Compression How to adress the different use case requirements and Applications ? Cat 1: static point clouds for cultural heritage and other related 3D photography applications Cat 2: for tele-immersive conferencing and broadcasts (volumetric video) Cat 3: mobile mapping data acquired via non-stationary lidar scanners (last use case) Each category has its own test conditions and test data, it is key to chose your category of interest when you like to do research on point cloud compression
Data sets cat 1 Point Cloud Compression Red and black Egyptian Mask House without a roof Frog Facade Loot
Datasets cat 2 Point Cloud Compression Red and black loot Long dress
Data sets cat 3 Point Cloud Compression
Quality Assessment Point Cloud Compression - Symmetric metrics adapted from mesh compression (symmetric rms, symmetric hausdorff) - point cloud PSNR by using logarithmic scale, instead of peak signal value, average nearest neighbour distance is used to determine the peak value - Subjective testing with a renderer and a camera path based on 60 seconds of uncompressed renderered videos
Objective Metric Point Cloud Compression Decoded coud Original cloud Symmetric distances: the max of D to O and O to D PSNR using scalar based on peak value (based on the size of the bounding cube) 𝑃𝑆𝑁𝑅 = 10 log10 3𝑝2 𝑀𝑆𝐸 𝑑 𝑟𝑚𝑠 𝑉𝑜𝑟 , 𝑉𝑑𝑒𝑔 = 1 √𝐾 ||𝑣𝑙 − 𝑣 𝑛𝑛 _𝑑𝑒𝑔 ||2𝑣 𝑙 ∈𝑉𝑜𝑟 , (4) 𝑑 𝑠𝑦𝑚 _𝑟𝑚𝑠 𝑉𝑜𝑟 , 𝑉𝑑𝑒𝑔 = 𝑚𝑎𝑥⁡( 𝑑 𝑟𝑚𝑠 𝑉𝑜𝑟 , 𝑉𝑑𝑒𝑔 , 𝑑 𝑟𝑚𝑠 𝑉𝑑𝑒𝑔 , 𝑉𝑜𝑟 ) (5) 𝑑ℎ 𝑎𝑢𝑠𝑠 𝑉𝑜𝑟 , 𝑉𝑑𝑒𝑔 = 𝑚𝑎𝑥 𝑣 𝑙 ∈𝑉𝑜𝑟 , ( ||𝑣𝑙 − 𝑣 𝑛𝑛 _𝑑𝑒𝑔 ||2 ) (6) 𝑑 𝑠𝑦𝑚 _ℎ 𝑎𝑢𝑠𝑠 𝑉𝑜𝑟 , 𝑉𝑑𝑒𝑔 = 𝑚𝑎 𝑥 𝑑ℎ 𝑎𝑢𝑠𝑠 𝑉𝑜𝑟 , 𝑉𝑑𝑒𝑔 , 𝑑ℎ 𝑎𝑢𝑠𝑠 𝑉𝑑𝑒𝑔 , 𝑉𝑜𝑟 (7) 𝑝𝑠𝑛𝑟𝑔𝑒𝑜𝑚 = 10𝑙𝑜𝑔10 ( | 𝑚𝑎𝑥 𝑥,𝑦,𝑧 𝑉𝑑𝑒𝑔 |2 2 / (𝑑 𝑠𝑦𝑚 𝑟𝑚𝑠 𝑉𝑜𝑟 , 𝑉𝑑𝑒𝑔 )2 ) (8) 𝑑 𝑦 𝑉𝑜𝑟 , 𝑉𝑑𝑒𝑔 = 1 √𝐾 ||𝑦(𝑣𝑙) − 𝑦(𝑣 𝑛𝑛 _𝑑𝑒𝑔 )||2𝑣 𝑙 ∈𝑉𝑜𝑟 , (9) 𝑝𝑠𝑛𝑟𝑦 = 10𝑙𝑜𝑔10 ( |𝑚𝑎𝑥 𝑦 𝑉𝑑𝑒𝑔 |2 2 / (𝑑 𝑦 𝑉𝑜𝑟 , 𝑉𝑑𝑒𝑔 )2 ) (10) Tool for computing metric available in MPEG, and as open source with the anchor codec: https://github.com/RufaelDev/pcc-mp3dg
Subjective Testing Point Cloud Compression - 60 seconds video - (secret) Camera path - No reference method - Renderer provided by technicolor, no lighting computations - Uncompressed video on HD screen
Common Test conditions cat 1 Point Cloud Compression Class R1 R2 R3 R4 A 0.4 1.25 3.5 6.8 B 0.61 2 4.5 8 C 0.07 0.25 1.0 1.6 Table 1 Target rates in bits per point (geometry + color), for each point cloud class Four rate points lossy geometry + lossy colors, measured in bits per input point bit-rates determined using the anchor codec based on octree
Common Test conditions cat 2 Point Cloud Compression Test Dataset Queen 3 5* 15* 30 55* Loot 3.5* 5 8* 16 27* Redandblack 3.5* 6 9* 18 30* Soldier 3.5* 6 11* 20 37.1* Longdress 3.9* 6 13* 27 42.7* Table 1 Target bitrates for Category 2 in Mbit/s Five rate points lossy geometry + lossy colors, measured in bits per input point, Both all intra and random access conditions
Common Test conditions cat 3 Point Cloud Compression 1. Two test sub-conditions are considered for Category 3, with target rates provided in terms of bits per input point (bpp): Geometry only: The target rates are 24, 12, 8, 4, 2, 1, 0.5 bpp. Geometry & Attributes: The target rates are 32, 16, 12, 6, 3, 1.5, 0.75 bpp. These rates are inclusive of geometry and all associated attribute data. It is mandatory for submissions to this condition for Category 3 to encode both reflectance and color attributes, if present in the input point cloud. 2. No subjective testing This application deals with non visual data and there is no subjective testing involved
Call for Proposals Response Overview Point Cloud Compression - 9 companies including Unified Streaming - HEVC based approaches (4) - Pure point cloud based approaches (e.g. octree based, 4) - Mesh based approach, convert to mesh and compress the mesh (1) - 3 in cat 1, 9 in cat 2 and 1 in cat 3 - HEVC/video based approach for cat 2, octree approach for cat 1 and cat 3 - 8 Mbit’s achieved for transparent cat2 coding - Information is based on respective input contributions and independent validation done by mines telecom, CWI and GBA Tech
Octree Based approaches cat 1/2 Point Cloud Compression 1. Re-coloring (after up-sampling decoded geometry) 2. Region adaptive hierarchical transform (RaHT) transform 3. Improving the colors by avoiding 4:2:0 subsampling in image mapping, interpolation 4. Octree Self similarity coding 5. Proposed attribute coding with Laplacian Sparsity Optimized Graph Transforms 6. Inter prediction using trajectories (octree based coding in category 2)
Re-coloring and upsampling Point Cloud Compression
Color Coding by mapping to HEVC Point Cloud Compression
Geometry coding Point Cloud Compression 1. Surface approximation at higher levels in the octree based on triangles that can be used for upsampling 2. Differential coding of vertices based in a gof 3. RAHT transform for attributes (superseded by HEVC based coding)
Octree Self similarity leaf nodes Point Cloud Compression Diff = 𝑘=0 63 𝑋𝑂𝑅 𝑙𝑛𝑜𝑑𝑒𝐴 𝑘 , 𝑙𝑛𝑜𝑑𝑒𝐵[𝑘] If (Diff <= TH) Merge the two nodes Else Handle the two nodes independently
Color Palette and HEVC based techniques in octree Point Cloud Compression - Reduce the number of colors by k-means (vector quantization) - Code colors by mapping the point attributes to an HEVC grid
Laplacian sparsity optimized attribute coding with GFT Point Cloud Compression 1. Facilitate the Graph transform 2. Best weights, best subgraphs and no isolated components in the graph 3. Kd-tree based partitioning 4. Graph transform (based on graph constrict and eigen value composition of laplacian) 5. Experiments assessing attribute coding performance checking influence of parameters sigma and tau (offline training restricting the ranges (0.1)
HEVC Based approach cat 2 Point Cloud Compression 1. Brute force mapping .ply file to HEVC image (bottom place) 2. Pure projection based (middle) 3. Projection/hybrid octree HEVC based approach (best of rest) 4. Patch based mapping to HEVC (winning proposal)
Direct HEVC Mapping approach Point Cloud Compression 1. Sort the points 2. Directly compress using HEVC codec HEVC 10-Bit Lossless Encoding 3D-Model Point Cloud (Geometry only) Compressed Bitstream Sort and store into 2D video frame Encoder HEVC 10-Bit Decoding 3D-Model Point Cloud (Geometry only) Compressed Bitstream Read point cloud data from 2D video frame Decoder
Direct HEVC approach after sort Point Cloud Compression 1. Sort the points, write to image 2. Directly compress using HEVC codec 10 bit profile HEVC 10-Bit Lossless Encoding 3D-Model Point Cloud (Geometry only) Compressed Bitstream Sort and store into 2D video frame Encoder HEVC 10-Bit Decoding 3D-Model Point Cloud (Geometry only) Compressed Bitstream Read point cloud data from 2D video frame Decoder
Direct HEVC approach after sort Point Cloud Compression 1. Better performance at low bit rates 2. Many artefacts due to occlusions, seems limited 3. Not applicable to category 1
Direct HEVC approach after sort Point Cloud Compression HEVC 10-Bit Encoding Mux 3D-Model Point Cloud Compressed Stream Sort and store into 2D video frame HEVC 8-Bit Encoding Encoder Binary Map Row/Column Subsampling Geometry Color
Projection HEVC Based Point Cloud Compression 2D encoding 3D -> 2D projection Texture plane(s) Geometry plane(s) 2D decoding Texture plane(s) Geometry plane(s) 2D -> 3D projection Input PLY sequence Output PLY sequence Bitstream Projection metadata Projection metadata 1. Projection to images 2. Meta data
Projection approach + HEVC Point Cloud Compression Problems: 1. Occlusions 2. Invalid points
HEVC projection + Octree (1) Point Cloud Compression
HEVC projection + Octree (2) Point Cloud Compression Enable fine grained projections by using octree and binary flags
LCU: Code points As octree instead of projection HEVC projection + Octree (3) Point Cloud Compression
HEVC projection + Octree (4) Point Cloud Compression
HEVC projection + Octree (5) Point Cloud Compression
HEVC projection + Patches Point Cloud Compression Decomposition into patches Packing Geometry image Generation Texture image Generation Attribute image generation Video Compression Occupancy map compression multiplexer Compressed bitstream Input point cloud frame Compression of intra point cloud frames Occupancy map Intra compression Auxiliary patch-info compression Patchinfo Figure 1. Overview of the PCC intra frame compression process.
HEVC projection + Patches Point Cloud Compression
HEVC projection + Patches Point Cloud Compression
HEVC projection + Octree (4) Point Cloud Compression Best results in subjective and objective evaluations test model for cat2 with significant industry support
Mesh based approach cat 2 Point Cloud Compression Good results for simple applications
MPEG Decision Point Cloud Compression - Test models based on octree for cat1 and cat3 content - Test model based on HEVC + Patch for cat 2 (dynamic) - Mesh based could move to a different activity (TBD)
Research directions Point Cloud Compression - HEVC mapping related issues: occlusion points, inter prediction Should be improved, difficult datasets without fixed normals enabling patches - Attribute and large scale point cloud data (tree coding, Graph transforms), out of core processing - Mesh based geometry coding - At some point the transport and storage formats will need attention - Requirements of specific applications and use cases, especially cat 3. mobile mapping should be further improved.
Industry directions Point Cloud Compression - Rapid need for PCC in AR/VR frameworks of mobile devices - Adoption application standards (khronos, gltf, but also ATSC, DVB) - Mapping geospatial market is interesting for large players, but slightly out of traditional MPEG domain - Software and implementation aspects will become key

More Related Content

PPTX
Optimizing the Graphics Pipeline with Compute, GDC 2016
Graham Wihlidal
 
PPTX
A Scalable Real-Time Many-Shadowed-Light Rendering System
Bo Li
 
PDF
Deferred rendering in Dying Light
Maciej Jamrozik
 
PDF
【Unite Tokyo 2019】Unityプログレッシブライトマッパー2019
UnityTechnologiesJapan002
 
PDF
Taking Killzone Shadow Fall Image Quality Into The Next Generation
Guerrilla
 
PDF
Dissecting the Rendering of The Surge
Philip Hammer
 
PDF
Forward+ (EUROGRAPHICS 2012)
Takahiro Harada
 
PDF
Object detection and Instance Segmentation
Hichem Felouat
 
Optimizing the Graphics Pipeline with Compute, GDC 2016
Graham Wihlidal
 
A Scalable Real-Time Many-Shadowed-Light Rendering System
Bo Li
 
Deferred rendering in Dying Light
Maciej Jamrozik
 
【Unite Tokyo 2019】Unityプログレッシブライトマッパー2019
UnityTechnologiesJapan002
 
Taking Killzone Shadow Fall Image Quality Into The Next Generation
Guerrilla
 
Dissecting the Rendering of The Surge
Philip Hammer
 
Forward+ (EUROGRAPHICS 2012)
Takahiro Harada
 
Object detection and Instance Segmentation
Hichem Felouat
 

What's hot (20)

PDF
Checkerboard Rendering in Dark Souls: Remastered by QLOC
QLOC
 
PPT
A Bit More Deferred Cry Engine3
guest11b095
 
PDF
輪読発表資料: Efficient Virtual Shadow Maps for Many Lights
omochi64
 
PPT
Frostbite Rendering Architecture and Real-time Procedural Shading & Texturing...
repii
 
PPTX
Filmic Tonemapping for Real-time Rendering - Siggraph 2010 Color Course
hpduiker
 
PDF
Deferred Rendering in Killzone 2
Guerrilla
 
PPTX
DirectX 11 Rendering in Battlefield 3
Electronic Arts / DICE
 
PDF
HDR Theory and practicce (JP)
Hajime Uchimura
 
PDF
[PR12] You Only Look Once (YOLO): Unified Real-Time Object Detection
Taegyun Jeon
 
PDF
Bindless Deferred Decals in The Surge 2
Philip Hammer
 
PPT
Light prepass
changehee lee
 
PDF
Rendering Techniques in Rise of the Tomb Raider
Eidos-Montréal
 
PPTX
Parallel Graphics in Frostbite - Current & Future (Siggraph 2009)
repii
 
PDF
Point Cloud Compression in MPEG
Marius Preda PhD
 
PPTX
U-Net (1).pptx
Changjin Lee
 
PDF
Advanced Scenegraph Rendering Pipeline
Narann29
 
PDF
中級グラフィックス入門～シャドウマッピング総まとめ～
ProjectAsura
 
PPTX
You only look once (YOLO) : unified real time object detection
Entrepreneur / Startup
 
PDF
Color Science for Games(JP)
Hajime Uchimura
 
PDF
実践的なHDR出力対応 ～レンダリングパイプラインの構築～
Silicon Studio Corporation
 
Checkerboard Rendering in Dark Souls: Remastered by QLOC
QLOC
 
A Bit More Deferred Cry Engine3
guest11b095
 
輪読発表資料: Efficient Virtual Shadow Maps for Many Lights
omochi64
 
Frostbite Rendering Architecture and Real-time Procedural Shading & Texturing...
repii
 
Filmic Tonemapping for Real-time Rendering - Siggraph 2010 Color Course
hpduiker
 
Deferred Rendering in Killzone 2
Guerrilla
 
DirectX 11 Rendering in Battlefield 3
Electronic Arts / DICE
 
HDR Theory and practicce (JP)
Hajime Uchimura
 
[PR12] You Only Look Once (YOLO): Unified Real-Time Object Detection
Taegyun Jeon
 
Bindless Deferred Decals in The Surge 2
Philip Hammer
 
Light prepass
changehee lee
 
Rendering Techniques in Rise of the Tomb Raider
Eidos-Montréal
 
Parallel Graphics in Frostbite - Current & Future (Siggraph 2009)
repii
 
Point Cloud Compression in MPEG
Marius Preda PhD
 
U-Net (1).pptx
Changjin Lee
 
Advanced Scenegraph Rendering Pipeline
Narann29
 
中級グラフィックス入門～シャドウマッピング総まとめ～
ProjectAsura
 
You only look once (YOLO) : unified real time object detection
Entrepreneur / Startup
 
Color Science for Games(JP)
Hajime Uchimura
 
実践的なHDR出力対応 ～レンダリングパイプラインの構築～
Silicon Studio Corporation
 
Ad

Similar to Tutorial on Point Cloud Compression and standardisation (20)

PPTX
Presentation NBMP and PCC
Rufael Mekuria
 
PPTX
XLcloud 3-d remote rendering
Marius Preda PhD
 
PDF
[Paper Presentation] FoveAR: Combining an Optically See-Through Near-Eye Disp...
Pei-Hsuan (Ike) Tsai
 
PDF
Realtime 3D Visualization without GPU
Tobias G
 
PPT
New coding techniques, standardisation, and quality metrics
Touradj Ebrahimi
 
PDF
[03 1][gpu용 개발자 도구 - parallel nsight 및 axe] miller axe
laparuma
 
PPTX
3DgraphicsAndAR
Marius Preda PhD
 
PDF
Interactive Editing of Signed Distance Fields
Matthias Trapp
 
PDF
"Image Sensor Formats and Interfaces for IoT Applications," a Presentation fr...
Edge AI and Vision Alliance
 
PDF
Unite 2013 optimizing unity games for mobile platforms
ナム-Nam Nguyễn
 
PDF
Intelligent internet of things with Google Cloud
Henrik Hammer Eliassen
 
PDF
Keynote (Mike Muller) - Is There Anything New in Heterogeneous Computing - by...
AMD Developer Central
 
PPTX
Sig13 ce future_gfx
Cass Everitt
 
PPT
Introduction to 2D/3D Graphics
Prabindh Sundareson
 
PPTX
Mitchell Reifel (pmdtechnologies ag): pmd Time-of-Flight – the Swiss Army Kni...
AugmentedWorldExpo
 
PPT
CAVE/RC-to-street
Videoguy
 
PDF
10.1.1.184.6612
NITC
 
PPTX
MR Studio Overview for AEC and Engineering
Jonathan Reeves
 
PDF
Dataset creation for Deep Learning-based Geometric Computer Vision problems
PetteriTeikariPhD
 
PDF
Keyshot 2023 Unleashing the Power of Next-Gen 3D Rendering and Animation Soft...
3 HTi - Engineering Software and Solutions
 
Presentation NBMP and PCC
Rufael Mekuria
 
XLcloud 3-d remote rendering
Marius Preda PhD
 
[Paper Presentation] FoveAR: Combining an Optically See-Through Near-Eye Disp...
Pei-Hsuan (Ike) Tsai
 
Realtime 3D Visualization without GPU
Tobias G
 
New coding techniques, standardisation, and quality metrics
Touradj Ebrahimi
 
[03 1][gpu용 개발자 도구 - parallel nsight 및 axe] miller axe
laparuma
 
3DgraphicsAndAR
Marius Preda PhD
 
Interactive Editing of Signed Distance Fields
Matthias Trapp
 
"Image Sensor Formats and Interfaces for IoT Applications," a Presentation fr...
Edge AI and Vision Alliance
 
Unite 2013 optimizing unity games for mobile platforms
ナム-Nam Nguyễn
 
Intelligent internet of things with Google Cloud
Henrik Hammer Eliassen
 
Keynote (Mike Muller) - Is There Anything New in Heterogeneous Computing - by...
AMD Developer Central
 
Sig13 ce future_gfx
Cass Everitt
 
Introduction to 2D/3D Graphics
Prabindh Sundareson
 
Mitchell Reifel (pmdtechnologies ag): pmd Time-of-Flight – the Swiss Army Kni...
AugmentedWorldExpo
 
CAVE/RC-to-street
Videoguy
 
10.1.1.184.6612
NITC
 
MR Studio Overview for AEC and Engineering
Jonathan Reeves
 
Dataset creation for Deep Learning-based Geometric Computer Vision problems
PetteriTeikariPhD
 
Keyshot 2023 Unleashing the Power of Next-Gen 3D Rendering and Animation Soft...
3 HTi - Engineering Software and Solutions
 
Ad

Recently uploaded (20)

PPT
POSITIONING IN OPERATION THEATRE ROOM.ppt
PrabakaranMP1
 
PDF
CAPERS-LRD-z9:AGas-enshroudedLittleRedDotHostingaBroad-lineActive GalacticNuc...
Sérgio Sacani
 
PDF
Biophysics 2.pdffffffffffffffffffffffffff
MiraMusleh
 
PDF
Formation of Supersonic Turbulence in the Primordial Star-forming Cloud
Sérgio Sacani
 
PPTX
Classification Systems_TAXONOMY_SCIENCE8.pptx
glofernignacio
 
PDF
SEHH2274 Organic Chemistry Notes 1 Structure and Bonding.pdf
ivan1108lau
 
PDF
Unveiling a 36 billion solar mass black hole at the centre of the Cosmic Hors...
Sérgio Sacani
 
PPTX
G5Q1W8 PPT SCIENCE.pptx 2025-2026 GRADE 5
Rodel Gordzkie
 
PDF
ELS_Q1_Module-11_Formation-of-Rock-Layers_v2.pdf
savannaesar07
 
PPTX
EPIDURAL ANESTHESIA ANATOMY AND PHYSIOLOGY.pptx
PrabakaranMP1
 
PPTX
2Systematics of Living Organisms t-.pptx
RachanaT6
 
PDF
Phytochemical Investigation of Miliusa longipes.pdf
IrfanShahirSharafi
 
PPTX
Derivatives of integument scales, beaks, horns,.pptx
Dr Showkat Ahmad Wani
 
PPTX
neck nodes and dissection types and lymph nodes levels
DipuSarma
 
PPTX
Protein & Amino Acid Structures Levels of protein structure (primary, seconda...
Muhammad Sajid Afridi
 
PPTX
ANEMIA WITH LEUKOPENIA MDS 07_25.pptx htggtftgt fredrctvg
vedvyas5668
 
PDF
VARICELLA VACCINATION: A POTENTIAL STRATEGY FOR PREVENTING MULTIPLE SCLEROSIS
ijab2
 
PDF
AlphaEarth Foundations and the Satellite Embedding dataset
gdgforscience
 
PDF
IFIT3 RNA-binding activity primores influenza A viruz infection and translati...
saracelre
 
PDF
Sciences of Europe No 170 (2025)
Sciences of Europe
 
POSITIONING IN OPERATION THEATRE ROOM.ppt
PrabakaranMP1
 
CAPERS-LRD-z9:AGas-enshroudedLittleRedDotHostingaBroad-lineActive GalacticNuc...
Sérgio Sacani
 
Biophysics 2.pdffffffffffffffffffffffffff
MiraMusleh
 
Formation of Supersonic Turbulence in the Primordial Star-forming Cloud
Sérgio Sacani
 
Classification Systems_TAXONOMY_SCIENCE8.pptx
glofernignacio
 
SEHH2274 Organic Chemistry Notes 1 Structure and Bonding.pdf
ivan1108lau
 
Unveiling a 36 billion solar mass black hole at the centre of the Cosmic Hors...
Sérgio Sacani
 
G5Q1W8 PPT SCIENCE.pptx 2025-2026 GRADE 5
Rodel Gordzkie
 
ELS_Q1_Module-11_Formation-of-Rock-Layers_v2.pdf
savannaesar07
 
EPIDURAL ANESTHESIA ANATOMY AND PHYSIOLOGY.pptx
PrabakaranMP1
 
2Systematics of Living Organisms t-.pptx
RachanaT6
 
Phytochemical Investigation of Miliusa longipes.pdf
IrfanShahirSharafi
 
Derivatives of integument scales, beaks, horns,.pptx
Dr Showkat Ahmad Wani
 
neck nodes and dissection types and lymph nodes levels
DipuSarma
 
Protein & Amino Acid Structures Levels of protein structure (primary, seconda...
Muhammad Sajid Afridi
 
ANEMIA WITH LEUKOPENIA MDS 07_25.pptx htggtftgt fredrctvg
vedvyas5668
 
VARICELLA VACCINATION: A POTENTIAL STRATEGY FOR PREVENTING MULTIPLE SCLEROSIS
ijab2
 
AlphaEarth Foundations and the Satellite Embedding dataset
gdgforscience
 
IFIT3 RNA-binding activity primores influenza A viruz infection and translati...
saracelre
 
Sciences of Europe No 170 (2025)
Sciences of Europe
 

Tutorial on Point Cloud Compression and standardisation

  • 1. Point Cloud Compression Tutorial Rufael Mekuria (PhD), Unified Streaming and point cloud compression IEEE Visual Communications and Image Processing 2017 St. Petersburg Florida Invited Tutorial rufael@unified-streaming.com Point Cloud Compression
  • 2. About Unified Streaming Unified Streaming Creator of video streaming software behind many of the large scale early deployments of HTTP Streaming (BBC iplayer etc.., NPO) www.unified-streaming.com key implementer of technologies like DASH, HLS, CMAF et… used by CDN, Telco, cloud, broadcaster etc. MPEG affiliate, DASH-IF Member DASH, CMAF, Network based media processing 5G Video streaming pioneer Involved in EU H2020 project Superfluidity with companies like Intel, RedHat and Nokia. Helped many telco/CDN on advanced streaming architectures Point Cloud Compression
  • 3. About Me Academic Background MSEE, Delft 2011, PhD CWI/Vrije Universiteit Amsterdam jan’17 25+ peered reviewed papers, best paper in ACM MMSys’13, 5+ invited papers in conferences like SPIE, ICASSP, etc…. Invited talks: UIUC, Huawei, Google etc… Professional Background TNO (2010-2011), CWI 2011-2016, Unified Streaming 2016-date, PCC (2017-) Active in large scale EU Funded projects: Superfluidity H2020, Reverie FP7, developed one of the first large scale tele-immersive system combining real and CGI content Standardization Work Chaired and pioneered PCC in MPEG (since jan.2014) CfP was responded by 9 companies in oct. 2017 including all major mobile device vendors (CfP response referred to as “Historic” by MPEG convenor). Point Cloud Compression
  • 4. About this Tutorial Before the break: Introduction to this (re-) emerging research area, point clouds for VR, point cloud compression architecture After the break: Standardization activity on point cloud compression in MPEG Applications and requirements, quality assessments State of art and research challenges Copyrights & Acknowledgement: Some of the images and parts of this tutorial fall under copyright of respective contributors/authors as noted to MPEG CfP. I do not name the contributors explicitly for now. Many opinions in this work are derived from standardization meetings representing rough industry consensus. I acknowledge 8i, Microsoft, Technicolor, Ford, Mitshubishi, et al. for contribution of test data. I thank Technicolor for large contributions to this body of work. Point Cloud Compression
  • 5.  A collection of points  Not related to each other  Typically no order  Typically no local topology (no mesh!)  Each is point is the given of  a position (X,Y,Z)  a color (R,G,B) or (Y,U,V)  possibly other things like transparency, time of acquisition, etc. Point Cloud Compression Point cloud content from Microsoft research laboratory Donated to MPEG Point Cloud Format
  • 6. .ply files = an example raw data format for point cloud  This is an example raw point coud file format (compare to YUV for video coding)  How many points?  Static case up to several tens of millions, depending on the application  Dynamic case ~1 million per frame, 30 fps  Probably more is needed for good VR  Format  Geometry XYZ  Fixed precision for VR applications  Float still often used  Colors RGB  As usual, integer 8/10 bits.  Possibly other attributes  (not present in the ply file here on the left) plyformat ascii 1.0 element vertex 764940 property float x property float y property float z property uchar red property uchar green property uchar blue end_header 211 63 63 127 98 73 213 63 61 134 109 87 212 62 63 122 97 75 212 63 62 129 102 79 212 63 63 124 98 76 213 62 63 122 98 76 213 63 62 128 104 81 213 63 63 124 99 78 215 61 63 120 97 76 214 63 60 141 117 95 214 63 61 135 111 89 215 63 60 144 120 97 215 63 61 133 109 87 214 62 62 126 102 80 214 62 63 122 98 77 214 63 62 128 104 82 etc. one point X Y Z R G B no order! Swapping points does not change the data Point Cloud Compression
  • 7. An application of point cloud: free-view point (6DoF) for sport  Scene model  360°/omnidirectional background  reshaping depending on viewpoint  3D object  occlusion, parallax (in HMD)  position relatively to the background  Free-view path  viewer body position freely chosen on the free-view path  + free head movement (in HMD) 360° background 3D objects free- view path https://www.youtube.com/watch?v=Q-LNA9KlHhw Point Cloud Compression
  • 8. Why Point Cloud? (for VR)  No occlusions  all angle of views are acceptable => parallax  free-view point (=6 DoF VR) is natively supported  Fine topology  volumes, hairs, fur can be represented by a point cloud  naturally captured by sensors without heavy processing  usually deduced from depth and/or disparity from multi-view capture
  • 9. Point Cloud Representation in Microsoft holoportation
  • 10. Real and Virtual Engagement in Realistic Immersive Environments Point Cloud Compression/transmission: Immersive Communications (2014) in Reverie FP7 Highly realistic representation for immersive communications reveriefp7.eu Human is reconstructed as a photo realistic 3D Cloud (or mesh) of Points in a 3D space! Challenges: Low bit rate, real-time encoding, color coding, inter frame coding, scalability
  • 11. Point Clouds for high-end AR/VR  Key Requirements for a 3DoF+/6DoF VR/AR format  Support of stereo imaging with view dependent parallax  360 video ruled out  Universal applicability  Effective handling of occlusions  2D plus depth ruled out  Easy acquisition & Rendering  Candidates: (Super-) Multi-view, Point Cloud, (Mesh)  Comparing MV and PC  Multi-view + Easy acquisition and existing compression technology - Number of views constraints occlusions and viewing angle  Combining with CGI models difficult – requires methods similar to point cloud creation  Point Cloud + Most versatile – works for live-action acquisitions as well as CGI. Composition of scenes is easy (just “cat” point clouds) + Occlusions only depend on acquisition technology o Known but comparably complex approaches for acquisition - No efficient compression technology yet
  • 12. How to capture point clouds?  Multi camera setups  Depth estimation/measurement  Color/feature detection  3D reconstruction  3D Modelling
  • 13. High Quality (studio level?) content
  • 14. Low Quality Setup (3D skype ?)
  • 15. Algorithms for point cloud reconstruction - From multiple depth images (Reverie project) - From rigs with stereo cameras (microsoft) - From multiple images (3D culture cloud) - Mobile devices (new sony experia for example) - Systems like 8i or owlii - Systems for point cloud capture will be key if pcc will be an impotrant medium For delivery
  • 16. How to render point clouds?  Giving size to points  Splats, rectangles, cubes (=3D pixels)  Trade-off size vs. texture high frequency  Meshing, and using illumation techniques  A demo using PCC contents (and renderer)  8i content  Technicolor based rendering
  • 17. Rendering Demonstration MPEG Data and rendering, data by 8i
  • 18. How to compress point clouds: examples of known technologies  How to compress PC geometry ?  octree-based  occupancy data to be entropy coded using prediction  intra local prediction (local plane, etc.)  inter-prediction (motion)  image based  depth coding but unable to handle occlusions  global/local  global envelope for surfacic objects, then geometrical residual  similar to mesh + height  How to compress PC colors ?  octree-based  palette in an octree, prediction and residual coding  wavelets on trees  image-based  projection on planes/surfaces, compression using a 2D video codec  local projection and tiling/packing in a unique image to avoid loss of occlusions due to global projections  block-based (mimicking video compression)  3D blocks, push points in a corner, prediction, 3D-DCT, quantization and entropy coding  inter prediction with 3D motion vectors
  • 19. Comparison with other format compression  Performance comparison with other formats  in bit per pixel/points to be displayed (bpp) Compression format (lossy, good quality) bpp 2D flat UHD, intra ~0.25-0.5 2D flat UHD, inter ~0.025-0.1 2D + smooth depth 2D flat + 25% (for parallax) Multi-View 2D*nb views*75% Point Cloud geo ~1-3, texture ~0.5-2 Mesh geo ~0.25, texture ~0.5 Light-field unknown yet  Challenges ahead  improve PC compression efficiency  in particular for inter coding  find a robust method to assess compressed geometry and texture quality  Industry wants fast adoption, re-use of existing hardware infrastructure (e.g. HEVC, AVC) will be important
  • 20. Questions (round 1) ?? Point Cloud Compression
  • 21. Design of PCC Anchor Point Cloud Compression - Design for Reverie Immersive online platform - Replicant (3D Point Cloud Reconstruction) by CERTH-ITI - Real-time requirements - Different quality representations (for rendering at different distances) - Publication: R. Mekuria, K. Blom and P. Cesar, "Design, Implementation, and Evaluation of a Point Cloud Codec for Tele-Immersive Video," in IEEE Transactions on Circuits and Systems for Video Technology, vol. 27, no. 4, pp. 828-842, April 2017. doi: 10.1109/TCSVT.2016.2543039 - Experimental work, codec used as anchor for MPEG PCC
  • 22. Reverie Immersive Framwork Media Router Media Router Large Scale Tele-Immersive Architecture User Analysis Scene structuring & NavigationScene structuring & NavigationServer Composition Renderers Content Decoding Stream Synchronization 3D Capturing 3D Reconstruction Scalable Content Coding Streaming User Generated Objects Network Monitoring Session Manager Session Set-up Social Network Media Coordinator Avatar Embodiment Media Router Stream Selector & Media Objects Audio Streams Perception Cognition Animation Avatar Reasoning Media Stream Demux Visual Streams Client Media Router Network Monitoring Spatial & Audio Composition
  • 23. Real and Virtual Engagement in Realistic Immersive Environments Point Cloud Compression: Immersive Communications Replicant (immersive communications) reveriefp7.eu Human is reconstructed as a photo realistic 3D Cloud (or mesh) of Points in a 3D space! Challenges: Low bit rate, real-time encoding, color coding, inter frame coding, scalability
  • 24. Architecture Hybrid Octree point cloud codec Point Cloud Compression
  • 25. Point Cloud Geometry Compression recursive sub-divsions: complexity O(2^(N)) Limit level N and differentially code of points in larger leafs for real-time coding [Kammerl12] Prediction of subdivisions based on the previous level [Schnabel06, Huang06] Context Adaptive Entropy encoding [Schnabel06, Huang06] range coding [Kammerl12] 10000000 10000000 [Kammerl12] Kammerl, J.; Blodow, N.; Rusu, R.B.; Gedikli, S.; Beetz, M.; Steinbach, E., "Real- time compression of point cloud streams," Robotics and Automation (ICRA), 2012 IEEE International Conference on , vol., no., pp.778,785, 14-18 May 2012 [Schnabel06] Ruwen Schnabel and Reinhard Klein. 2006. Octree-based point-cloud compression. In Proceedings of the 3rd Eurographics / IEEE VGTC conference on Point-Based Graphics (SPBG'06), [Huang06]Huang Yan Huang, Jingliang Peng, C.-C. Jay Kuo, and M. Gopi. 2006. Octree-based progressive geometry coding of point clouds. In Proceedings of the 3rd Eurographics / IEEE VGTC conference on Point-Based Graphics (SPBG'06) Point Cloud Compression
  • 26. Real and Virtual Engagement in Realistic Immersive Environments Point Cloud (Color) Attribute Compression DPCM [Kammerl12] Colorization [Huang06] Octree Based [schnabel06] Based on Graph Transform [Zhang14] Mapping to jpeg image grid ?? Table 1 scan order for mapping octree centroid colours to image grid (8x10 sample block) 0 1 2 3 4 5 6 7 64 … 15 14 13 12 11 10 9 8 79 … 16 17 18 19 20 21 22 23 80 … 31 30 29 28 27 26 25 24 95 … 32 33 34 35 36 37 38 39 … … 47 46 45 44 43 42 41 40 … … 48 49 50 51 52 53 54 55 … … 63 62 61 60 59 58 57 56 … … 0 0.5 1 1.5 2 2.5 3 dense octree (11) sparse octree (9) bitrate(bytesperoutput vertex/voxel) Achieved Compression Ratio original (5 bits colors) proposed (jpeg mapping) Overall coding gain compared to legacy point cloud codec available in PCL (at comparable objective quality) Traverse octree Write color attributes to an image grid JPEG Encode
  • 27. Real and Virtual Engagement in Realistic Immersive Environments Inter Predictive Coding of Octree Compressed Point Clouds Basic Algorithm Overview 1. Input Cloud I and Input Cloud P 2. Align bounding box Cloud I and P 3. Compute octrees of I and P at level N – M 4. Find common leafs at level N-M occupied in I and P 5. Try to predict the vertices in leafs in P from leafs in I 6. If P can be predicted, code vertices as a rigid transform on the input vertices 7. Code all other vertices in P via an intra coding scheme Algorithm Details 1. Color variance and point count used to decide wether or not to do the prediction 2. Compute prediction via ICP procedure (iterative closest points) 3. Compress the rigid transform as a quaternion or 2 vectors 4. Use an octree coding upto level N to code the points that cannot be predicted
  • 28. Real and Virtual Engagement in Realistic Immersive Environments Bounding Box Alignment I input cloud compute BB compute BB P input cloud expand BB I normalize I on BB_IE normalize P on BB_IE P Coding of P I Coding of I expand BB P normalize P on BB_PE I Coding of P BB_P fits BB_IE BB_I BB_P BB_IE BB_IE N BB_IE Y BB_PE
  • 29. Real and Virtual Engagement in Realistic Immersive Environments Inter prediction Algorithm normalized I Cloud normalized P Cloud Generate Macroblocks of I and P For each Macroblock in P corr macroblock in I ? color_var < TRESH icp converged ? encode rigid transform fittness < thresh ? Intra Encoding of Macro Block Intra Encoding of Macro Block Intra Encoding of Macro Block store key, rigid tf, color_offset I coded part p coded part Compute rigid transform via ICP between corr Macroblocks I and P N Y Y N Y N N Y # of points ok ? 1. 2. 3. 4. 5. 6. 7. 8. 15 bytes N Y N-M Octree !!
  • 30. Real and Virtual Engagement in Realistic Immersive Environments Results: Bounding Box Alignment Alex at 12 fps, Alex at 24 fps Bounding box factor Alignment % (12 fps) Alignment % (24 fps) 5 % 44 65 10 % 68 76 15 % 72 83 20 % 74 83 25 % 82 88
  • 31. Real and Virtual Engagement in Realistic Immersive Environments Shared Macroblocks, convergence percentage Alex at 24 fps (100 frames) Alex at 12 fps (50 frames), bounding box factor 20 % 11 level octree Dataset % shared macroblocks % ICP converged % points p coded (apr.) Alex (12 fps) 65 % 32 % ~ 25 % Alex (24 fps) 75 % 35 % ~ 30 % Christos (12 fps) 66 % 34 % ~ 30 % Christos (24 fps) 78 % 40 % ~ 35 % Dimitrios (12 fps) 69,9% 39 % ~ 35 % Dimitrios (24 fps) 78 % 42 % ~ 40 %
  • 32. Real and Virtual Engagement in Realistic Immersive Environments Point Cloud Compression, Quality Assessment (m36529) Table 2 – MOS on the perceived quality d_sym_rms Symmetric rms distance betw. Clouds (5) d_sym_haussdorf Symmetric Haussdorf dist. betw. clouds (7) psnr_geom PSNR(vertexpositions) (8) psnr_y PSNR(colorsY) (10) psnr_u PSNR (colors U) (as 10 for u) psnr_v PSNR (colors V) (as 10 for v) 𝑑 𝑟𝑚𝑠 𝑉𝑜𝑟 , 𝑉𝑑𝑒𝑔 = 1 √𝐾 ||𝑣𝑙 − 𝑣 𝑛𝑛 _𝑑𝑒𝑔 ||2𝑣 𝑙 ∈𝑉𝑜𝑟 , (4) 𝑑 𝑠𝑦𝑚 _𝑟𝑚𝑠 𝑉𝑜𝑟 , 𝑉𝑑𝑒𝑔 = 𝑚𝑎𝑥⁡( 𝑑 𝑟𝑚𝑠 𝑉𝑜𝑟 , 𝑉𝑑𝑒𝑔 , 𝑑 𝑟𝑚𝑠 𝑉𝑑𝑒𝑔 , 𝑉𝑜𝑟 ) (5) 𝑑ℎ 𝑎𝑢𝑠𝑠 𝑉𝑜𝑟 , 𝑉𝑑𝑒𝑔 = 𝑚𝑎𝑥 𝑣 𝑙 ∈𝑉𝑜𝑟 , ( ||𝑣𝑙 − 𝑣 𝑛𝑛 _𝑑𝑒𝑔 ||2 ) (6) 𝑑 𝑠𝑦𝑚 _ℎ 𝑎𝑢𝑠𝑠 𝑉𝑜𝑟 , 𝑉𝑑𝑒𝑔 = 𝑚𝑎 𝑥 𝑑ℎ 𝑎𝑢𝑠𝑠 𝑉𝑜𝑟 , 𝑉𝑑𝑒𝑔 , 𝑑ℎ 𝑎𝑢𝑠𝑠 𝑉𝑑𝑒𝑔 , 𝑉𝑜𝑟 (7) 𝑝𝑠𝑛𝑟𝑔𝑒𝑜𝑚 = 10𝑙𝑜𝑔10 ( | 𝑚𝑎𝑥 𝑥,𝑦,𝑧 𝑉𝑑𝑒𝑔 |2 2 / (𝑑 𝑠𝑦𝑚 𝑟𝑚𝑠 𝑉𝑜𝑟 , 𝑉𝑑𝑒𝑔 )2 ) (8) 𝑑 𝑦 𝑉𝑜𝑟 , 𝑉𝑑𝑒𝑔 = 1 √𝐾 ||𝑦(𝑣𝑙) − 𝑦(𝑣 𝑛𝑛 _𝑑𝑒𝑔 )||2𝑣 𝑙 ∈𝑉𝑜𝑟 , (9) 𝑝𝑠𝑛𝑟𝑦 = 10𝑙𝑜𝑔10 ( |𝑚𝑎𝑥 𝑦 𝑉𝑑𝑒𝑔 |2 2 / (𝑑 𝑦 𝑉𝑜𝑟 , 𝑉𝑑𝑒𝑔 )2 ) (10) We compare the output point to the nearest input point and vice versa!
  • 33. Real and Virtual Engagement in Realistic Immersive Environments Results Inter Prediction For Alex, Chris and Dimitrios > 65 % of blocks are shared and ~ 30 percent can be predicted Jog and ski don’t work well because outlier points mess up the bounding box algorithm 0 1 2 3 4 5 alex chris dim Bytesperoutputvertex Dataset (ITI/CERTH, 24 fps) bit rate gain predictive coding intra pred 60 65 70 75 80 alex chris dim Quality[dB] Dataset (ITI/CERTH 24 fps) Quality Geometry [dB] intra pred 0 200 400 600 800 alex chris dim Encodertime[ms] Dataset Encoder Time intra pred
  • 34. Real and Virtual Engagement in Realistic Immersive Environments PCC Reference Software (m36527) http://wg11.sc29.org/svn/repos/MPEG-04/Part16- Animation_Framework_eXtension_(AFX)/trunk/3Dgraphics/3DG- PCC/trunk/pcc-mp3dg (official mpeg version) https://github.com/RufaelDev/pcc-mp3dg (open source) Tested: Win7, Mac, Ubuntu Linux Cloud Codec V2 compression_eval Quality
  • 35. Real and Virtual Engagement in Realistic Immersive Environments Octree Based Point Cloud Table 2 – MOS on the perceived quality
  • 36. Real and Virtual Engagement in Realistic Immersive Environments Coding Color Attributes with JPEG Table 2 – MOS on the perceived quality
  • 37. Real and Virtual Engagement in Realistic Immersive Environments Temporal Prediction via ICP Table 2 – MOS on the perceived quality
  • 38. Questions (round 2) ?? Point Cloud Compression
  • 39. Standardization of PCC Outline Point Cloud Compression - PCC Exploration in MPEG - Use Cases / Requirements - Test Datasets - Quality Assessment - Common Test Conditions - Rendering - Anchor - Call for Proposals (10 responses, october 2017)
  • 40. MPEG Standardization Point Cloud Compression - Industry consortium for developing standardized specification under ISO/IEC - Meets 3-4 times per year - Joint work with ITU on video coding (AVC, HEVC) - Much used standards in the media industry - Still relatively strong attendance (around 300 per meeting) - PCC AhG group under 3D Graphics groups (MPEG only) - Next Meeting in Gwanju, Korea, in January
  • 41. Exploration on PCC/3DGC Point Cloud Compression - Started in 2014 in San Diego Meeting - Naturalistic content - Virtual Reality use case (not yet popular back then) - Techniques for point clouds in addition to meshes - Evaluation methodology (objective, subjective metrics) - Anchor codec framework - Work done as PhD student at CWI - Contributions from Reverie EU project - Discussion with major industries in 2016 on the requirements and methodology (e.g. Apple, Nokia, Huawei, Technicolor, 8i, owlii, Orange, Samsung, Orange, Canon, Sony, Panasonic etc..) - Call for Proposals in Jan. 2017, (10 responses, october 2017) - Industry standard specification expected by mid 2019
  • 42. Table 2 – MOS on the perceived quality Point cloudImages Point Cloud Compression in VR Ecosystem (1) Point Cloud Compression
  • 43. Use Cases (1) Point Cloud Compression - High Quality Broadcast of content (volumetric video) - Tele-immersive mixed reality (real-time volumetric video), both AR/VR like - Cultural Heritage, 3D photography, CAD scans - Mobile mapping data - Large scale geographic information - https://mpeg.chiariglione.org/standards/exploration/point-cloud- compression/use-cases-point-cloud-compression
  • 44. Use cases (2) Gis: Lidar: http://www.cadalyst.com/cadalyst/gis-tech-news-99-13394 Robotics: lidar, kinect: www.pointclouds.org GIS: Aereal Imaging http://www.flightlinegeographics.com/3D_Point_Cloud.html Cultural heritage (images): Culture3D Cloud Mixed Reality
  • 45. Use Cases (3) Point Cloud Compression 3D Recon Struction Software Reconstructed 3D Human 3D Point Cloud Representation 3D Source Encoding IP Network Multi Depth Camera Capture Or other 3D Capture Real-Time 3D Rendering Composition In Virtual World Packetization & Transmission 3D Source Decoding N RGB + Depth Images Or other sensor data Reception & Synchronization Figure 1: Transmission pipeline for conferencing with 3D geometry Key requirements for this application are: Lossy compression with bit-rate control is needed low complexity and/or support for real-time encoding/ decoding is needed error resilience to transmission errors is needed color attributes coding is needed for realistic rendering material/appearance related attributes coding is needed to code additional attributes to support the rendering view dependence: for streaming optimization view dependence can be used to optimize the transmission process Typical point clouds in this use case have the following characteristics: Between 100,000 and 10,000,000 points to represent a reconstructed human color attributes with 8-10 bits per color component Normals and or material properties to support the rendering using a shader
  • 46. Use Cases (4) Point Cloud Compression Key requirements for this application are: Lossy compression with bit-rate control is needed error resilience to transmission errors is needed color attributes coding is needed for realistic rendering material/appearance related attributes coding is needed to code additional attributes to support the rendering view dependence: the view dependence can be used to optimize the encoding process Typical point clouds in this use case have the following characteristics: Between 100,000 and 10,000,000 points to represent closeby objects in the scene, Color attributes with 8-10 bits per color component Global parameters defining the spatial constraints of the rendering viewport
  • 47. Use Cases (5) Point Cloud Compression Figure: Cultural heritage in the 3D Cloud project Key requirements for this application are: Progressive coding to enable increasing quality. Color attributes coding is needed, preferable 8-12 bits per component Generic attributes coding such as for material properties. Lossless is important to enable the best representation when possible Typical point clouds in this use case have the following characteristics: 1 Million upto billions of points (e.g. [13]) Color attributes of 8-12 bits per color component Can contain multiple clusters/groups of points
  • 48. Use Cases (6) Point Cloud Compression Key requirements for this application are: a) High precision is needed to support localization needs b) Low complexity and/or support for real-time encoding/decoding is needed c) Low delay is needed for real-time communication of dynamic parts of the map d) Region selectivity is important to maintain and access the map data e) Color attributes coding is needed for realistic rendering and visualization f) Additional attributes coding for reflectance and other scene properties Typical point clouds in this use case have the following characteristics: a) Millions to billions of 3D points with up to 1cm precision b) Color attributes with 8-12 bits per color component c) Normals and/or reflectance properties as additional attributes
  • 49. Use Cases (7) Point Cloud Compression How to adress the different use case requirements and Applications ? Cat 1: static point clouds for cultural heritage and other related 3D photography applications Cat 2: for tele-immersive conferencing and broadcasts (volumetric video) Cat 3: mobile mapping data acquired via non-stationary lidar scanners (last use case) Each category has its own test conditions and test data, it is key to chose your category of interest when you like to do research on point cloud compression
  • 50. Data sets cat 1 Point Cloud Compression Red and black Egyptian Mask House without a roof Frog Facade Loot
  • 51. Datasets cat 2 Point Cloud Compression Red and black loot Long dress
  • 52. Data sets cat 3 Point Cloud Compression
  • 53. Quality Assessment Point Cloud Compression - Symmetric metrics adapted from mesh compression (symmetric rms, symmetric hausdorff) - point cloud PSNR by using logarithmic scale, instead of peak signal value, average nearest neighbour distance is used to determine the peak value - Subjective testing with a renderer and a camera path based on 60 seconds of uncompressed renderered videos
  • 54. Objective Metric Point Cloud Compression Decoded coud Original cloud Symmetric distances: the max of D to O and O to D PSNR using scalar based on peak value (based on the size of the bounding cube) 𝑃𝑆𝑁𝑅 = 10 log10 3𝑝2 𝑀𝑆𝐸 𝑑 𝑟𝑚𝑠 𝑉𝑜𝑟 , 𝑉𝑑𝑒𝑔 = 1 √𝐾 ||𝑣𝑙 − 𝑣 𝑛𝑛 _𝑑𝑒𝑔 ||2𝑣 𝑙 ∈𝑉𝑜𝑟 , (4) 𝑑 𝑠𝑦𝑚 _𝑟𝑚𝑠 𝑉𝑜𝑟 , 𝑉𝑑𝑒𝑔 = 𝑚𝑎𝑥⁡( 𝑑 𝑟𝑚𝑠 𝑉𝑜𝑟 , 𝑉𝑑𝑒𝑔 , 𝑑 𝑟𝑚𝑠 𝑉𝑑𝑒𝑔 , 𝑉𝑜𝑟 ) (5) 𝑑ℎ 𝑎𝑢𝑠𝑠 𝑉𝑜𝑟 , 𝑉𝑑𝑒𝑔 = 𝑚𝑎𝑥 𝑣 𝑙 ∈𝑉𝑜𝑟 , ( ||𝑣𝑙 − 𝑣 𝑛𝑛 _𝑑𝑒𝑔 ||2 ) (6) 𝑑 𝑠𝑦𝑚 _ℎ 𝑎𝑢𝑠𝑠 𝑉𝑜𝑟 , 𝑉𝑑𝑒𝑔 = 𝑚𝑎 𝑥 𝑑ℎ 𝑎𝑢𝑠𝑠 𝑉𝑜𝑟 , 𝑉𝑑𝑒𝑔 , 𝑑ℎ 𝑎𝑢𝑠𝑠 𝑉𝑑𝑒𝑔 , 𝑉𝑜𝑟 (7) 𝑝𝑠𝑛𝑟𝑔𝑒𝑜𝑚 = 10𝑙𝑜𝑔10 ( | 𝑚𝑎𝑥 𝑥,𝑦,𝑧 𝑉𝑑𝑒𝑔 |2 2 / (𝑑 𝑠𝑦𝑚 𝑟𝑚𝑠 𝑉𝑜𝑟 , 𝑉𝑑𝑒𝑔 )2 ) (8) 𝑑 𝑦 𝑉𝑜𝑟 , 𝑉𝑑𝑒𝑔 = 1 √𝐾 ||𝑦(𝑣𝑙) − 𝑦(𝑣 𝑛𝑛 _𝑑𝑒𝑔 )||2𝑣 𝑙 ∈𝑉𝑜𝑟 , (9) 𝑝𝑠𝑛𝑟𝑦 = 10𝑙𝑜𝑔10 ( |𝑚𝑎𝑥 𝑦 𝑉𝑑𝑒𝑔 |2 2 / (𝑑 𝑦 𝑉𝑜𝑟 , 𝑉𝑑𝑒𝑔 )2 ) (10) Tool for computing metric available in MPEG, and as open source with the anchor codec: https://github.com/RufaelDev/pcc-mp3dg
  • 55. Subjective Testing Point Cloud Compression - 60 seconds video - (secret) Camera path - No reference method - Renderer provided by technicolor, no lighting computations - Uncompressed video on HD screen
  • 56. Common Test conditions cat 1 Point Cloud Compression Class R1 R2 R3 R4 A 0.4 1.25 3.5 6.8 B 0.61 2 4.5 8 C 0.07 0.25 1.0 1.6 Table 1 Target rates in bits per point (geometry + color), for each point cloud class Four rate points lossy geometry + lossy colors, measured in bits per input point bit-rates determined using the anchor codec based on octree
  • 57. Common Test conditions cat 2 Point Cloud Compression Test Dataset Queen 3 5* 15* 30 55* Loot 3.5* 5 8* 16 27* Redandblack 3.5* 6 9* 18 30* Soldier 3.5* 6 11* 20 37.1* Longdress 3.9* 6 13* 27 42.7* Table 1 Target bitrates for Category 2 in Mbit/s Five rate points lossy geometry + lossy colors, measured in bits per input point, Both all intra and random access conditions
  • 58. Common Test conditions cat 3 Point Cloud Compression 1. Two test sub-conditions are considered for Category 3, with target rates provided in terms of bits per input point (bpp): Geometry only: The target rates are 24, 12, 8, 4, 2, 1, 0.5 bpp. Geometry & Attributes: The target rates are 32, 16, 12, 6, 3, 1.5, 0.75 bpp. These rates are inclusive of geometry and all associated attribute data. It is mandatory for submissions to this condition for Category 3 to encode both reflectance and color attributes, if present in the input point cloud. 2. No subjective testing This application deals with non visual data and there is no subjective testing involved
  • 59. Call for Proposals Response Overview Point Cloud Compression - 9 companies including Unified Streaming - HEVC based approaches (4) - Pure point cloud based approaches (e.g. octree based, 4) - Mesh based approach, convert to mesh and compress the mesh (1) - 3 in cat 1, 9 in cat 2 and 1 in cat 3 - HEVC/video based approach for cat 2, octree approach for cat 1 and cat 3 - 8 Mbit’s achieved for transparent cat2 coding - Information is based on respective input contributions and independent validation done by mines telecom, CWI and GBA Tech
  • 60. Octree Based approaches cat 1/2 Point Cloud Compression 1. Re-coloring (after up-sampling decoded geometry) 2. Region adaptive hierarchical transform (RaHT) transform 3. Improving the colors by avoiding 4:2:0 subsampling in image mapping, interpolation 4. Octree Self similarity coding 5. Proposed attribute coding with Laplacian Sparsity Optimized Graph Transforms 6. Inter prediction using trajectories (octree based coding in category 2)
  • 61. Re-coloring and upsampling Point Cloud Compression
  • 62. Color Coding by mapping to HEVC Point Cloud Compression
  • 63. Geometry coding Point Cloud Compression 1. Surface approximation at higher levels in the octree based on triangles that can be used for upsampling 2. Differential coding of vertices based in a gof 3. RAHT transform for attributes (superseded by HEVC based coding)
  • 64. Octree Self similarity leaf nodes Point Cloud Compression Diff = 𝑘=0 63 𝑋𝑂𝑅 𝑙𝑛𝑜𝑑𝑒𝐴 𝑘 , 𝑙𝑛𝑜𝑑𝑒𝐵[𝑘] If (Diff <= TH) Merge the two nodes Else Handle the two nodes independently
  • 65. Color Palette and HEVC based techniques in octree Point Cloud Compression - Reduce the number of colors by k-means (vector quantization) - Code colors by mapping the point attributes to an HEVC grid
  • 66. Laplacian sparsity optimized attribute coding with GFT Point Cloud Compression 1. Facilitate the Graph transform 2. Best weights, best subgraphs and no isolated components in the graph 3. Kd-tree based partitioning 4. Graph transform (based on graph constrict and eigen value composition of laplacian) 5. Experiments assessing attribute coding performance checking influence of parameters sigma and tau (offline training restricting the ranges (0.1)
  • 67. HEVC Based approach cat 2 Point Cloud Compression 1. Brute force mapping .ply file to HEVC image (bottom place) 2. Pure projection based (middle) 3. Projection/hybrid octree HEVC based approach (best of rest) 4. Patch based mapping to HEVC (winning proposal)
  • 68. Direct HEVC Mapping approach Point Cloud Compression 1. Sort the points 2. Directly compress using HEVC codec HEVC 10-Bit Lossless Encoding 3D-Model Point Cloud (Geometry only) Compressed Bitstream Sort and store into 2D video frame Encoder HEVC 10-Bit Decoding 3D-Model Point Cloud (Geometry only) Compressed Bitstream Read point cloud data from 2D video frame Decoder
  • 69. Direct HEVC approach after sort Point Cloud Compression 1. Sort the points, write to image 2. Directly compress using HEVC codec 10 bit profile HEVC 10-Bit Lossless Encoding 3D-Model Point Cloud (Geometry only) Compressed Bitstream Sort and store into 2D video frame Encoder HEVC 10-Bit Decoding 3D-Model Point Cloud (Geometry only) Compressed Bitstream Read point cloud data from 2D video frame Decoder
  • 70. Direct HEVC approach after sort Point Cloud Compression 1. Better performance at low bit rates 2. Many artefacts due to occlusions, seems limited 3. Not applicable to category 1
  • 71. Direct HEVC approach after sort Point Cloud Compression HEVC 10-Bit Encoding Mux 3D-Model Point Cloud Compressed Stream Sort and store into 2D video frame HEVC 8-Bit Encoding Encoder Binary Map Row/Column Subsampling Geometry Color
  • 72. Projection HEVC Based Point Cloud Compression 2D encoding 3D -> 2D projection Texture plane(s) Geometry plane(s) 2D decoding Texture plane(s) Geometry plane(s) 2D -> 3D projection Input PLY sequence Output PLY sequence Bitstream Projection metadata Projection metadata 1. Projection to images 2. Meta data
  • 73. Projection approach + HEVC Point Cloud Compression Problems: 1. Occlusions 2. Invalid points
  • 74. HEVC projection + Octree (1) Point Cloud Compression
  • 75. HEVC projection + Octree (2) Point Cloud Compression Enable fine grained projections by using octree and binary flags
  • 76. LCU: Code points As octree instead of projection HEVC projection + Octree (3) Point Cloud Compression
  • 77. HEVC projection + Octree (4) Point Cloud Compression
  • 78. HEVC projection + Octree (5) Point Cloud Compression
  • 79. HEVC projection + Patches Point Cloud Compression Decomposition into patches Packing Geometry image Generation Texture image Generation Attribute image generation Video Compression Occupancy map compression multiplexer Compressed bitstream Input point cloud frame Compression of intra point cloud frames Occupancy map Intra compression Auxiliary patch-info compression Patchinfo Figure 1. Overview of the PCC intra frame compression process.
  • 80. HEVC projection + Patches Point Cloud Compression
  • 81. HEVC projection + Patches Point Cloud Compression
  • 82. HEVC projection + Octree (4) Point Cloud Compression Best results in subjective and objective evaluations test model for cat2 with significant industry support
  • 83. Mesh based approach cat 2 Point Cloud Compression Good results for simple applications
  • 84. MPEG Decision Point Cloud Compression - Test models based on octree for cat1 and cat3 content - Test model based on HEVC + Patch for cat 2 (dynamic) - Mesh based could move to a different activity (TBD)
  • 85. Research directions Point Cloud Compression - HEVC mapping related issues: occlusion points, inter prediction Should be improved, difficult datasets without fixed normals enabling patches - Attribute and large scale point cloud data (tree coding, Graph transforms), out of core processing - Mesh based geometry coding - At some point the transport and storage formats will need attention - Requirements of specific applications and use cases, especially cat 3. mobile mapping should be further improved.
  • 86. Industry directions Point Cloud Compression - Rapid need for PCC in AR/VR frameworks of mobile devices - Adoption application standards (khronos, gltf, but also ATSC, DVB) - Mapping geospatial market is interesting for large players, but slightly out of traditional MPEG domain - Software and implementation aspects will become key

Editor's Notes

  • #10: For comparison purpose we set the SC3DMC coders to code with 8 bits and differential coding options. We compared the orignal captured meshes to the meshes decoded using a tool to measure the symmetric distance between the surfaces. This metric is often known as haussdorf distance. A set of live captured models compressed and decoded shows that the qualities are comparible. Note that lower value implies better quality.
  • #11: For comparison purpose we set the SC3DMC coders to code with 8 bits and differential coding options. We compared the orignal captured meshes to the meshes decoded using a tool to measure the symmetric distance between the surfaces. This metric is often known as haussdorf distance. A set of live captured models compressed and decoded shows that the qualities are comparible. Note that lower value implies better quality.
  • #24: For comparison purpose we set the SC3DMC coders to code with 8 bits and differential coding options. We compared the orignal captured meshes to the meshes decoded using a tool to measure the symmetric distance between the surfaces. This metric is often known as haussdorf distance. A set of live captured models compressed and decoded shows that the qualities are comparible. Note that lower value implies better quality.
  • #26: For comparison purpose we set the SC3DMC coders to code with 8 bits and differential coding options. We compared the orignal captured meshes to the meshes decoded using a tool to measure the symmetric distance between the surfaces. This metric is often known as haussdorf distance. A set of live captured models compressed and decoded shows that the qualities are comparible. Note that lower value implies better quality.
  • #27: For comparison purpose we set the SC3DMC coders to code with 8 bits and differential coding options. We compared the orignal captured meshes to the meshes decoded using a tool to measure the symmetric distance between the surfaces. This metric is often known as haussdorf distance. A set of live captured models compressed and decoded shows that the qualities are comparible. Note that lower value implies better quality.
  • #28: For comparison purpose we set the SC3DMC coders to code with 8 bits and differential coding options. We compared the orignal captured meshes to the meshes decoded using a tool to measure the symmetric distance between the surfaces. This metric is often known as haussdorf distance. A set of live captured models compressed and decoded shows that the qualities are comparible. Note that lower value implies better quality.
  • #29: For comparison purpose we set the SC3DMC coders to code with 8 bits and differential coding options. We compared the orignal captured meshes to the meshes decoded using a tool to measure the symmetric distance between the surfaces. This metric is often known as haussdorf distance. A set of live captured models compressed and decoded shows that the qualities are comparible. Note that lower value implies better quality.
  • #30: For comparison purpose we set the SC3DMC coders to code with 8 bits and differential coding options. We compared the orignal captured meshes to the meshes decoded using a tool to measure the symmetric distance between the surfaces. This metric is often known as haussdorf distance. A set of live captured models compressed and decoded shows that the qualities are comparible. Note that lower value implies better quality.
  • #31: For comparison purpose we set the SC3DMC coders to code with 8 bits and differential coding options. We compared the orignal captured meshes to the meshes decoded using a tool to measure the symmetric distance between the surfaces. This metric is often known as haussdorf distance. A set of live captured models compressed and decoded shows that the qualities are comparible. Note that lower value implies better quality.
  • #32: For comparison purpose we set the SC3DMC coders to code with 8 bits and differential coding options. We compared the orignal captured meshes to the meshes decoded using a tool to measure the symmetric distance between the surfaces. This metric is often known as haussdorf distance. A set of live captured models compressed and decoded shows that the qualities are comparible. Note that lower value implies better quality.
  • #33: For comparison purpose we set the SC3DMC coders to code with 8 bits and differential coding options. We compared the orignal captured meshes to the meshes decoded using a tool to measure the symmetric distance between the surfaces. This metric is often known as haussdorf distance. A set of live captured models compressed and decoded shows that the qualities are comparible. Note that lower value implies better quality.
  • #34: For comparison purpose we set the SC3DMC coders to code with 8 bits and differential coding options. We compared the orignal captured meshes to the meshes decoded using a tool to measure the symmetric distance between the surfaces. This metric is often known as haussdorf distance. A set of live captured models compressed and decoded shows that the qualities are comparible. Note that lower value implies better quality.
  • #35: For comparison purpose we set the SC3DMC coders to code with 8 bits and differential coding options. We compared the orignal captured meshes to the meshes decoded using a tool to measure the symmetric distance between the surfaces. This metric is often known as haussdorf distance. A set of live captured models compressed and decoded shows that the qualities are comparible. Note that lower value implies better quality.
  • #36: For comparison purpose we set the SC3DMC coders to code with 8 bits and differential coding options. We compared the orignal captured meshes to the meshes decoded using a tool to measure the symmetric distance between the surfaces. This metric is often known as haussdorf distance. A set of live captured models compressed and decoded shows that the qualities are comparible. Note that lower value implies better quality.
  • #37: For comparison purpose we set the SC3DMC coders to code with 8 bits and differential coding options. We compared the orignal captured meshes to the meshes decoded using a tool to measure the symmetric distance between the surfaces. This metric is often known as haussdorf distance. A set of live captured models compressed and decoded shows that the qualities are comparible. Note that lower value implies better quality.
  • #38: For comparison purpose we set the SC3DMC coders to code with 8 bits and differential coding options. We compared the orignal captured meshes to the meshes decoded using a tool to measure the symmetric distance between the surfaces. This metric is often known as haussdorf distance. A set of live captured models compressed and decoded shows that the qualities are comparible. Note that lower value implies better quality.
  • #43: For comparison purpose we set the SC3DMC coders to code with 8 bits and differential coding options. We compared the orignal captured meshes to the meshes decoded using a tool to measure the symmetric distance between the surfaces. This metric is often known as haussdorf distance. A set of live captured models compressed and decoded shows that the qualities are comparible. Note that lower value implies better quality.
  • #45: For comparison purpose we set the SC3DMC coders to code with 8 bits and differential coding options. We compared the orignal captured meshes to the meshes decoded using a tool to measure the symmetric distance between the surfaces. This metric is often known as haussdorf distance. A set of live captured models compressed and decoded shows that the qualities are comparible. Note that lower value implies better quality.