![High-quality 3D Content Creation
Meshy [2024]
LRM [Adobe,
2023]
Shap-E [OpenAI, 2023]
Automatic generation of 3D assets has attracted major industrial attention given its application in
● AR/VR ● Game industry ● Film industry](https://image.slidesharecdn.com/deepmarchingtetrahedra-241107071333-9c0c46ef/85/Deep-Marching-Tetrahedra-a-Hybrid-Representation-for-High-Resolution-3D-Shape-Synthesis-2-320.jpg)
![Flexible Isosurface Extraction for
Gradient-Based Mesh
Optimization
Tianchang Shen, Jacob Munkberg, Jon Hasselgren, Kangxue Yin, Zian Wang,
Wenzheng Chen, Zan Gojcic, Sanja Fidler, Nicholas Sharp, Jun Gao
Slides credit: Most of slides are adapted from the video “[SIGGRAPH'23] Flexible Isosurface Extraction for Gradient-Based Mesh
Optimization”](https://image.slidesharecdn.com/deepmarchingtetrahedra-241107071333-9c0c46ef/85/Deep-Marching-Tetrahedra-a-Hybrid-Representation-for-High-Resolution-3D-Shape-Synthesis-24-320.jpg)
![Deep Marching Tetrahedra (DMTet)
Marching
Tetrahedra
Marching
Tetrahedra [1]
DMTet
[1] Image credit: Jun Gao on Towards Generative Modeling of 3D Objects Learned from Images | Toronto AIR Seminar
No independent
repositioning of
vertices
Why DMTet is
deformable?](https://image.slidesharecdn.com/deepmarchingtetrahedra-241107071333-9c0c46ef/85/Deep-Marching-Tetrahedra-a-Hybrid-Representation-for-High-Resolution-3D-Shape-Synthesis-28-320.jpg)
Deep Marching Tetrahedra: a Hybrid Representation for High-Resolution 3D Shape Synthesis
- 1. Deep Marching Tetrahedra: a Hybrid Representation for High-Resolution 3D Shape Synthesis NeurIPS 2021 Tianchang Shen, Jun Gao, Kangxue Yin, Ming-Yu Liu, Sanja Fidler
- 2. High-quality 3D Content Creation Meshy [2024] LRM [Adobe, 2023] Shap-E [OpenAI, 2023] Automatic generation of 3D assets has attracted major industrial attention given its application in ● AR/VR ● Game industry ● Film industry
- 3. Representation for 3D Content Creation The core of a 3D generation system is the choice of 3D representation: ● Unlike 2D image generation, the “optimal” representation for 3D generation is still debatable.
- 4. Representation for 3D Content Creation The core of a 3D generation system is the choice of 3D representation: ● Unlike 2D image generation, the “optimal” representation for 3D generation is still debatable. DMTet proposes a new 3D representation and design their model based on it: ● enables them to achieve surprisingly good results at that time (2021).
- 5. Advantage of DMTet representation DMTet proposes a differentiable shape representation that marries implicit representations and meshes:
- 6. Advantage of DMTet representation DMTet proposes a differentiable shape representation that marries implicit representations and meshes: ● Takes the best of both worlds. Unlike most mesh representations, not restricted to fixed topologies. Unlike implicit representations, directly leverages surface losses.
- 7. DMTet: Overview ● Deformable tetrahedral grid ● Selective tetrahedra subdivision ● Differentiable marching tetrahedra layer and surface subdivision ● Learning a generative model with DMTet
- 8. Deformable tetrahedral grid ● The base 3D representation is the deformable tetrahedral grid (from DefTet): ○ Fully tetrahedralizes (or, tightly covers) the 3D unit cube. ○ Basic unit is tetrahedron. Each tetrahedron consists of 4 vertices and 4 faces. ○ Vertex records SDF values, interpolate for any other 3D points in the tetrahedron. Learning Deformable Tetrahedral Meshes for 3D Reconstruction, Gao et al., NeurIPS 2020 tetrahed ron 2D illustration
- 9. Tetrahedra subdivision ● The representation is multi-scale: ○ When SDF sign changes between vertices, can subdivide the tetrahedra and its neighbors; ○ Subdivision: adding the midpoints as vertices and use average SDF value. Each surface tet.(blue) is divided into 8 tet.(red) by adding midpoints. 2D illustration
- 10. Tetrahedra to surface ● Marching Tetrahedra algorithm: ○ Converts SDF grids into explicit triangular mesh. ○ Only 3 unique surface-generation configurations with rotation symmetry. ○ Mesh vertex calculated as linear interpolation of tetrahedra vertices using SDF for weighting. ○ Gradient directly backpropagated to SDF values and vertex positions.
- 11. Tetrahedra to surface ● Marching Tetrahedra algorithm: ○ Converts SDF grids into explicit triangular mesh. ○ Only 3 unique surface-generation configurations with rotation symmetry. ○ Mesh vertex calculated as linear interpolation of tetrahedra vertices using SDF for weighting. ○ Gradient directly backpropagated to SDF values and vertex positions. ● The mesh can be further subdivided following simplified Neural Subdivision Neural subdivision, Liu et al., TOG 2020
- 12. Generative modeling based on tedrahedron ● Problem setting: Point-conditioned or voxel-conditioned shape generation. ● Input Encoder is PVCNN: extract feature volume from point clouds. ● SDF prediction via MLP: ● MLP also output features for each vertices for surface refinement
- 13. Learning surface refinement with GCN ● Build a graph using all surface tetrahedra; ● Learn a GCN to dynamically deform the vertices and update the SDF; ● Subdivide surface tetrahedra and prune unused tetrahedra; ● Further refinement and subdivision of the extracted mesh.
- 14. Mesh discriminator ● Learn an additional discriminator for better local details: ○ 3D CNN Discriminator from DECOR-GAN. ○ Input is local NxNxN SDF grid, which can be differentiably calculated from mesh. Decor-gan: 3d shape detailization by conditional refinement, Chen et al., CVPR 2021
- 15. Learning objectives ● Surface alignment: ○ Chamfer Distance (with normal) between prediction and groundtruth
- 16. Learning objectives ● Surface alignment: ○ Chamfer Distance (with normal) between prediction and groundtruth ● Adversarial loss follows LSGAN Least squares generative adversarial networks, Mao et al., ICCV 2017
- 17. Learning objectives ● Surface alignment: ○ Chamfer Distance (with normal) between prediction and groundtruth ● Adversarial loss follows LSGAN ● Direct SDF loss: ○ Make sure whole space receives supervision signal and avoid flipping of SDF signs Least squares generative adversarial networks, Mao et al., ICCV 2017
- 18. Learning objectives ● Surface alignment: ○ Chamfer Distance (with normal) between prediction and groundtruth ● Adversarial loss follows LSGAN ● Direct SDF loss: ○ Make sure whole space receives supervision signal and avoid flipping of SDF signs ● Deformation regularization: ○ Make sure the deformation stays small. Final loss is a weighted average of all these. Least squares generative adversarial networks, Mao et al., ICCV 2017
- 19. Learning objectives ● Surface alignment: ○ Chamfer Distance (with normal) between prediction and groundtruth ● Adversarial loss follows LSGAN ● Direct SDF loss: ○ Make sure whole space receives supervision signal and avoid flipping of SDF signs ● Deformation regularization: ○ Make sure the deformation stays small. ● Final loss is a weighted average of all these. Least squares generative adversarial networks, Mao et al., ICCV 2017
- 20. Results – Point cloud conditioned generation on ShapeNet
- 21. Results – Voxel conditioned generation on animals
- 22. Why is DMTet better than previous approaches?
- 23. Why is DMTet better than previous approaches? ● No topology constraints; ● Direct surface supervision and regularization; ● Less artifacts given the deformable grids; ● Better efficiency and faithfulness of MT over MC under low sampling budget.
- 24. Flexible Isosurface Extraction for Gradient-Based Mesh Optimization Tianchang Shen, Jacob Munkberg, Jon Hasselgren, Kangxue Yin, Zian Wang, Wenzheng Chen, Zan Gojcic, Sanja Fidler, Nicholas Sharp, Jun Gao Slides credit: Most of slides are adapted from the video “[SIGGRAPH'23] Flexible Isosurface Extraction for Gradient-Based Mesh Optimization”
- 25. Generating Mesh via Gradient-Based Optimization Allow local flexible adjustments
- 26. Existing Isosurface Extraction Methods
- 27. Marching Cube Marching Cubes Increase grid resolution Signed Distance Field Mesh vertices are constrained on lattice edges Small triangles in low curvature areas Requires more SDF samples
- 28. Deep Marching Tetrahedra (DMTet) Marching Tetrahedra Marching Tetrahedra [1] DMTet [1] Image credit: Jun Gao on Towards Generative Modeling of 3D Objects Learned from Images | Toronto AIR Seminar No independent repositioning of vertices Why DMTet is deformable?
- 29. Dual Contouring Dual Contouring Produce non-manifold results Vertex outside the cube
- 30. Previous Methods Fail to Satisfy Both Properties
- 31. FlexiCubes Improve fitting with additional flexibility + Well-defined gradient differentiation
- 32. FlexiCubes Improve fitting with additional flexibility + Well-defined gradient differentiation Recall prior dual methods diverge during optimization.
- 33. Dual Marching Cubes (Centroid) No QEF for vertex positioning Lack of freedom Quadrilateral faces
- 34. FlexiCubes FlexiCubes introduces 3 types of parameters into Dual Marching Cubes: - Interpolation weights to position dual vertices in space. - Splitting weights to control how to split quadrilaterals into triangles. - Deformation vectors for spatial alignment. (DMTet)
- 35. Interpolation weights 𝛼 per-cell adjusting interpolation along each edge 𝛽 per-cell adjusting vertex position within each dual face 𝛼, 𝛽 ℝ+ ∈
- 36. Splitting weights Raw output from Dual Marching Cubes are typically non-planar quadrilateral faces - Quad-split weights 𝛾 controlling how quads get split to tris. During optimization, 𝛾 interpolates the surface between two possible splits. At inference, we split along the diagonal with
- 38. Extensions Tetrahedral mesh extraction Hierarchically adaptive meshing
- 39. Applications
- 40. Photogrammetry Through Differentiable Rendering Nvdiffrec jointly optimizes shape, materials, and lighting from images. FlexiCubes improves geometric fidelity and mesh quality.
- 41. 3D Mesh Generation FlexiCubes can serve as a plug-and-play differentiable mesh extraction module in a 3D generative model GET3D w/ FlexiCubes generates meshes with better details and tessellation.
- 42. Differentiable Physics Simulation with Tetrahedral Mesh Combine Flexicubes with a differentiable physics simulation framework (gradSim) and a differentiable rendering pipeline (nvdiffrast) Jointly recover 3D shapes and physical parameters from multi-view videos
- 43. Mesh Simplification of Animated Objects FlexiCubes allows us to differentiably skin and deform the mesh, and simultaneously optimize with respect to the entire animated sequence. End-to-end optimization w/ FlexiCubes avoids artifacts from mesh stretching.
- 44. Adding Mesh Regularizations FlexiCubes representation is flexible enough to support optimizing regularizations defined on meshes