Light Field Networks: Neural Scene Representations with Single-Evaluation Rendering, NeurIPS 2021
Download as PPTX, PDF0 likes84 views
The document presents Light Field Networks (LFNs), a new approach for neural scene representations that enables real-time rendering at 500 fps with a single evaluation per ray, significantly improving memory efficiency and training speed. It addresses challenges in traditional light field representation methods by using Plücker coordinates for parameterization, allowing 360-degree coverage and practical applications without discrete structures. Additionally, the authors explore multi-view consistency and geometry extraction capabilities of LFNs, showcasing robust results in various applications such as 3D scene reconstruction.
![Vincent Sitzmann & Semon Rezchikov, NeurIPS 2021
DVR [Niemeyer et al. 2020], IDR [Yariv et al. 2020], …
3D-structured Neural Scene Representations
Neural Radiance Fields
Mildenhall et al. 2020
Scene Representation Networks
Sitzmann et al. 2019
pixelNeRF
Yu et al. 2020](https://image.slidesharecdn.com/cameraready-220629180140-1571eb2d/85/Light-Field-Networks-Neural-Scene-Representations-with-Single-Evaluation-Rendering-NeurIPS-2021-4-320.jpg)
![Vincent Sitzmann & Semon Rezchikov, NeurIPS 2021
: ℝ3 → ℝn
Light Field
[Adelson et al. 1991, Levoy et al. 1996, Gortler et al. 1996]](https://image.slidesharecdn.com/cameraready-220629180140-1571eb2d/85/Light-Field-Networks-Neural-Scene-Representations-with-Single-Evaluation-Rendering-NeurIPS-2021-13-320.jpg)
![Vincent Sitzmann & Semon Rezchikov, NeurIPS 2021
Decode embedding into scene representation
embedding
𝑧0
1[Schmidhuber et al. 1992, Schmidhuber et al. 1993, Stanley et al. 2009, Ha et al., 2016]
LFN
Hypernetwork1
Ψ𝝍 Φ𝜙=Ψ𝜓(𝑧0)
Rendering](https://image.slidesharecdn.com/cameraready-220629180140-1571eb2d/85/Light-Field-Networks-Neural-Scene-Representations-with-Single-Evaluation-Rendering-NeurIPS-2021-50-320.jpg)
![Vincent Sitzmann & Semon Rezchikov, NeurIPS 2021
Global vs. Local Conditioning – Single-Shot Rec.
pixelNeRF [Yu et al. 2020] comparison, randomly selected
Input PN Ours GT Input PN Ours GT Inpu
t
P
N
Our
s
G
T
Inpu
t
P
N
Our
s
G
T](https://image.slidesharecdn.com/cameraready-220629180140-1571eb2d/85/Light-Field-Networks-Neural-Scene-Representations-with-Single-Evaluation-Rendering-NeurIPS-2021-67-320.jpg)
Light Field Networks: Neural Scene Representations with Single-Evaluation Rendering, NeurIPS 2021
- 1. Vincent Sitzmann & Semon Rezchikov, NeurIPS 2021 Light Field Networks: Neural Scene Representations with Single-Evaluation Rendering Vincent Sitzmann* Frédo Durand Semon Rezchikov* Joshua B. Tenenbaum William T. Freeman
- 2. Vincent Sitzmann & Semon Rezchikov, NeurIPS 2021 Graphics Physics Simulation Autonomous Navigation & Planning Photogrammetry Robotic Vision Robotic Grasping 2
- 3. Vincent Sitzmann & Semon Rezchikov, NeurIPS 2021 Self-Supervised Scene Representation Learning Graphics Physics Simulation Autonomous Navigation & Planning Photogrammetry Robotic Vision Robotic Grasping Learned feature representation of 3D scenes. Neural 3
- 4. Vincent Sitzmann & Semon Rezchikov, NeurIPS 2021 DVR [Niemeyer et al. 2020], IDR [Yariv et al. 2020], … 3D-structured Neural Scene Representations Neural Radiance Fields Mildenhall et al. 2020 Scene Representation Networks Sitzmann et al. 2019 pixelNeRF Yu et al. 2020
- 5. Vincent Sitzmann & Semon Rezchikov, NeurIPS 2021 3D-structured Neural Scene Representations : ℝ3 → ℝn
- 6. Vincent Sitzmann & Semon Rezchikov, NeurIPS 2021 3D-structured Neural Scene Representations : ℝ3 → ℝn
- 7. Vincent Sitzmann & Semon Rezchikov, NeurIPS 2021 3D-structured Neural Scene Representations : ℝ3 → ℝn
- 8. Vincent Sitzmann & Semon Rezchikov, NeurIPS 2021 3D-structured Neural Scene Representations : ℝ3 → ℝn
- 9. Vincent Sitzmann & Semon Rezchikov, NeurIPS 2021 3D-structured Neural Scene Representations : ℝ3 → ℝn
- 10. Vincent Sitzmann & Semon Rezchikov, NeurIPS 2021 3D-structured Neural Scene Representations : ℝ3 → ℝn
- 11. Vincent Sitzmann & Semon Rezchikov, NeurIPS 2021 3D-structured Neural Scene Representations : ℝ3 → ℝn
- 12. Vincent Sitzmann & Semon Rezchikov, NeurIPS 2021 3D-structured Neural Scene Representations : ℝ3 → ℝn Hundreds of samples per ray. 256x256 image takes >30 seconds (volumetric). Time- and memory-intensive training.
- 13. Vincent Sitzmann & Semon Rezchikov, NeurIPS 2021 : ℝ3 → ℝn Light Field [Adelson et al. 1991, Levoy et al. 1996, Gortler et al. 1996]
- 14. Vincent Sitzmann & Semon Rezchikov, NeurIPS 2021 : ℝ3 → ℝn Light FieldNetworks
- 15. Vincent Sitzmann & Semon Rezchikov, NeurIPS 2021 : ℝ3 → ℝn Light FieldNetworks
- 16. Vincent Sitzmann & Semon Rezchikov, NeurIPS 2021 Light FieldNetworks Conditioning Plücker Coords. An Alternative Scene Representation
- 17. Vincent Sitzmann & Semon Rezchikov, NeurIPS 2021 Light Field Networks Volumetric Rendering (pixelNeRF) 500 FPS 1 evaluation per ray 0.033 FPS 196 evaluations per ray Real-time. No post-processing, no discrete data structures (octrees, voxelgrids, …). >100x reduction in memory: Can be trained on small GPUs! 15,000x speed 1,000x speed 100x speed 10x speed 1x speed
- 18. Vincent Sitzmann & Semon Rezchikov, NeurIPS 2021 Light Field Networks 500 FPS 1 evaluation per ray
- 19. Vincent Sitzmann & Semon Rezchikov, NeurIPS 2021 Also Encode Depth in their 4D derivatives: can be extracted via single evaluation of neural network and its gradient!
- 20. Vincent Sitzmann & Semon Rezchikov, NeurIPS 2021 Results Limitations LFN Geometry Parameterization Meta-Learning Ψ𝝍 ΦΨ
- 21. Vincent Sitzmann & Semon Rezchikov, NeurIPS 2021 Results Limitations LFN Geometry Parameterization Meta-Learning Ψ𝝍 ΦΨ Ray Parameterizations for LFNs
- 22. Vincent Sitzmann & Semon Rezchikov, NeurIPS 2021 Conventional Light Field Parameterizations Two-Plane Lumigraph Two-Sphere Cylindrical Not 360° Not 360° Not Continuous Bounded Scenes Difficult to use as a complete scene representation
- 23. Vincent Sitzmann & Semon Rezchikov, NeurIPS 2021 “Point-direction” coordinates
- 24. Vincent Sitzmann & Semon Rezchikov, NeurIPS 2021 “Point-direction” coordinates Not unique: Same ray, two different coordinates.
- 25. Vincent Sitzmann & Semon Rezchikov, NeurIPS 2021 Unique: invariant to choice of x. Parameterize all rays without special cases. Impractical for discrete representations, since ∈ ℝ⁶. Plücker coordinates
- 26. Vincent Sitzmann & Semon Rezchikov, NeurIPS 2021 Plücker coordinates
- 27. Vincent Sitzmann & Semon Rezchikov, NeurIPS 2021 Plücker coordinates Parameterize 360 degree light fields of unbounded scenes.
- 28. Vincent Sitzmann & Semon Rezchikov, NeurIPS 2021 Results Limitations LFN Geometry Parameterization Meta-Learning Ψ𝝍 ΦΨ Extracting Scene Geometry from LFNs
- 29. Vincent Sitzmann & Semon Rezchikov, NeurIPS 2021 The geometry of LFNs
- 30. Vincent Sitzmann & Semon Rezchikov, NeurIPS 2021 The geometry of LFNs p
- 31. Vincent Sitzmann & Semon Rezchikov, NeurIPS 2021 The geometry of LFNs p p
- 32. Vincent Sitzmann & Semon Rezchikov, NeurIPS 2021 The geometry of LFNs p p
- 33. Vincent Sitzmann & Semon Rezchikov, NeurIPS 2021 The geometry of LFNs p p
- 34. Vincent Sitzmann & Semon Rezchikov, NeurIPS 2021 The geometry of LFNs p p
- 35. Vincent Sitzmann & Semon Rezchikov, NeurIPS 2021 c(s,t) The geometry of LFNs p p Epipolar Plane Image
- 36. Vincent Sitzmann & Semon Rezchikov, NeurIPS 2021 The geometry of LFNs p p c(s,t) τ Epipolar Plane Image
- 37. Vincent Sitzmann & Semon Rezchikov, NeurIPS 2021 The geometry of LFNs p p c(s,t) τ τ p Epipolar Plane Image
- 38. Vincent Sitzmann & Semon Rezchikov, NeurIPS 2021 The geometry of LFNs p p c(s,t) τ Points give lines of constant color in EPI c(s,t) – line is a levelset of the EPI. τ p Epipolar Plane Image
- 39. Vincent Sitzmann & Semon Rezchikov, NeurIPS 2021 The geometry of LFNs p c(s,t) p p Points give lines of constant color in EPI c(s,t) – line is a levelset of the EPI. Slope of line decreases as point moves closer. Epipolar Plane Image
- 40. Vincent Sitzmann & Semon Rezchikov, NeurIPS 2021 The geometry of LFNs p c(s,t) p p Points give lines of constant color in EPI c(s,t) – line is a levelset of the EPI. Slope of line decreases as point moves closer. Epipolar Plane Image
- 41. Vincent Sitzmann & Semon Rezchikov, NeurIPS 2021 The geometry of LFNs p c(s,t) p p Points give lines of constant color in EPI c(s,t) – line is a levelset of the EPI. Slope of line decreases as point moves closer. Epipolar Plane Image
- 42. Vincent Sitzmann & Semon Rezchikov, NeurIPS 2021 The geometry of LFNs p c(s,t) p Points give lines of constant color in EPI c(s,t) – line is a levelset of the EPI. Slope of line decreases as point moves closer. Gradient of c(s,t) is orthogonal to levelset - can extract depth from gradients of light field. ∇c(s,t) p Epipolar Plane Image
- 43. Vincent Sitzmann & Semon Rezchikov, NeurIPS 2021 The geometry of LFNs p c(s,t) p p Points give lines of constant color in EPI c(s,t) – line is a levelset of the EPI. Slope of line decreases as point moves closer. Gradient of c(s,t) is orthogonal to levelset - can extract depth from gradients of light field. ∇c(s,t) Epipolar Plane Image
- 44. Vincent Sitzmann & Semon Rezchikov, NeurIPS 2021 The geometry of LFNs p c(s,t) p p Points give lines of constant color in EPI c(s,t) – line is a levelset of the EPI. Slope of line decreases as point moves closer. ∇c(s,t) Gradient of c(s,t) is orthogonal to levelset - can extract depth from gradients of light field. Epipolar Plane Image
- 45. Vincent Sitzmann & Semon Rezchikov, NeurIPS 2021 Multi-view consistency c(s,t)
- 46. Vincent Sitzmann & Semon Rezchikov, NeurIPS 2021 Multi-view consistency c(s,t)
- 47. Vincent Sitzmann & Semon Rezchikov, NeurIPS 2021 Multi-view consistency c(s,t) τ
- 48. Vincent Sitzmann & Semon Rezchikov, NeurIPS 2021 Results Limitations LFN Geometry Parameterization Meta-Learning Ψ𝝍 ΦΨ Meta-Learning Multi-View Consistency
- 49. Vincent Sitzmann & Semon Rezchikov, NeurIPS 2021 Learning a space of multi-view consistent light fields + } ,… embedding 𝑧0 { embedding 𝑧1 embedding 𝑧𝑛 𝑧𝑗=0,…,𝑛~𝒩(0, 𝜎2 ) } ,… { + + } ,… {
- 50. Vincent Sitzmann & Semon Rezchikov, NeurIPS 2021 Decode embedding into scene representation embedding 𝑧0 1[Schmidhuber et al. 1992, Schmidhuber et al. 1993, Stanley et al. 2009, Ha et al., 2016] LFN Hypernetwork1 Ψ𝝍 Φ𝜙=Ψ𝜓(𝑧0) Rendering
- 51. Vincent Sitzmann & Semon Rezchikov, NeurIPS 2021 Decode embedding into scene representation Φ𝜙=Ψ𝜓(𝑧0) embedding 𝑧0 LFN Ψ𝝍 arg min 𝑧𝑗 𝑗=1 𝑀 ,𝝍 𝑗 𝑖 REN D ER (Φ𝜙=Ψ𝜓(𝑧𝑗), 𝜉𝑖) − ℐ𝑖 𝑗 Rendering Hypernetwork
- 52. Vincent Sitzmann & Semon Rezchikov, NeurIPS 2021 Test time: Initialize new embedding Φ𝜙=Ψ𝜓(𝑧new) LFN Ψ𝝍 Rendering embedding 𝑧new Hypernetwork
- 53. Vincent Sitzmann & Semon Rezchikov, NeurIPS 2021 Freeze weights & optimize latent code only. Φ𝜙=Ψ𝜓(𝑧new) LFN Ψ𝝍 Rendering 𝑧 = arg min 𝑧 REN D ER (Φ𝜙=Ψ𝜓(𝑧0), 𝜉) − ℐ embedding 𝑧new Hypernetwork
- 54. Vincent Sitzmann & Semon Rezchikov, NeurIPS 2021 Results Limitations LFN Geometry Parameterization Meta-Learning Ψ𝝍 ΦΨ Results
- 55. Vincent Sitzmann & Semon Rezchikov, NeurIPS 2021 LFNs learn multi-view consistent 360-degree light fields 500 FPS, single evaluation per ray. Shapenet Cars Shapenet Chairs GQN Rooms
- 56. Vincent Sitzmann & Semon Rezchikov, NeurIPS 2021 Pointclouds extracted from 4 views of cars & chairs Single evaluation of network & its gradient per ray, constant complexity
- 57. Vincent Sitzmann & Semon Rezchikov, NeurIPS 2021 Multi-Class Single-Shot Reconstruction DVR SRN Ours GT DVR SRN Ours GT Input Input
- 58. Vincent Sitzmann & Semon Rezchikov, NeurIPS 2021 Multi-Class Single-Shot Ours GT
- 59. Vincent Sitzmann & Semon Rezchikov, NeurIPS 2021 View-dependent effects. Ours GT
- 60. Vincent Sitzmann & Semon Rezchikov, NeurIPS 2021 Multi-Class Single-Shot Screens and Phones DVR SRN Ours GT Input
- 61. Vincent Sitzmann & Semon Rezchikov, NeurIPS 2021 Multi-Class Single-Shot: Random samples DVR SRN Ours G T Input DVR SRN Ours G T Input DVR SRN Ours G T Input DVR SRN Ours G T Input
- 62. Vincent Sitzmann & Semon Rezchikov, NeurIPS 2021 Multi-Class Single-Shot: Random samples DVR SRN Ours G T Input DVR SRN Ours G T Input DVR SRN Ours G T Input DVR SRN Ours G T Input
- 63. Vincent Sitzmann & Semon Rezchikov, NeurIPS 2021 Multi-Class Single-Shot: Random samples DVR SRN Ours G T Input DVR SRN Ours G T Input DVR SRN Ours G T Input DVR SRN Ours G T Input
- 64. Vincent Sitzmann & Semon Rezchikov, NeurIPS 2021 Multi-Class Single-Shot: Random samples DVR SRN Ours G T Input DVR SRN Ours G T Input DVR SRN Ours G T Input DVR SRN Ours G T Input
- 65. Vincent Sitzmann & Semon Rezchikov, NeurIPS 2021 Single-Class Single-Shot (SRNs, cars) Randomly selected. SRN Ours G T Input SRN Ours G T Input SRN Ours G T Input
- 66. Vincent Sitzmann & Semon Rezchikov, NeurIPS 2021 Single-Class Single-Shot (SRNs, chairs) Randomly selected. SRN Ours G T Input SRN Ours G T Input SRN Ours G T Input
- 67. Vincent Sitzmann & Semon Rezchikov, NeurIPS 2021 Global vs. Local Conditioning – Single-Shot Rec. pixelNeRF [Yu et al. 2020] comparison, randomly selected Input PN Ours GT Input PN Ours GT Inpu t P N Our s G T Inpu t P N Our s G T
- 68. Vincent Sitzmann & Semon Rezchikov, NeurIPS 2021 Results Limitations LFN Geometry Parameterization Meta-Learning Ψ𝝍 ΦΨ Limitations
- 69. Vincent Sitzmann & Semon Rezchikov, NeurIPS 2021 Limitations Multi-view Consistency Local conditioning One color per ray
- 70. Vincent Sitzmann & Semon Rezchikov, NeurIPS 2021 Limitations Multi-view Consistency Local conditioning One color per ray
- 71. Vincent Sitzmann & Semon Rezchikov, NeurIPS 2021 Limitations Multi-view Consistency Local conditioning One color per ray
- 72. Vincent Sitzmann & Semon Rezchikov, NeurIPS 2021 Limitations Multi-view Consistency Local conditioning One color per ray
- 73. Vincent Sitzmann & Semon Rezchikov, NeurIPS 2021 Limitations Multi-view Consistency Local conditioning One color per ray
- 74. Vincent Sitzmann & Semon Rezchikov, NeurIPS 2021 Limitations Multi-view Consistency Local conditioning One color per ray Context Views Overfitting single scene (with positional encoding) Intermediate Views
- 75. Vincent Sitzmann & Semon Rezchikov, NeurIPS 2021 Limitations Multi-view Consistency Local conditioning One color per ray pixelNeRF Yu et al. 2020
- 76. Vincent Sitzmann & Semon Rezchikov, NeurIPS 2021 Light Field Networks: Neural Scene Representations with Single-Evaluation Rendering Vincent Sitzmann* Semon Rezchikov* Joshua B. Tenenbaum William T. Freeman Frédo Durand vsitzmann.github.io/lfns