CharNeRF: 3D Character Generation from Concept Art using Neural Radiance Field
0 likes42 views
The document presents 'charnerf,' a method for generating 3D character models from 2D concept art, aiming to streamline the costly and time-consuming process of 3D modeling in virtual and augmented reality. The authors detail their approach, including data collection, methodological framework, results, and limitations, highlighting the benefits of capturing essential features from concept art. Future developments include improving rendering quality and enhancing interactivity for artists to supply additional sketches.
CharNeRF: 3D Character Generation from Concept Art using Neural Radiance Field
- 1. CharNeRF: 3D Character Generation from 2D Concept Art (IEEE AIxVR 2024) Eddy Chu1 , Yiyang Chen1 , Chedy Raissi2 , Anand Bhojan1 1 National University of Singapore1 , Riot Games2
- 2. ● Problem Overview ● Method ● Data Collection ● Results ● Ablation Test ● Limitations & Future Extensions Content 2
- 4. 3D modeling is essential in VR and AR as it enables creative and realistic expression of ideas, but it’s a costly process! 4
- 5. Current 3D Modeling in AR/VR/Game 5 Concept Art Image Source: https://www.behance.net/gallery/5594425/3D-Models-from-Concept-Art Image Source: https://zechatactics.com/blog/2021/03/how-to-build-a-zecha/ Image Source: https://substance3d.adobe.com/plugins/mixamo-in-blender/ 3D Models Shading & Texturing Rigging & Animation Optional Costly & Tedious
- 6. Types of Concept Art 6 Initial Concept Art Image Source: https://www.behance.net/gallery/5594425/3D-Models-from-Concept-Art Image Source: https://fgfactory.com/how-to-design-3d-character-from-scratch-to-final-game-model Image Source: https://www.weasyl.com/~magnus/submissions/55047/lewis-model-sheet Image Source: https://www.artstation.com/artwork/rRy3em Initial Sketches Turnaround Concept Art 3D Modeling-Ready
- 7. Input: 3-View Turnaround Concept Art G: Goal Model Output: 3D model (Volumetric Representation, NeRF) - Front, Back & Side faces - Meshes of different LoD can be generated easily 7 Problem Setting:
- 8. Industrial Values - Speed up the process of character modelling which usually takes hundreds of hours (source: WallaWalla Studio) - Speed up the process of skin production Research Values - There are a lot of existing works/researches on human or object reconstruction from 2D image(s), but to the best of our knowledge, reconstruction from concept art for virtual character is NEW! - Reconstruction for “virtual” character means only RGB values are available (not depth), which is challenging computer vision problem by itself - The shape variation for virtual characters is much greater than daily objects or human beings. A network that is able to capture the distribution of virtual characters would need to be extremely resourceful. 8
- 9. Method 9
- 10. Capture essential information from the concept art - Source Image Encoding 10
- 11. Pixel-aligned Features Image Encoder - Encode local information - Store at each pixel location Choose PixelNeRF (CVPR 2021) as our baseline 11
- 12. 12 Instead of ResNet, we find multi-level (one coarse and one fine) feature extraction greatly improves the rendering quality.
- 13. Image Encoder We find two-level encoder effective in extracting local information 13 Source images are first downsized to extract coarse information Fine feature Coarse featur
- 14. H/8 x W/8 x256 H/2 x W/2 x256 Image Encoder - Simplified Feature Combination 14 Fine feature Coarse feature MLP H/2 x W/2 x256 NeRF
- 15. Regularize NeRF through - a Mix of Sampling Methods 15
- 16. 2 x Neural Radiance Field (NeRF) 16
- 18. novel view Ray Sampling Wi Coarse Sampling 18
- 19. novel view Ray Sampling Wi ΣWj Wi Weight Distribution 19
- 20. novel view Ray Sampling Fine Sampling Wi ΣWj 20
- 21. NeRF is trained by minimizing the color difference between the image pixel and the “composited” color along a ray. It is therefore a common problem for a NeRF to “cheat” by distributing volume density more evenly along a ray than it should. Our idea is essentially to encourage NeRF to to assign high volume density near the surface area, which also allows NeRF to learn the shape directly. 21 Entropy Loss (InfoNeRF CVPR 2022) Emptiness Loss (SJC CVPR 2023)
- 22. novel view Ray Sampling Surface Sampling 22 We first distribute the target number of surface sampling points based on the sub-mesh area distribution, and then, for each sub-meshes, sampling is done using triangle point picking algorithm.
- 23. Pixel-aligned Features novel view src view src view Ray Sampling Surface Sampling 23
- 24. Mimic how a 3D modelist creates a 3D model - View Direction attended Feature Combination 24
- 25. Pixel-aligned Features novel view src view src view Ray Sampling Surface Sampling 25
- 26. Pixel-aligned Features novel view src view src view Early MLP Z’2 Z’1 Z’3 Ray Sampling Surface Sampling 26
- 27. Two More MLP Blocks - Feature Vector Similarity - View Direction Similarity View Direction Attended Multi-Head Self Attention 27
- 28. 28
- 29. Loss Function 29
- 30. 30
- 31. Simple Guideline for Mesh Reconstruction from CharNeRF 31
- 32. In the original NeRF, mesh is reconstructed through setting view direction to be zero vector. Nonetheless, the approach nullifies our view-direction attended feature combination mechanism. Therefore, to fully leverage the power of CharNeRF, we propose to supply multiple view directions and reconstruct the mesh through the average volume density. 32 Equivalent to setting view directions to0
- 33. Data 33
- 34. We collect 95 3D character from online resources - 75 in training dataset - 10 in validation dataset - 10 in test dataset 34
- 35. Each 3D Character, we generate - 303 images (3 views as concept art, 300 cameras evenly distributed using fibonacci lattice method) - 1080x1080 - White background - Pose.txt 4x4 camera to world matrix 1m 2m 4m 35 Image Source: https://www.mixamo.com/#/?page=2&type=Character
- 36. - Front, back, side images as concept art 36 Image Source: https://www.mixamo.com/#/?page=2&type=Character
- 37. - Intrinsics.json - focal length: fx, fy - measured in pixel - principle point coordinate: cx, cy - width & height - aabb_scale - Obj + Mtl + texture images 37
- 38. Results 38
- 39. Concept Art from 2D Artists: 39
- 40. 40 Concept Art from 2D Artists:
- 42. Ablation Test 42
- 43. CharNeRF 4 2 1 3 43 PixelNeRF { With a mix of sampling method
- 44. 44
- 45. Limitation & Future Extension 45
- 46. Extension 1 - Improve rendering quality 46
- 47. CharNeRF suffers from blurriness at the extremities of the characters, especially when the viewing angle is far from the concept art views… 47
- 48. 48 Later research in NeRF leverages diffusion models as prior by lifting its strong 2D generative power to 3D… Text to 3D through diffusion model - Score Jacobian Chaining (CVPR 2023) - Dreamfusion (ICLR 2023) Image to 3D through diffusion model - 3DiM (ICLR 2023) - RealFusion 360 (CVPR 2023) - zero-1-to-3 (ICCV 2023)
- 49. 49
- 50. 50
- 51. 51 NeRFDiff (ICML 2023) finetunes NeRF using conditional diffusion model Given the similar architecture of CharNeRF to PixelNeRF, the fine tuning method used in NeRFDiff is a ready-made extension to improve CharNeRF
- 52. Extension 2 - Interactivity & Editability 52
- 53. 53 The Final Character Concept Art typically comes with close-up sketches for the occluded parts. A model that allows 2D artists to interactively supply close-up sketches would be an important extension. Image Source: https://www.artstation.com/artwork/rRy3em
- 54. Thank You 54
- 55. Q & A 55