![PCA Expression Model
Raw Difference Model
X =
D1 D2 D3 · · · Dm
Di =
µD = Mean of training data
Bi = Principal component of training data
FE = Expression face mesh to be fit (e.g. mouthOpen)
FN = Neutral face mesh
FE = FN + s[µD + ω1B1 + ω2B2 + · · · + ωpBp]
Ian Dewancker Internship Summary](https://image.slidesharecdn.com/aabb779d-087d-4fff-8406-f202a2fa9b5f-151231033023/85/EPFL_presentation-29-320.jpg)
![PCA Expression Model
Deformation Transfer Model
X =
L1 L2 L3 · · · Lm
Li =
µL = Mean of deform transferred training data
DTBi = Deformed transferred PCs of training data
DTF(FN →µL) = Neutral face deform transferred to µL
FE = Expression face mesh to be fit (e.g. mouthOpen)
FN = Neutral face mesh (provided)
FE = FN+s[(DTF(FN →µL)−FN)+ω1DTB1+ω2DTB2+· · ·+ωpDTBp]
Ian Dewancker Internship Summary](https://image.slidesharecdn.com/aabb779d-087d-4fff-8406-f202a2fa9b5f-151231033023/85/EPFL_presentation-31-320.jpg)
EPFL_presentation
- 1. Internship Summary Ian Dewancker August 25, 2011 Ian Dewancker Internship Summary
- 2. Problem Ian Dewancker Internship Summary
- 3. Problem Realtime Performance-Based Facial Animation Ian Dewancker Internship Summary
- 4. Problem “We demonstrate that compelling 3D facial dynamics can be reconstructed in realtime without the use of face markers, intrusive lighting, or complex scanning hardware.” Ian Dewancker Internship Summary
- 5. Problem “We demonstrate that compelling 3D facial dynamics can be reconstructed in realtime without the use of face markers, intrusive lighting, or complex scanning hardware.” Ian Dewancker Internship Summary
- 6. Problem Offline Preproccesing Ian Dewancker Internship Summary
- 7. Problem Offline Preproccesing “Manually selected feature correspondences guide the reconstruction of user-specific expressions” Ian Dewancker Internship Summary
- 8. Problem Offline Preproccesing “Manually selected feature correspondences guide the reconstruction of user-specific expressions” Ian Dewancker Internship Summary
- 9. Problem Goal : Simplify generation of user-specific expressions (while maintaining quality) Ian Dewancker Internship Summary
- 10. Problem Goal : Simplify generation of user-specific expressions (while maintaining quality) Ian Dewancker Internship Summary
- 11. Approach Looking for model(s) that define how face mesh vertices should deform to fit scan data. Ian Dewancker Internship Summary
- 12. Approach Looking for model(s) that define how face mesh vertices should deform to fit scan data. Several existing techniques : Thin-Shell Deformation Deformation Graph Principal Component Analysis (PCA) As-Rigid-as-possible Radial Basis Functions PriMo Ian Dewancker Internship Summary
- 13. Approach Looking for model(s) that define how face mesh vertices should deform to fit scan data. Several existing techniques : Thin-Shell Deformation Deformation Graph Principal Component Analysis (PCA) As-Rigid-as-possible Radial Basis Functions PriMo Here we focus on investigating the use of PCA models for generating user specific expressions Ian Dewancker Internship Summary
- 14. PCA Overview Idea : From a set of mean adjusted observations, produce a set of orthogonal basis vectors, ordered by decreasing variance. Ian Dewancker Internship Summary
- 15. PCA Overview Idea : From a set of mean adjusted observations, produce a set of orthogonal basis vectors, ordered by decreasing variance. How : Take the singular value decomposition (SVD) of the mean adjusted data matrix. Select the first n left singular vectors as an approximate orthogonal basis. These are the principal components and represent vectors along the axes of maximum variance Ian Dewancker Internship Summary
- 16. PCA Overview In 2D: X = x1 x2 · · · xm y1 y2 · · · ym Ian Dewancker Internship Summary
- 17. PCA Overview In 2D: X = x1 x2 · · · xm y1 y2 · · · ym µ = xµ yµ Ian Dewancker Internship Summary
- 18. PCA Overview In 2D: X = x1 x2 · · · xm y1 y2 · · · ym µ = xµ yµ G = X−µ 1 1 · · · 1 G = UΣV ∗ Ian Dewancker Internship Summary
- 19. PCA Overview In 2D: X = x1 x2 · · · xm y1 y2 · · · ym µ = xµ yµ G = X−µ 1 1 · · · 1 G = UΣV ∗ U = b1x b2x b1y b2y Σ = 3 0 0 1 Ian Dewancker Internship Summary
- 20. PCA Identity Model How does this help us with fitting faces? Ian Dewancker Internship Summary
- 21. PCA Identity Model How does this help us with fitting faces? Ni = xi1 yi1 zi1 . . . xik yik zik ∈ R 3k Suppose we treat a collection of m fitted neutral face meshes as data observations and generate the following: Ian Dewancker Internship Summary
- 22. PCA Identity Model How does this help us with fitting faces? Ni = xi1 yi1 zi1 . . . xik yik zik ∈ R 3k Suppose we treat a collection of m fitted neutral face meshes as data observations and generate the following: X = N1 N2 N3 · · · Nm µN = 1 m m i=1 Ni Ian Dewancker Internship Summary
- 23. PCA Identity Model As before in the example, we compute the matrix G and its SVD: G = X − µN 1 1 1 · · · 1 G = UΣV ∗ Ian Dewancker Internship Summary
- 24. PCA Identity Model As before in the example, we compute the matrix G and its SVD: G = X − µN 1 1 1 · · · 1 G = UΣV ∗ In this case, U now holds the principal components of the difference vectors from the average neutral face to those in the data set. U = B1 B2 B3 · · · B3k Ian Dewancker Internship Summary
- 25. PCA Identity Model As before in the example, we compute the matrix G and its SVD: G = X − µN 1 1 1 · · · 1 G = UΣV ∗ In this case, U now holds the principal components of the difference vectors from the average neutral face to those in the data set. U = B1 B2 B3 · · · B3k We expect that any neutral face (provided training set is representative) can be reasonably approximated by adding a linear combination of the PCs to the mean neutral face: Fneutral = µN + ω1B1 + ω2B2 + ω3B3 + · · · + ωpBp p 3k Ian Dewancker Internship Summary
- 26. PCA Expression Model Ian Dewancker Internship Summary
- 27. PCA Expression Model Consider deformations from neutral face to expressions like “mouth open”. Ian Dewancker Internship Summary
- 28. PCA Expression Model Raw Difference Model X = D1 D2 D3 · · · Dm Di = Ian Dewancker Internship Summary
- 29. PCA Expression Model Raw Difference Model X = D1 D2 D3 · · · Dm Di = µD = Mean of training data Bi = Principal component of training data FE = Expression face mesh to be fit (e.g. mouthOpen) FN = Neutral face mesh FE = FN + s[µD + ω1B1 + ω2B2 + · · · + ωpBp] Ian Dewancker Internship Summary
- 30. PCA Expression Model Deformation Transfer Model X = L1 L2 L3 · · · Lm Li = Ian Dewancker Internship Summary
- 31. PCA Expression Model Deformation Transfer Model X = L1 L2 L3 · · · Lm Li = µL = Mean of deform transferred training data DTBi = Deformed transferred PCs of training data DTF(FN →µL) = Neutral face deform transferred to µL FE = Expression face mesh to be fit (e.g. mouthOpen) FN = Neutral face mesh (provided) FE = FN+s[(DTF(FN →µL)−FN)+ω1DTB1+ω2DTB2+· · ·+ωpDTBp] Ian Dewancker Internship Summary
- 32. Training Data Training data is absolutely necessary for the PCA models. A larger number of training examples will likely be more representative of the expression space and thus, our model will be better able to approximate arbitrary faces In our case, we require the following to generate training data: 3D Scans Fitted Meshes Ian Dewancker Internship Summary
- 33. 3D Scans We used the structured light scanner to record everyone in the LGG + Keenan (12 people) in 4 different expressions ( Neutral, MouthOpen, Smile, and Kiss ) for a total of 48 scans. The scanner also provides an image that can be used for 2D texture constraints when fitting the mesh Ian Dewancker Internship Summary
- 34. Face Fitting We needed a way to efficiently fit the template or neutral face mesh to the point clouds produced by the scanner. Ian Dewancker Internship Summary
- 35. Face Fitting We needed a way to efficiently fit the template or neutral face mesh to the point clouds produced by the scanner. FaceFitting Application Ian Dewancker Internship Summary
- 36. Face Fitting Addition of UI to more easily control deformation parameters and perform other operations. Fitting HD version of loki quad mesh (27k vertices, 54k faces) Ian Dewancker Internship Summary
- 37. Face Fitting One of the most tedious parts of fitting is setting the 2D image constraints (what we are trying to shield the user from). App can save and load markers, camera matrices and images easily. Added ability to create 2D constraints using rendered image instead of one from real camera Ian Dewancker Internship Summary
- 38. Face Fitting Included a heat map visualization to show distances between correspondences Ian Dewancker Internship Summary
- 39. Building PCA Models With our collection of fitted meshes we can now generate PCA expression models Ian Dewancker Internship Summary
- 40. Building PCA Models With our collection of fitted meshes we can now generate PCA expression models Somewhat tedious to type various commands to generate PCA models, especially for deformation transfer PCA. Also, if we want to evaluate our PCA model, the person we are fitting should not be included in the model. Ian Dewancker Internship Summary
- 41. Building PCA Models With our collection of fitted meshes we can now generate PCA expression models Somewhat tedious to type various commands to generate PCA models, especially for deformation transfer PCA. Also, if we want to evaluate our PCA model, the person we are fitting should not be included in the model. Python script generates PCA expression models for each model type (2) and expression (3), with 1 model including all faces and 12 with one person from the set discluded, giving a total of 78 PCA models. Ian Dewancker Internship Summary
- 42. Results The two PCA expression models were compared using the same fitted meshes as training data. Neutral faces were fit until convergence using only the expression model for deformation Ian Dewancker Internship Summary
- 43. Results The two PCA expression models were compared using the same fitted meshes as training data. Neutral faces were fit until convergence using only the expression model for deformation Raw Difference PCA Expression Mean Error (mm) Mean Error (90%) (mm) smile 0.906 0.574 mouthOpen 0.948 0.558 kiss 1.037 0.593 Ian Dewancker Internship Summary
- 44. Results The two PCA expression models were compared using the same fitted meshes as training data. Neutral faces were fit until convergence using only the expression model for deformation Raw Difference PCA Expression Mean Error (mm) Mean Error (90%) (mm) smile 0.906 0.574 mouthOpen 0.948 0.558 kiss 1.037 0.593 Deformation Transfer PCA Expression Mean Error (mm) Mean Error (90%) (mm) smile 0.904 0.582 mouthOpen 0.946 0.551 kiss 0.970 0.596 Ian Dewancker Internship Summary
- 45. Results The following screen captures are taken from the faceshift app after pressing the fit (automagic) button Using raw difference PCA expression model, scans from kinect Ian Dewancker Internship Summary
- 46. Results Ian Dewancker Internship Summary
- 47. Results Ian Dewancker Internship Summary
- 48. Future Work Smile and Mouth Open expressions look pretty good, however Kiss expression is not that great. Curling of lip not properly modeled. Lip vertices get contorted or flattened unnaturally. Ian Dewancker Internship Summary
- 49. Farewell Thanks everyone for an awesome term! Ian Dewancker Internship Summary