Gabriel Bianconi - Introduction to Face Processing with Computer Vision
Download as PPTX, PDF0 likes70 views
This document provides an overview of face processing techniques using computer vision and machine learning. It begins with definitions of face detection, recognition, and classification. Various approaches are reviewed, including traditional computer vision methods like HOG features as well as deep learning techniques using convolutional neural networks. Popular libraries for face processing tasks like face detection, landmark localization, and embeddings are demonstrated. Real-world applications of these techniques at companies like Facebook and Snapchat are also discussed.
![Images
2D Data x 1D Data = 3D Data
image[x][y][c]
Shape = (width, height, 3)](https://image.slidesharecdn.com/jm7wl0ltm66qdwueshnj-signature-5d1bd651097f8d6731765c5da0331cf03e8094c340a1bde578c3e3559cbc34e6-poli-190325214651/85/Gabriel-Bianconi-Introduction-to-Face-Processing-with-Computer-Vision-31-320.jpg)
![[[...,
[0.2, 0.6, 0.0],
[0.1, 0.1, 0.0],
[0.0, 0.0, 0.0],
[1.0, 0.4, 0.3],
[0.1, 0.5, 0.0],
...]]
≡](https://image.slidesharecdn.com/jm7wl0ltm66qdwueshnj-signature-5d1bd651097f8d6731765c5da0331cf03e8094c340a1bde578c3e3559cbc34e6-poli-190325214651/85/Gabriel-Bianconi-Introduction-to-Face-Processing-with-Computer-Vision-36-320.jpg)
![Face Embeddings
import face_recognition as fr
image = fr.load_image_file("file.jpg")
face_embedding = fr.face_encodings(image)[0]
github.com/ageitgey/face_recognition](https://image.slidesharecdn.com/jm7wl0ltm66qdwueshnj-signature-5d1bd651097f8d6731765c5da0331cf03e8094c340a1bde578c3e3559cbc34e6-poli-190325214651/85/Gabriel-Bianconi-Introduction-to-Face-Processing-with-Computer-Vision-52-320.jpg)
![Face Landmarks
import face_recognition as fr
image = fr.load_image_file("file.jpg")
face_landmarks = fr.face_landmarks(image)[0]
print(face_landmarks.keys())
# left_eyebrow, right_eyebrow, lower_lip, top_lip, …
github.com/ageitgey/face_recognition](https://image.slidesharecdn.com/jm7wl0ltm66qdwueshnj-signature-5d1bd651097f8d6731765c5da0331cf03e8094c340a1bde578c3e3559cbc34e6-poli-190325214651/85/Gabriel-Bianconi-Introduction-to-Face-Processing-with-Computer-Vision-57-320.jpg)
![Example: Snapchat Filters
from PIL import Image, ImageDraw
…
pil_image = Image.fromarray(image)
d = ImageDraw.Draw(pil_image, 'RGBA’)
lip_fill = (150, 0, 0, 128) # shade of red, 50% alpha
d.polygon(face_landmarks['top_lip'], fill=lip_fill)
d.polygon(face_landmarks['bottom_lip'], fill=lip_fill)
…
github.com/ageitgey/face_recognition](https://image.slidesharecdn.com/jm7wl0ltm66qdwueshnj-signature-5d1bd651097f8d6731765c5da0331cf03e8094c340a1bde578c3e3559cbc34e6-poli-190325214651/85/Gabriel-Bianconi-Introduction-to-Face-Processing-with-Computer-Vision-59-320.jpg)
Gabriel Bianconi - Introduction to Face Processing with Computer Vision
- 1. Gabriel Bianconi Introduction to Face Processing with Computer Vision
- 2. Gabriel Bianconi Founder, Scalar Research AI & Data Science Consulting Firm Previously at the Stanford AI Lab
- 3. Agenda 1. What is Face Processing? 2. Review of Computer Imagery 3. Intro to ML & ComputerVision 4. Face ProcessingTechniques & Examples
- 4. What is Face Processing?
- 5. Face Detection Localize faces in pictures
- 8. Face Detection Localize faces in pictures Enables software & users to focus on points of interest Used as input for more complex face processing tasks
- 9. Face Recognition Identify the faces
- 10. Source: Facebook
- 11. Source: Apple
- 12. Source: Apple
- 13. Face Classification Infer properties about the faces
- 14. Female
- 16. 39 8
- 17. Glasses
- 18. Closed Eyes
- 19. Face Manipulation Modify or add features to faces
- 20. Source: Teen Vogue / Snapchat
- 21. and many more…
- 24. Source: Sela, M. et al. (2017), “Unrestricted Facial Geometry Reconstruction Using Image-to-Image Translation”
- 25. Review of Computer Imagery
- 26. Image = Grid of Pixels
- 28. Pixels RGB Color Model = 3 channels (red, green, blue)
- 30. Source: AeroLab
- 31. Images 2D Data x 1D Data = 3D Data image[x][y][c] Shape = (width, height, 3)
- 32. Tensors “Matrices in higher dimensions” Intuition: nested arrays with fixed length
- 34. Image Data Tensor: (Width, Height, Channels) Specific Pixel: (X Position,Y Position, Channel)
- 35. Parameters iPhone X Photos 4032 x 3024 pixels = ~12M pixels (12 MP) ~12M pixels x 3 channels = ~36M parameters
- 36. [[..., [0.2, 0.6, 0.0], [0.1, 0.1, 0.0], [0.0, 0.0, 0.0], [1.0, 0.4, 0.3], [0.1, 0.5, 0.0], ...]] ≡
- 37. Intro to ML & ComputerVision
- 38. Approaches Traditional CV: Extract Features Model Inference Deep Learning: Neural Network Inference
- 39. HOG: Histogram of Oriented Gradients “Summarize” photo by splitting into a grid and computing gradient pattern in each cell
- 40. Source: Carcagnì, P. et al (2015), "Facial expression recognition and histograms of oriented gradients: a comprehensive study"
- 41. Sources: dlib; Rojas, M. et al (2011), "Automatic prediction of facial trait judgments: appearance vs. structural models."
- 42. Neural Networks (MLPs) Source: Dixon et al. (2016), “Classification-Based Financial Markets Prediction Using Deep Neural Networks”
- 43. Convolutional Neural Networks Source: MathWorks
- 44. Convolutional Neural Networks Source: Lee et al. (2009), “Convolutional Deep Belief Networks for Scalable Unsupervised Learning of Hierarchical Representations”
- 45. Neural Networks NNs require massive datasets, and training can be slow and tricky. However, pre-trained models can be used. Transfer learning greatly improves adaptation to new domains.
- 47. Face Detection CNN, HOG, etc.
- 48. Face Detection import face_recognition as fr image = fr.load_image_file("file.jpg") face_locations = fr.face_locations(image) github.com/ageitgey/face_recognition
- 49. Embeddings Map samples to a coordinate system Similar samples close to each other Dissimilar samples far from each other Words, graphs, …, faces
- 51. Face Embeddings “Similar” can be the same person, a physical characteristic, etc.
- 52. Face Embeddings import face_recognition as fr image = fr.load_image_file("file.jpg") face_embedding = fr.face_encodings(image)[0] github.com/ageitgey/face_recognition
- 53. Example: Facebook Photos 1. Collect known face embeddings from previously tagged photos and store in database. 2. Whenever a user uploads a new picture: detect face, compute embedding, search using distance function
- 54. - 0.31 0.59 0.69 0.31 - 0.52 0.63 0.59 0.52 - 0.50 0.69 0.63 0.50 - Source: Wikimedia
- 55. Face Landmarks Locate key points in face (e.g. eyes, nose, mouth, chin)
- 57. Face Landmarks import face_recognition as fr image = fr.load_image_file("file.jpg") face_landmarks = fr.face_landmarks(image)[0] print(face_landmarks.keys()) # left_eyebrow, right_eyebrow, lower_lip, top_lip, … github.com/ageitgey/face_recognition
- 58. Example: Snapchat Filters 1. Find face landmarks in picture 2. Draw lines or polygons around lips and eyes
- 59. Example: Snapchat Filters from PIL import Image, ImageDraw … pil_image = Image.fromarray(image) d = ImageDraw.Draw(pil_image, 'RGBA’) lip_fill = (150, 0, 0, 128) # shade of red, 50% alpha d.polygon(face_landmarks['top_lip'], fill=lip_fill) d.polygon(face_landmarks['bottom_lip'], fill=lip_fill) … github.com/ageitgey/face_recognition
- 61. Face Processing in Practice
- 62. Dozens of Libraries face_recognition OpenFace dlib FaceNet Tensorflow simplicity flexibility … and many more
- 63. Face Processing at Scale Microsoft Azure Face API Amazon AWS Rekognition Face++ … $1 / 1000 images or less!