Deep Audio and Vision (DLSL D2L4 2018 UPC Deep Learning for Speech and Language)
2 likes945 views
The document covers various techniques in audio-visual learning, focusing on the integration of visual features from video with audio representations using self-supervised learning methods. It discusses specific research contributions, including the exploitation of ambient sounds for visual learning and convolutional neural networks (CNNs) for sound representation. The document also highlights several studies related to cross-modal retrieval and speech synthesis from silent video.
![29
[Paper draft]
Cross-modal Retrieval
Surís, Didac, Amanda Duarte, Amaia Salvador, Jordi Torres, and Xavier Giró-i-Nieto. "Cross-modal
Embeddings for Video and Audio Retrieval." arXiv preprint arXiv:1801.02200 (2018).](https://image.slidesharecdn.com/d4l3audioandvision-180203101055/85/Deep-Audio-and-Vision-DLSL-D2L4-2018-UPC-Deep-Learning-for-Speech-and-Language-29-320.jpg)
Deep Audio and Vision (DLSL D2L4 2018 UPC Deep Learning for Speech and Language)
- 1. http://bit.ly/dlsl2018 Xavier Giro-i-Nieto xavier.giro@upc.edu Associate Professor Universitat Politecnica de Catalunya Technical University of Catalonia Audio and Vision Day 4 Lecture 3 #DLUPC
- 3. 3 Audio & Vision Vision Audio Speech Video Synchronization among modalities captured by video is exploited in a self-supervised manner.
- 4. 4 Audio & Vision ● Feature Learning ● Cross-modal Retrieval ● Cross-modal Translation
- 5. 5 Audio & Vision ● Feature Learning ● Cross-modal Retrieval ● Cross-modal Translation
- 7. 7 Visual Feature Learning Owens, Andrew, Jiajun Wu, Josh H. McDermott, William T. Freeman, and Antonio Torralba. "Ambient sound provides supervision for visual learning." ECCV 2016 Based on the assumption that ambient sound in video is related to the visual semantics.
- 8. 8 Owens, Andrew, Jiajun Wu, Josh H. McDermott, William T. Freeman, and Antonio Torralba. "Ambient sound provides supervision for visual learning." ECCV 2016 Use videos to train a CNN that predicts the audio statistics of a frame. Visual Feature Learning
- 9. 9 Task: Use the predicted audio stats to clusters images. Audio clusters built with K-means over training set Owens, Andrew, Jiajun Wu, Josh H. McDermott, William T. Freeman, and Antonio Torralba. "Ambient sound provides supervision for visual learning." ECCV 2016 Average statsCluster assignments at test time (one row=one cluster) Visual Feature Learning
- 10. 10 Although the CNN was not trained with class labels, local units with semantic meaning emerge. Owens, Andrew, Jiajun Wu, Josh H. McDermott, William T. Freeman, and Antonio Torralba. "Ambient sound provides supervision for visual learning." ECCV 2016 Visual Feature Learning
- 12. 12 Aytar, Yusuf, Carl Vondrick, and Antonio Torralba. "Soundnet: Learning sound representations from unlabeled video." NIPS 2016.
- 13. 13 Audio Feature Learning: SoundNet Aytar, Yusuf, Carl Vondrick, and Antonio Torralba. "Soundnet: Learning sound representations from unlabeled video." NIPS 2016 Pretrained visual ConvNets supervise the training of a model for sound representation
- 14. 14 Videos for training are unlabeled. Relies on Convnets trained on labeled images. Aytar, Yusuf, Carl Vondrick, and Antonio Torralba. "Soundnet: Learning sound representations from unlabeled video." NIPS 2016 Audio Feature Learning: SoundNet
- 15. 15 Hidden layers of Soundnet are used to train a standard SVM classifier that outperforms state of the art. Aytar, Yusuf, Carl Vondrick, and Antonio Torralba. "Soundnet: Learning sound representations from unlabeled video." NIPS 2016 Audio Feature Learning: SoundNet
- 16. 16 Visualization of the 1D filters over raw audio in conv1. Aytar, Yusuf, Carl Vondrick, and Antonio Torralba. "Soundnet: Learning sound representations from unlabeled video." NIPS 2016 Audio Feature Learning: SoundNet
- 17. 17 Visualization of the 1D filters over raw audio in conv1. Aytar, Yusuf, Carl Vondrick, and Antonio Torralba. "Soundnet: Learning sound representations from unlabeled video." NIPS 2016 Audio Feature Learning: SoundNet
- 18. 18 Visualize samples that mostly activate a neuron in a late layer (conv7) Aytar, Yusuf, Carl Vondrick, and Antonio Torralba. "Soundnet: Learning sound representations from unlabeled video." NIPS 2016 Audio Feature Learning: SoundNet
- 19. 19 Visualization of the video frames associated to the sounds that activate some of the last hidden units (conv7): Aytar, Yusuf, Carl Vondrick, and Antonio Torralba. "Soundnet: Learning sound representations from unlabeled video." NIPS 2016 Audio Feature Learning: SoundNet
- 20. 20 Hearing sounds that most activate a neuron in the sound network (conv7) Aytar, Yusuf, Carl Vondrick, and Antonio Torralba. "Soundnet: Learning sound representations from unlabeled video." NIPS 2016 Audio Feature Learning: SoundNet
- 21. 21 Hearing sounds that most activate a neuron in the sound network (conv5) Aytar, Yusuf, Carl Vondrick, and Antonio Torralba. "Soundnet: Learning sound representations from unlabeled video." NIPS 2016 Audio Feature Learning: SoundNet
- 22. 22 Vision Audio Audio & Visual Feature Learning Video
- 23. 2323Arandjelović, Relja, and Andrew Zisserman. "Look, Listen and Learn." ICCV 2017. Audio and visual features learned by assessing correspondence. Audio & Visual Feature Learning
- 24. 24 Audio & Vision ● Feature Learning ● Cross-modal retrieval ● Cross-modal Translation
- 25. 25 Owens, Andrew, Phillip Isola, Josh McDermott, Antonio Torralba, Edward H. Adelson, and William T. Freeman. "Visually indicated sounds." CVPR 2016.
- 26. 26 Cross-modal Retrieval Owens, Andrew, Phillip Isola, Josh McDermott, Antonio Torralba, Edward H. Adelson, and William T. Freeman. "Visually indicated sounds." CVPR 2016. Learn synthesized sounds from videos of people hitting objects with a drumstick.
- 27. 27 Owens, Andrew, Phillip Isola, Josh McDermott, Antonio Torralba, Edward H. Adelson, and William T. Freeman. "Visually indicated sounds." CVPR 2016. Not end-to-end Cross-modal Retrieval
- 28. 28 The Greatest Hits Dataset Owens, Andrew, Phillip Isola, Josh McDermott, Antonio Torralba, Edward H. Adelson, and William T. Freeman. "Visually indicated sounds." CVPR 2016. Cross-modal Retrieval
- 29. 29 [Paper draft] Cross-modal Retrieval Surís, Didac, Amanda Duarte, Amaia Salvador, Jordi Torres, and Xavier Giró-i-Nieto. "Cross-modal Embeddings for Video and Audio Retrieval." arXiv preprint arXiv:1801.02200 (2018).
- 30. 30 Best match Visual feature Audio feature Video sonorization Cross-modal Retrieval Surís, Didac, Amanda Duarte, Amaia Salvador, Jordi Torres, and Xavier Giró-i-Nieto. "Cross-modal Embeddings for Video and Audio Retrieval." arXiv preprint arXiv:1801.02200 (2018).
- 31. 31 Surís, Didac, Amanda Duarte, Amaia Salvador, Jordi Torres, and Xavier Giró-i-Nieto. "Cross-modal Embeddings for Video and Audio Retrieval." arXiv preprint arXiv:1801.02200 (2018). Visual feature Audio feature Best match Audio coloring Cross-modal Retrieval
- 32. 32 Audio & Vision ● Feature Learning ● Cross-modal retrieval ● Cross-modal Translation
- 33. 33 Audio & Vision Vision Speech Video
- 34. 34 Audio & Vision Vision Speech Video
- 35. 35 Ephrat, Ariel, Tavi Halperin, and Shmuel Peleg. "Improved speech reconstruction from silent video." In ICCV 2017 Workshop on Computer Vision for Audio-Visual Media. 2017.
- 36. 36 Ephrat et al. Vid2speech: Speech Reconstruction from Silent Video. ICASSP 2017 Speech Generation from Video CNN (VGG) Frame from a silent video Audio feature Post-hoc synthesis
- 37. 37 Speech Generation from Video Ephrat, Ariel, Tavi Halperin, and Shmuel Peleg. "Improved speech reconstruction from silent video." In ICCV 2017 Workshop on Computer Vision for Audio-Visual Media. 2017.
- 38. 38Chung, Joon Son, Amir Jamaludin, and Andrew Zisserman. "You said that?." BMVC 2017.
- 39. 39 Audio & Vision Vision Speech Video
- 40. 40Chung, Joon Son, Amir Jamaludin, and Andrew Zisserman. "You said that?." BMVC 2017. Speech to Video Synthesis (mouth)
- 41. 41 Karras, Tero, Timo Aila, Samuli Laine, Antti Herva, and Jaakko Lehtinen. "Audio-driven facial animation by joint end-to-end learning of pose and emotion." SIGGRAPH 2017
- 42. 42 Suwajanakorn, Supasorn, Steven M. Seitz, and Ira Kemelmacher-Shlizerman. "Synthesizing Obama: learning lip sync from audio." SIGGRAPH 2017. Speech to Video Synthesis (mouth)
- 43. 43 Karras, Tero, Timo Aila, Samuli Laine, Antti Herva, and Jaakko Lehtinen. "Audio-driven facial animation by joint end-to-end learning of pose and emotion." SIGGRAPH 2017
- 44. 44 Karras, Tero, Timo Aila, Samuli Laine, Antti Herva, and Jaakko Lehtinen. "Audio-driven facial animation by joint end-to-end learning of pose and emotion." SIGGRAPH 2017 Speech to Video Synthesis (pose & emotion)
- 45. 45 L. Chen, S. Srivastava, Z. Duan and C. Xu. Deep Cross-Modal Audio-Visual Generation. ACM Multimedia Thematic Workshops 2017. Audio & Visual Generation
- 47. 47 Audio & Vision ● Feature Learning ● Cross-modal retrieval ● Cross-modal Translation
- 48. 48 Questions ?