Shuffle and learn: Unsupervised Learning using Temporal Order Verification (UPC Reading Group)
4 likes715 views
The document discusses unsupervised learning, particularly in the context of video representation and temporal order verification. It highlights the significance of unsupervised learning frameworks like autoencoders and GANs, and the challenges involved in effectively sampling frame tuples for training. The conclusions emphasize the potential of temporal verification networks to understand video sequences better, while suggesting the need for further exploration in capturing longer temporal logics.
![Shuffle and Learn: Unsupervised Learning
using Temporal Order Verification
Slides by Xunyu Lin
ReadCV, UPC
20th February, 2017
Ishan Misra, C. Lawrence Zitnick, Martial Hebert
[arxiv] (26 July 2016) [code] [demo]](https://image.slidesharecdn.com/02shuffleandlearn-unsupervisedlearningusingtemporalorderverification-170220175721/85/Shuffle-and-learn-Unsupervised-Learning-using-Temporal-Order-Verification-UPC-Reading-Group-1-320.jpg)
Shuffle and learn: Unsupervised Learning using Temporal Order Verification (UPC Reading Group)
- 1. Shuffle and Learn: Unsupervised Learning using Temporal Order Verification Slides by Xunyu Lin ReadCV, UPC 20th February, 2017 Ishan Misra, C. Lawrence Zitnick, Martial Hebert [arxiv] (26 July 2016) [code] [demo]
- 2. Index 1. Introduction 2. Unsupervised Representations Learning 3. Video Representations Learning 4. Temporal Order Verification 5. In Practice 6. Evaluations 7. Conclusions
- 3. Index 1. Introduction 2. Unsupervised Representations Learning 3. Video Representations Learning 4. Temporal Order Verification 5. In Practice 6. Evaluations 7. Conclusions
- 4. Introduction What is Unsupervised Learning? ● Unsupervised Learning is the machine learning task of inferring a function to describe hidden structure from unlabeled data. ● The key of Unsupervised Learning is how to do clustering:
- 5. Introduction Why Unsupervised Learning? “Most of human and animal learning is unsupervised learning. If intelligence was a cake, unsupervised learning would be the cake, supervised learning would be the icing on the cake, and reinforcement learning would be the cherry on the cake. We know how to make the icing and the cherry, but we don’t know how to make the cake.” —— Yann LeCun
- 6. Introduction Why Unsupervised Learning? ● It is the nature of how intelligent beings percept the world. ● It can save us tons of efforts to build a human-alike intelligent agent compared to a totally supervised fashion. ● It’ll be the new breakthroughs to get true AI!
- 7. Index 1. Introduction 2. Unsupervised Representations Learning 3. Video Representations Learning 4. Temporal Order Verification 5. In Practice 6. Evaluations 7. Conclusions
- 8. Unsupervised Representations Learning Popular Unsupervised Representations Learning frameworks Auto-Encoder
- 9. Unsupervised Representations Learning Popular Unsupervised Representations Learning frameworks Variational Auto-Encoder (VAE) Tutorial
- 10. Unsupervised Representations Learning Popular Unsupervised Representations Learning frameworks GAN
- 11. Index 1. Introduction 2. Unsupervised Representations Learning 3. Video Representations Learning 4. Temporal Order Verification 5. In Practice 6. Evaluations 7. Conclusions
- 12. Video Representations Learning ● Human percept the world through observing the dynamic changing of our daily lives, which can be regarded as videos. ● Thus the unsupervised video representations learning plays an unneglectable role in building a human-alike intelligent agent.
- 13. Video Representations Learning Related Works Video Prediction with LSTMs
- 14. Video Representations Learning Related Works Spatiotemporally Coherent Reconstruction
- 15. Index 1. Introduction 2. Unsupervised Representations Learning 3. Video Representations Learning 4. Temporal Order Verification 5. In Practice 6. Evaluations 7. Conclusions
- 16. Temporal Order Verification The internal temporal order of videos
- 18. Temporal Order Verification Take temporal order as the supervisory signals for learning Shuffled sequences Binary classification In order Not in order
- 19. Index 1. Introduction 2. Unsupervised Representations Learning 3. Video Representations Learning 4. Temporal Order Verification 5. In Practice 6. Evaluations 7. Conclusions
- 20. In Practice How to sample the tuple of frames? 1. The number of frames for each tuple - 2 frames: may be ambiguous (picking up or placing down a cup?) - 3 frames: practically useful, but still not enough for a cyclical case - ...
- 21. In Practice How to sample the tuple of frames? a b c d e b c d ab d eb d Positive Negative Original Video
- 22. In Practice How to sample the tuple of frames? 2. Ambiguity in frames with small motion - The order of a small motion is indistinguishable. - Only sample from frames with high motion (smart sampling). - Use coarse frame level optical flow as a proxy to measure the motion between frames.
- 23. In Practice How to sample the tuple of frames? 3. The distance of frames in positive tuples (difficulty of the task) - Too close: results in ambiguous small motion or overly easy task - Too far: consecutive frames are not highly related which makes the learning task too difficult.
- 24. In Practice How to sample the tuple of frames? 3. The distance of frames in positive tuples (difficulty of the task) - Two metrics which control the difficulty of positive and negative samples. b c d ab d eb d
- 25. In Practice How to sample the tuple of frames? 4. Ratio of negative and positive samples
- 26. Index 1. Introduction 2. Unsupervised Representations Learning 3. Video Representations Learning 4. Temporal Order Verification 5. In Practice 6. Evaluations 7. Conclusions
- 27. Evaluation Action Recognition on UCF-101 & HMDB-51 - Comparison to random initialization & transfer learning - Pre-trained on ImageNet and finetuned on UCF-101 gives an accuracy of 67.1%. - Pre-trained on ImageNet and finetuned on HMDB-51 gives an accuracy of 28.5%. + 11.6 % + 4.8 % * UCF-101 is two times larger than HMDB-51
- 28. Evaluation Action Recognition on UCF-101 & HMDB-51 - Comparison to other unsupervised frameworks - Two Close: measure if two frames are close or not. - Two Order: temporal verification with only 2 frames. - DrLim: measure temporal coherency with L2 loss. - TempCoh: measure temporal coherency with L1 loss. - Obj. Patch: basically imitates human’s instinct eyes fixation ability. Paper link
- 30. Evaluation Visualizing pool5 Unit Responses
- 31. Evaluation Pose Estimation on FLIC & MPII
- 32. Index 1. Introduction 2. Unsupervised Representations Learning 3. Video Representations Learning 4. Temporal Order Verification 5. In Practice 6. Evaluations 7. Conclusions
- 33. Conclusions ● Temporal verification exploits the potential of a network to capture the sequential logics in videos. ● Further works should be explored by capturing a longer temporal logics. For now it only utilizes single frames in less than around 60 frames. Architectures like RNN could be further utilized to extend the temporal range. ● The only drawbacks lie in its weak constraint and tedious sampling techniques. ● More general constraint with simplified procedure? → My research line