YouTube-8M: A Large-Scale Video Classification Benchmark (UPC Reading Group)
2 likes834 views
1. The document describes the YouTube-8M dataset, which contains over 8 million YouTube videos labeled with visual entities. It explores several baseline machine learning models for multi-label video classification on the dataset. 2. The best performing models were deep learning models that aggregated frame-level features, such as deep bag-of-frames pooling and LSTMs. These achieved mean average precision scores consistent with human ratings on a test set. 3. It also briefly introduces Google Cloud Machine Learning Engine, a cloud platform for training and deploying machine learning models at scale, which was used to train models on the YouTube-8M dataset.
![YouTube-8M: A Large-Scale Video Classification
Benchmark (and Google Cloud ML Engine)
Slides by Dídac Surís
ReadAI Reading Group, UPC
13th March, 2017
Sami Abu-El-Haija, Nisarg Kothari, Joonseok Lee, Paul
Natsev, George Toderici, Balakrishnan Varadarajan,
Sudheendra Vijayanarasimhan
[arxiv] (27 Sep 2016) [web]](https://image.slidesharecdn.com/04youtube-8m-alarge-scalevideoclassificationbenchmark-readai-170320185832/85/YouTube-8M-A-Large-Scale-Video-Classification-Benchmark-UPC-Reading-Group-1-320.jpg)
YouTube-8M: A Large-Scale Video Classification Benchmark (UPC Reading Group)
- 1. YouTube-8M: A Large-Scale Video Classification Benchmark (and Google Cloud ML Engine) Slides by Dídac Surís ReadAI Reading Group, UPC 13th March, 2017 Sami Abu-El-Haija, Nisarg Kothari, Joonseok Lee, Paul Natsev, George Toderici, Balakrishnan Varadarajan, Sudheendra Vijayanarasimhan [arxiv] (27 Sep 2016) [web]
- 2. Index 1. YouTube-8M a. Dataset b. Baseline approaches c. Results 2. Google Cloud ML Engine
- 3. Index 1. YouTube-8M a. Dataset b. Baseline approaches c. Results 2. Google Cloud ML Engine
- 4. YouTube-8M: Dataset Main features ● Multi-label (average 1.8) ● 4800 entities (24 top-level categories) ● 8, 264, 650 videos ● 500K hours of video ● Only visual entities ● Remove computational barriers
- 5. YouTube-8M: Dataset Obtention ● YouTube video annotation system (metadata, context, …) ● First step: define entities ○ Human ratings to define entities (only visual ones) ○ At least 200 videos per entity ● Second step: collect videos ○ 10 M randomly sampled videos ○ Discard according to several criteria ○ Split into train/validate/test
- 6. YouTube-8M: Dataset Feature Extraction ● 50 years of video real time: impractical ● Sampling at 1 frame per second ● Frame-level feature extraction: fetch the ReLu activation of the last hidden layer from the Inception network trained on ImageNet ● 2048 dimensions. With PCA + quantization size reduced 8x ● Audio features also extracted later: https://www.kaggle.com/c/youtube8m/discussion/29475
- 7. YouTube-8M: Dataset Not perfect ground truth ● 78.8 % precision ● 14.5 % recall
- 8. Index 1. YouTube-8M a. Dataset b. Baseline approaches c. Results 2. Google Cloud ML Engine
- 9. YouTube-8M: Baseline approaches Frame-level Training of 4800 independent one-vs-all classifiers 1. Average pooling + logistic ○ The frame-level probabilities are aggregated to the video-level using a simple average 2. Deep Bag of Frame (DBoF) Pooling ○ k frames projected to an M-dimensional space with RELU activations ○ Batch normalization ○ Aggregation of frames with max-pooling 3. LSTM ○ 2 LSTM layers with 1024 hidden units ○ Linearly increasing per-frame weights going from 1/N to 1 for the last frame.
- 10. YouTube-8M: Baseline approaches Video-level Only difference is that now we combine features before the neural network: fixed-length video features ● Mean, standard deviation, top 5 ordinal statistics ● Posterior normalization (subtract mean, PCA) Online learning algorithms instead of batch optimization (¿?) 1. Logistic regression 2. SVM (online) + Hinge loss 3. Mixture of Experts
- 11. Index 1. YouTube-8M a. Dataset b. Baseline approaches c. Results 2. Google Cloud ML Engine
- 12. YouTube-8M: Results Evaluation metrics and comparison ● Mean Average Precision (Precision, Recall) ● Hit @k ● Precision at equal recall rate (PERR) These are results on the validation set. On the human rated test set the results are consistent.
- 13. YouTube-8M: Results Results on other databases (transfer learning) ● Sports 1M ● Activity Net
- 14. Index 1. YouTube-8M a. Dataset b. Baseline approaches c. Results 2. Google Cloud ML Engine
- 15. Google Cloud Machine Learning Engine Basics ● Google Cloud Platform: 300 $ trial ● Google Cloud Shell ● Pricing ○ Training: in ML units (depending on scale tier) * hours ○ Prediction: Per hour + # of predictions ● Google Cloud Storage for the results
- 16. Google Cloud Machine Learning Engine Task submission
- 17. Google Cloud Machine Learning Engine TensorBoard