![Automatic
Tagging Using
Deep
Convolutional
Neural
Networks
Keunwoo.Choi
@qmul.ac.uk
Problem
definition
The proposed
architectures
Experiments
MagnaTagATune
MSD: Reported
(and incorrect)
results
MSD: Correct
results
Conclusions
Experiments and discussions
MagnaTagATune - Number of layers
Methods AUC
FCN-3, mel-spectrogram .852
FCN-4, mel-spectrogram .894
FCN-5, mel-spectrogram .890
FCN-4, STFT .846
FCN-4, MFCC .862
FCN-4>FCN-3: Depth worked!
FCN-4>FCN-5 by .004
Deeper model might make it equal after ages of training
Deeper models requires more data
Deeper models take more time (deep residual network[4])
4 layers are enough vs. matter of size(data)?
10/19](https://image.slidesharecdn.com/slide-ismir-2016-160818032005/85/Automatic-Tagging-using-Deep-Convolutional-Neural-Networks-ISMIR-2016-10-320.jpg)
![Automatic
Tagging Using
Deep
Convolutional
Neural
Networks
Keunwoo.Choi
@qmul.ac.uk
Problem
definition
The proposed
architectures
Experiments
MagnaTagATune
MSD: Reported
(and incorrect)
results
MSD: Correct
results
Conclusions
Experiments and discussions
MagnaTagATune
Methods AUC
The proposed system, FCN-4 .894
2015, Bag of features and RBM [5] .888
2014, 1-D convolutions[2] .882
2014, Transferred learning [6] .88
2012, Multi-scale approach [1] .898
2011, Pooling MFCC [3] .861
All deep and NN approaches are around .88-.89
Are we touching the glass ceiling?
Perhaps due to the noise of MTT, but tricky to prove it
26K tracks are not enough for millions of parameters
11/19](https://image.slidesharecdn.com/slide-ismir-2016-160818032005/85/Automatic-Tagging-using-Deep-Convolutional-Neural-Networks-ISMIR-2016-11-320.jpg)
![Automatic
Tagging Using
Deep
Convolutional
Neural
Networks
Keunwoo.Choi
@qmul.ac.uk
Problem
definition
The proposed
architectures
Experiments
MagnaTagATune
MSD: Reported
(and incorrect)
results
MSD: Correct
results
Conclusions
Smaller (narrower) convnet
No. of feature maps: [128@1 – 2048@5] → [32@1 – 256@5],
i.e.narrower network, because there’s no difference between
FCN-4 and FCN-5.
100 101 102
Number of epochs
0.50
0.55
0.60
0.65
0.70
0.75
0.80
0.85
0.90
AUC MSD, small convnet, log
FCN3_small
FCN4_small
FCN5_small
15/19](https://image.slidesharecdn.com/slide-ismir-2016-160818032005/85/Automatic-Tagging-using-Deep-Convolutional-Neural-Networks-ISMIR-2016-15-320.jpg)
Automatic Tagging using Deep Convolutional Neural Networks - ISMIR 2016
- 1. Automatic Tagging Using Deep Convolutional Neural Networks Keunwoo.Choi @qmul.ac.uk Problem definition The proposed architectures Experiments Automatic Tagging Using Deep Convolutional Neural Networks Keunwoo.Choi @qmul.ac.uk Centre for Digital Music, Queen Mary University of London, UK Ɓ @keunwoochoi 11 Aug 2016, ISMIR 2016, NY 1/19
- 2. Automatic Tagging Using Deep Convolutional Neural Networks Keunwoo.Choi @qmul.ac.uk Problem definition The proposed architectures Experiments 1 Problem definition What is auto-tagging? 2 The proposed architectures But why? 3 Experiments MagnaTagATune MSD: Reported (and incorrect) results MSD: Correct results Conclusions 2/19
- 3. Automatic Tagging Using Deep Convolutional Neural Networks Keunwoo.Choi @qmul.ac.uk Problem definition What is auto-tagging? The proposed architectures Experiments Problem definition What is auto-tagging? Tags Descriptive keywords that people (just) put on music Multi-label nature E.g. {rock, guitar, drive, 90’s} Music tags include Genres (rock, pop, alternative, indie), Instruments (vocalists, guitar, violin), Emotions (mellow, chill), Activities (party, drive), Eras (00’s, 90’s, 80’s). Collaboratively created (Last.fm ) → noisy and ill-defined (of course) false negative synonyms (vocal/vocals/vocalist/vocalists/voice/voices. guitar/guitars) popularity bias typo (harpsicord) irrelevant tags (abcd, ilikeit, fav) 3/19
- 4. Automatic Tagging Using Deep Convolutional Neural Networks Keunwoo.Choi @qmul.ac.uk Problem definition What is auto-tagging? The proposed architectures Experiments Problem definition What is auto-tagging? Multi-label classification Criteria: AUC-ROC (Area Under an ROC Curve) 0.5 <= AUC-ROC <= 1.0 Robust to unbalanced datasets Higher if lower false positive rate Higher if higher true positive rate 4/19
- 5. Automatic Tagging Using Deep Convolutional Neural Networks Keunwoo.Choi @qmul.ac.uk Problem definition The proposed architectures But why? Experiments The proposed architectures 1×96×1366 melgram → conv’s/pooling’s → 2048×1×1 All ReLU All 3x3 convolutions 2048 feature maps at the end 3,4,5,6,7 layers 5/19
- 6. Automatic Tagging Using Deep Convolutional Neural Networks Keunwoo.Choi @qmul.ac.uk Problem definition The proposed architectures But why? Experiments Assumptions Why (I think) would it work? conv-MP-conv-MP-conv-MP.. N × M Convolution: There are some useful patterns in input and feature maps that are local, location-invariant, and equal or smaller than N × M. L × K Max-Pooling: We are generous up to L × K so we allow variances within this range. Which means, We see big picture, some macroscopic patterns ...assuming/hoping that they are related to tag 6/19
- 7. Automatic Tagging Using Deep Convolutional Neural Networks Keunwoo.Choi @qmul.ac.uk Problem definition The proposed architectures Experiments MagnaTagATune MSD: Reported (and incorrect) results MSD: Correct results Conclusions Experiments and discussions MTT MSD # tracks 25k 214K (out of total 1M) # songs 5-6k 214K (out of total 1M) Length 29.1s 30-60s Benchmarks 10+ 0 Labels Tags, genres Tags, genres, EchoNest features, bag-of-word lyrics,... 7/19
- 8. Automatic Tagging Using Deep Convolutional Neural Networks Keunwoo.Choi @qmul.ac.uk Problem definition The proposed architectures Experiments MagnaTagATune MSD: Reported (and incorrect) results MSD: Correct results Conclusions Experiments and discussions For Dataset Specificaions Input representation MTT STFT/MFCC/Melgram # Layers MTT 3/4/5/6/7 Benchmark MTT FCN-4 vs 5 previous methods # Layers1 MSD 3/4/5 # Layers2 MSD 3/4/5, Narrower structure 1 Different from the paper 2 Not in the paper 8/19
- 9. Automatic Tagging Using Deep Convolutional Neural Networks Keunwoo.Choi @qmul.ac.uk Problem definition The proposed architectures Experiments MagnaTagATune MSD: Reported (and incorrect) results MSD: Correct results Conclusions Experiments and discussions MagnaTagATune - Input representations Same depth (l=4), melgram>MFCC>STFT melgram: 96 mel-frequency bins STFT: 128 frequency bins MFCC: 90 (30 MFCC, 30 MFCCd, 30 MFCCdd) Methods AUC FCN-3, mel-spectrogram .852 FCN-4, mel-spectrogram .894 FCN-5, mel-spectrogram .890 FCN-4, STFT .846 FCN-4, MFCC .862 Still, ConvNet may outperform frequency aggregation than mel-frequency (if there’s more data). But not yet. ConvNet outperformed MFCC 9/19
- 10. Automatic Tagging Using Deep Convolutional Neural Networks Keunwoo.Choi @qmul.ac.uk Problem definition The proposed architectures Experiments MagnaTagATune MSD: Reported (and incorrect) results MSD: Correct results Conclusions Experiments and discussions MagnaTagATune - Number of layers Methods AUC FCN-3, mel-spectrogram .852 FCN-4, mel-spectrogram .894 FCN-5, mel-spectrogram .890 FCN-4, STFT .846 FCN-4, MFCC .862 FCN-4>FCN-3: Depth worked! FCN-4>FCN-5 by .004 Deeper model might make it equal after ages of training Deeper models requires more data Deeper models take more time (deep residual network[4]) 4 layers are enough vs. matter of size(data)? 10/19
- 11. Automatic Tagging Using Deep Convolutional Neural Networks Keunwoo.Choi @qmul.ac.uk Problem definition The proposed architectures Experiments MagnaTagATune MSD: Reported (and incorrect) results MSD: Correct results Conclusions Experiments and discussions MagnaTagATune Methods AUC The proposed system, FCN-4 .894 2015, Bag of features and RBM [5] .888 2014, 1-D convolutions[2] .882 2014, Transferred learning [6] .88 2012, Multi-scale approach [1] .898 2011, Pooling MFCC [3] .861 All deep and NN approaches are around .88-.89 Are we touching the glass ceiling? Perhaps due to the noise of MTT, but tricky to prove it 26K tracks are not enough for millions of parameters 11/19
- 12. Automatic Tagging Using Deep Convolutional Neural Networks Keunwoo.Choi @qmul.ac.uk Problem definition The proposed architectures Experiments MagnaTagATune MSD: Reported (and incorrect) results MSD: Correct results Conclusions Experiments and discussions Million Song Dataset - on the paper Methods AUC FCN-3, mel-spectrogram .786 FCN-4, — .808 FCN-5, — .848 FCN-6, — .851 FCN-7, — .845 12/19
- 13. Automatic Tagging Using Deep Convolutional Neural Networks Keunwoo.Choi @qmul.ac.uk Problem definition The proposed architectures Experiments MagnaTagATune MSD: Reported (and incorrect) results MSD: Correct results Conclusions WARNING! The MSD results are not reproduced. I suspect a incorrect learning rate controlling and this is why we shouldn’t rush before deadline.. Ran the experiments again without weird learning rate controlling, and more epochs (240→480) 13/19
- 14. Automatic Tagging Using Deep Convolutional Neural Networks Keunwoo.Choi @qmul.ac.uk Problem definition The proposed architectures Experiments MagnaTagATune MSD: Reported (and incorrect) results MSD: Correct results Conclusions Experiments and discussions Million Song Dataset - re-run Methods AUC FCN-3, mel-spectrogram .839 FCN-4, — .852 FCN-5, — .855 100 101 102 Number of epochs 0.50 0.55 0.60 0.65 0.70 0.75 0.80 0.85 0.90 AUC MSD in Log FCN3 FCN4 FCN5 14/19
- 15. Automatic Tagging Using Deep Convolutional Neural Networks Keunwoo.Choi @qmul.ac.uk Problem definition The proposed architectures Experiments MagnaTagATune MSD: Reported (and incorrect) results MSD: Correct results Conclusions Smaller (narrower) convnet No. of feature maps: [128@1 – 2048@5] → [32@1 – 256@5], i.e.narrower network, because there’s no difference between FCN-4 and FCN-5. 100 101 102 Number of epochs 0.50 0.55 0.60 0.65 0.70 0.75 0.80 0.85 0.90 AUC MSD, small convnet, log FCN3_small FCN4_small FCN5_small 15/19
- 16. Automatic Tagging Using Deep Convolutional Neural Networks Keunwoo.Choi @qmul.ac.uk Problem definition The proposed architectures Experiments MagnaTagATune MSD: Reported (and incorrect) results MSD: Correct results Conclusions Conclusions Assumptions - about macroscopic view seems fine In general, the behaviour agrees with computer vision community, which are.. the deeper, the better (or equal) the wider, the better (or equal), but not as much as depth Melgram+feature learning > MFCC Melgram > STFT At some point, we will argue STFT + learning > melgram MTT is too small, even MSD might be small Future work: More investigation, variable input length, better dataset, re-thinking the problem... 16/19
- 17. Automatic Tagging Using Deep Convolutional Neural Networks Keunwoo.Choi @qmul.ac.uk Problem definition The proposed architectures Experiments MagnaTagATune MSD: Reported (and incorrect) results MSD: Correct results Conclusions Thank you for listening and... You can plug-and-predict The pre-trained weights and model is open! https://github.com/keunwoochoi/music-auto_tagging-keras 17/19
- 18. Automatic Tagging Using Deep Convolutional Neural Networks Keunwoo.Choi @qmul.ac.uk Problem definition The proposed architectures Experiments MagnaTagATune MSD: Reported (and incorrect) results MSD: Correct results Conclusions References I Dieleman, S., Schrauwen, B.: Multiscale approaches to music audio feature learning. In: ISMIR. pp. 3–8 (2013) Dieleman, S., Schrauwen, B.: End-to-end learning for music audio. In: Acoustics, Speech and Signal Processing (ICASSP), 2014 IEEE International Conference on. pp. 6964–6968. IEEE (2014) Hamel, P., Lemieux, S., Bengio, Y., Eck, D.: Temporal pooling and multiscale learning for automatic annotation and ranking of music audio. In: ISMIR. pp. 729–734 (2011) He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. arXiv preprint arXiv:1512.03385 (2015) 18/19
- 19. Automatic Tagging Using Deep Convolutional Neural Networks Keunwoo.Choi @qmul.ac.uk Problem definition The proposed architectures Experiments MagnaTagATune MSD: Reported (and incorrect) results MSD: Correct results Conclusions References II Nam, J., Herrera, J., Lee, K.: A deep bag-of-features model for music auto-tagging. arXiv preprint arXiv:1508.04999 (2015) Van Den Oord, A., Dieleman, S., Schrauwen, B.: Transfer learning by supervised pre-training for audio-based music classification. In: Conference of the International Society for Music Information Retrieval (ISMIR 2014) (2014) 19/19