Deep Within-Class Covariance Analysis for Robust Deep Audio Representation Learning
2 likes439 views
The document presents a study on deep within-class covariance analysis (DWCCA) for robust audio representation learning using convolutional neural networks (CNNs). It highlights the challenges of distribution mismatches between training and test datasets, emphasizing how such mismatches lead to increased within-class variability. The proposed DWCCA layer is shown to significantly reduce this variability and improve model generalization in scenarios with mismatched data distributions.
Deep Within-Class Covariance Analysis for Robust Deep Audio Representation Learning
- 2. Deep Within-Class Covariance Analysis for Robust Deep Audio Representation Learning Hamid Eghbal-zadeh 1,2 , Matthias Dorfer 1 , Gerhard Widmer 1,2 1 2
- 3. Deep Within-Class Covariance Analysis for Robust Deep Audio Representation Learning Hamid Eghbal-zadeh 1,2 , Matthias Dorfer 1 , Gerhard Widmer 1,2 1 2
- 4. Motivation Covariance Analysis WCCN DWCCA Results Summary Deep Within-Class Covariance Analysis for Robust Deep Audio Representation Learning Motivation
- 5. Motivation Covariance Analysis WCCN DWCCA Results Summary Deep Within-Class Covariance Analysis for Robust Deep Audio Representation Learning ● Convolutional Neural Networks learn useful features and build good representations
- 6. Motivation Covariance Analysis WCCN DWCCA Results Summary Deep Within-Class Covariance Analysis for Robust Deep Audio Representation Learning ● Convolutional Neural Networks learn useful features and build good representations ● CNNs are also known to generalize on the unseen data
- 7. Motivation Covariance Analysis WCCN DWCCA Results Summary Deep Within-Class Covariance Analysis for Robust Deep Audio Representation Learning ● Convolutional Neural Networks learn useful features and build good representations ● CNNs are also known to generalize on the unseen data ● Many of the benchmark datasets have similar train/test distributions
- 8. Motivation Covariance Analysis WCCN DWCCA Results Summary Deep Within-Class Covariance Analysis for Robust Deep Audio Representation Learning ● Convolutional Neural Networks learn useful features and build good representations ● CNNs are also known to generalize on the unseen data ● Many of the benchmark datasets have similar train/test distributions ● How about a distribution mismatch between training and test?
- 9. Motivation Covariance Analysis WCCN DWCCA Results Summary Deep Within-Class Covariance Analysis for Robust Deep Audio Representation Learning Distribution mismatch: When the distribution of the data in training and validation sets differ from the test set
- 10. Motivation Covariance Analysis WCCN DWCCA Results Summary Deep Within-Class Covariance Analysis for Robust Deep Audio Representation Learning Distribution mismatch: When the distribution of the data in training and validation sets differ from the test set ● Speaker Recognition: Training on English, testing on Chinese
- 11. Motivation Covariance Analysis WCCN DWCCA Results Summary Deep Within-Class Covariance Analysis for Robust Deep Audio Representation Learning Distribution mismatch: When the distribution of the data in training and validation sets differ from the test set ● Speaker Recognition: Training on English, testing on Chinese ● Acoustic Scene Classification: Training on Scenes in one country, testing on scenes of another country, in another period of time
- 12. Motivation Covariance Analysis WCCN DWCCA Results Summary Deep Within-Class Covariance Analysis for Robust Deep Audio Representation Learning Distribution mismatch: When the distribution of the data in training and validation sets differ from the test set ● Speaker Recognition: Training on English, testing on Chinese ● Acoustic Scene Classification: Training on Scenes in one country, testing on scenes of another country, in another period of time
- 13. Motivation Covariance Analysis WCCN DWCCA Results Summary Deep Within-Class Covariance Analysis for Robust Deep Audio Representation Learning Performance of end-to-end CNNs (no mismatch vs mismatched): ● We use DCASE2016 (no mismatch) and DCASE2017 (mismatched) datasets1 ● Same training and validation, different test set ● Look at several end-to-end CNNs 1) Detection and Classification of Acoustic Scenes and Events, http://dcase.community
- 14. Motivation Covariance Analysis WCCN DWCCA Results Summary Deep Within-Class Covariance Analysis for Robust Deep Audio Representation Learning Covariance Analysis of the representation
- 15. Motivation Covariance Analysis WCCN DWCCA Results Summary Deep Within-Class Covariance Analysis for Robust Deep Audio Representation Learning Covariance Eigenvalue Analysis: ● We train a VGG network on No mismatch and Mismatched using spectrograms
- 16. Motivation Covariance Analysis WCCN DWCCA Results Summary Deep Within-Class Covariance Analysis for Robust Deep Audio Representation Learning Covariance Eigenvalue Analysis: ● We train a VGG network on No mismatch and Mismatched using spectrograms ● We analyse the internal representation of the VGG
- 17. Motivation Covariance Analysis WCCN DWCCA Results Summary Deep Within-Class Covariance Analysis for Robust Deep Audio Representation Learning Covariance Eigenvalue Analysis: ● We train a VGG network on No mismatch and Mismatched using spectrograms ● We analyse the internal representation of the VGG ● We use covariance analysis ○ Eigen-values of the covariances matrix ○ Visualisation of the representations projected via PCA
- 18. Motivation Covariance Analysis WCCN DWCCA Results Summary Deep Within-Class Covariance Analysis for Robust Deep Audio Representation Learning Nomismatch Covariance Eigenvalue Analysis: Train Test Mismatched Validation
- 19. Motivation Covariance Analysis WCCN DWCCA Results Summary Deep Within-Class Covariance Analysis for Robust Deep Audio Representation Learning NomismatchVisualisation of the VGG representations: Train Validation Test Mismatched
- 20. Motivation Covariance Analysis WCCN DWCCA Results Summary Deep Within-Class Covariance Analysis for Robust Deep Audio Representation Learning Within-Class Covariance Normalisation (WCCN)
- 21. Motivation Covariance Analysis WCCN DWCCA Results Summary Deep Within-Class Covariance Analysis for Robust Deep Audio Representation Learning Within-Class Covariance Normalization1,2 : ● Proposed for Speaker Recognition to reduce the false positive/negatives 1) Hatch, Andrew O., et al. "Within-class covariance normalization for SVM-based speaker recognition." Ninth international conference on spoken language processing. 2006. 2) Hatch, Andrew O., et al. "Generalized linear kernels for one-versus-all classification: application to speaker recognition." Acoustics, Speech and Signal Processing, 2006. ICASSP 2006 Proceedings. 2006 IEEE International Conference on. Vol. 5. IEEE, 2006.
- 22. Motivation Covariance Analysis WCCN DWCCA Results Summary Deep Within-Class Covariance Analysis for Robust Deep Audio Representation Learning Within-Class Covariance Normalization1,2 : ● Proposed for Speaker Recognition to reduce the false positive/negatives ● Used to reduce the within-class variability in features such as GMM supervectors or i-vector features 1) Hatch, Andrew O., et al. "Within-class covariance normalization for SVM-based speaker recognition." Ninth international conference on spoken language processing. 2006. 2) Hatch, Andrew O., et al. "Generalized linear kernels for one-versus-all classification: application to speaker recognition." Acoustics, Speech and Signal Processing, 2006. ICASSP 2006 Proceedings. 2006 IEEE International Conference on. Vol. 5. IEEE, 2006.
- 23. Motivation Covariance Analysis WCCN DWCCA Results Summary Deep Within-Class Covariance Analysis for Robust Deep Audio Representation Learning Within-Class Covariance Normalization1,2 : 1) Hatch, Andrew O., et al. "Within-class covariance normalization for SVM-based speaker recognition." Ninth international conference on spoken language processing. 2006. 2) Hatch, Andrew O., et al. "Generalized linear kernels for one-versus-all classification: application to speaker recognition." Acoustics, Speech and Signal Processing, 2006. ICASSP 2006 Proceedings. 2006 IEEE International Conference on. Vol. 5. IEEE, 2006.
- 24. Motivation Covariance Analysis WCCN DWCCA Results Summary Deep Within-Class Covariance Analysis for Robust Deep Audio Representation Learning Deep Within-Class Covariance Analysis
- 25. Motivation Covariance Analysis WCCN DWCCA Results Summary Deep Within-Class Covariance Analysis for Robust Deep Audio Representation Learning Deep Within-Class Covariance Analysis (DWCCA): ● A deep learning compatible version of WCCN
- 26. Motivation Covariance Analysis WCCN DWCCA Results Summary Deep Within-Class Covariance Analysis for Robust Deep Audio Representation Learning Deep Within-Class Covariance Analysis (DWCCA): ● A deep learning compatible version of WCCN ● A statistical DL layer, trained end-to-end using SGD with minibatches
- 27. Motivation Covariance Analysis WCCN DWCCA Results Summary Deep Within-Class Covariance Analysis for Robust Deep Audio Representation Learning Deep Within-Class Covariance Analysis (DWCCA): ● A deep learning compatible version of WCCN ● A statistical DL layer, trained end-to-end using SGD with minibatches ● Can be placed anywhere to reduce the within-class variability
- 28. Motivation Covariance Analysis WCCN DWCCA Results Summary Deep Within-Class Covariance Analysis for Robust Deep Audio Representation Learning Deep Within-Class Covariance Analysis (DWCCA): ● A deep learning compatible version of WCCN ● A statistical DL layer, trained end-to-end using SGD with minibatches ● Can be placed anywhere to reduce the within-class variability ● B in training is equal to Bb in forward pass
- 29. Motivation Covariance Analysis WCCN DWCCA Results Summary Deep Within-Class Covariance Analysis for Robust Deep Audio Representation Learning Deep Within-Class Covariance Analysis (DWCCA): ● A deep learning compatible version of WCCN ● A statistical DL layer, trained end-to-end using SGD with minibatches ● Can be placed anywhere to reduce the within-class variability ● B in training is equal to Bb in forward pass ● Gradients wrt B are computed and used in backward pass
- 30. Motivation Covariance Analysis WCCN DWCCA Results Summary Deep Within-Class Covariance Analysis for Robust Deep Audio Representation Learning Deep Within-Class Covariance Analysis (DWCCA): ● A deep learning compatible version of WCCN ● A statistical DL layer, trained end-to-end using SGD with minibatches ● Can be placed anywhere to reduce the within-class variability ● B in training is equal to Bb in forward pass ● Gradients wrt B are computed and used in backward pass ● A running average is computed for test time (similar to batchnorm)
- 31. Motivation Covariance Analysis WCCN DWCCA Results Summary Deep Within-Class Covariance Analysis for Robust Deep Audio Representation Learning Deep Within-Class Covariance Analysis (DWCCA): ● A deep learning compatible version of WCCN ● A statistical DL layer, trained end-to-end using SGD with minibatches ● Can be placed anywhere to reduce the within-class variability ● B in training is equal to Bb in forward pass ● Gradients wrt B are computed and used in backward pass ● A running average is computed for test time (similar to batchnorm) ● Compatible with different supervised tasks (Classification, Detection, metric learning...) and data (raw audio...)
- 32. Motivation Covariance Analysis WCCN DWCCA Results Summary Deep Within-Class Covariance Analysis for Robust Deep Audio Representation Learning Deep Within-Class Covariance Analysis (DWCCA): ● A deep learning compatible version of WCCN ● A statistical DL layer, trained end-to-end using SGD with minibatches ● Can be placed anywhere to reduce the within-class variability ● B in training is equal to Bb in forward pass ● Gradients wrt B are computed and used in backward pass ● A running average is computed for test time (similar to batchnorm) ● Compatible with different supervised tasks (Classification, Detection, metric learning...) and data (raw audio...) ● Can be used with different supervised losses (CCE, BCE, l2 , ...)
- 33. Motivation Covariance Analysis WCCN DWCCA Results Summary Deep Within-Class Covariance Analysis for Robust Deep Audio Representation Learning Results
- 34. Motivation Covariance Analysis WCCN DWCCA Results Summary Deep Within-Class Covariance Analysis for Robust Deep Audio Representation Learning Nomismatch Within-Class Covariance Eigenvalue Analysis (Without DWCCA): Train Validation Test Mismatched
- 35. Motivation Covariance Analysis WCCN DWCCA Results Summary Deep Within-Class Covariance Analysis for Robust Deep Audio Representation Learning Nomismatch Within-Class Covariance Eigenvalue Analysis (With DWCCA): Train Test Mismatched Validation
- 36. Motivation Covariance Analysis WCCN DWCCA Results Summary Deep Within-Class Covariance Analysis for Robust Deep Audio Representation Learning Nomismatch Eigenvalue Analysis (With vs without DWCCA): Train Test Mismatched Validation
- 37. Motivation Covariance Analysis WCCN DWCCA Results Summary Deep Within-Class Covariance Analysis for Robust Deep Audio Representation Learning Nomismatch K-NN classification results on VGG representations Validation Test Mismatched
- 38. Motivation Covariance Analysis WCCN DWCCA Results Summary Deep Within-Class Covariance Analysis for Robust Deep Audio Representation Learning *: Single model, Single-channel features : Multi-channel features :Ensemble of various models NomismatchMismatched End-to-end classification:
- 39. Motivation Covariance Analysis WCCN DWCCA Results Summary Deep Within-Class Covariance Analysis for Robust Deep Audio Representation Learning *: Single model, Single-channel features : Multi-channel features :Ensemble of various models NomismatchMismatched End-to-end classification:
- 40. Motivation Covariance Analysis WCCN DWCCA Results Summary Deep Within-Class Covariance Analysis for Robust Deep Audio Representation Learning *: Single model, Single-channel features : Multi-channel features :Ensemble of various models NomismatchMismatched End-to-end classification:
- 41. Motivation Covariance Analysis WCCN DWCCA Results Summary Deep Within-Class Covariance Analysis for Robust Deep Audio Representation Learning *: Single model, Single-channel features : Multi-channel features :Ensemble of various models NomismatchMismatched End-to-end classification:
- 42. Motivation Covariance Analysis WCCN DWCCA Results Summary Deep Within-Class Covariance Analysis for Robust Deep Audio Representation Learning *: Single model, Single-channel features : Multi-channel features :Ensemble of various models NomismatchMismatched End-to-end classification:
- 43. Motivation Covariance Analysis WCCN DWCCA Results Summary Deep Within-Class Covariance Analysis for Robust Deep Audio Representation Learning MismatchedNo mismatch End-to-end class-wise F1:
- 44. Motivation Covariance Analysis WCCN DWCCA Results Summary Deep Within-Class Covariance Analysis for Robust Deep Audio Representation Learning MismatchedNo mismatch End-to-end class-wise F1:
- 45. Motivation Covariance Analysis WCCN DWCCA Results Summary Deep Within-Class Covariance Analysis for Robust Deep Audio Representation Learning Summary
- 46. Motivation Covariance Analysis WCCN DWCCA Results Summary Deep Within-Class Covariance Analysis for Robust Deep Audio Representation Learning Summary: ● We analysed covariance of the representations in a VGG network Nomismatch Train Test Mismatched Validation
- 47. Motivation Covariance Analysis WCCN DWCCA Results Summary Deep Within-Class Covariance Analysis for Robust Deep Audio Representation Learning Summary: ● We analysed covariance of the representations in a VGG network ● We showed that the more mismatch there is between training and test, the more within-class variability increases in the representation Nomismatch Train Test Mismatched Validation
- 48. Motivation Covariance Analysis WCCN DWCCA Results Summary Deep Within-Class Covariance Analysis for Robust Deep Audio Representation Learning Summary: ● We analysed covariance of the representations in a VGG network ● We showed that the more mismatch there is between training and test, the more within-class variability increases in the representation ● We proposed Deep Within-class Covariance Analysis, a deep learning compatible layer capable of significantly reducing within-class variability of a network’s representation
- 49. Motivation Covariance Analysis WCCN DWCCA Results Summary Deep Within-Class Covariance Analysis for Robust Deep Audio Representation Learning Summary: ● We analysed covariance of the representations in a VGG network ● We showed that the more mismatch there is between training and test, the more within-class variability increases in the representation ● We proposed Deep Within-class Covariance Analysis, a deep learning compatible layer capable of significantly reducing within-class variability of a network’s representation ● We empirically showed that DWCCA improves the generalisation when the training and test have mismatched distributions. Nomismatch Validation Test Mismatched
- 50. Motivation Covariance Analysis WCCN DWCCA Results Summary Deep Within-Class Covariance Analysis for Robust Deep Audio Representation Learning Thank you for your attention! Come to the poster for more discussions. hamid.eghbal-zadeh@jku.at heghbalz