“Comparing ML-Based Audio with ML-Based Vision: An Introduction to ML Audio for ML Vision Engineers,” a Presentation from DSP Concepts
0 likes51 views
The document introduces a comparison between machine learning (ML)-based audio and vision, highlighting the growing significance and capabilities of audio processing. It discusses audio feature extraction techniques, similarities and differences with vision processing methods, and presents common problems in both fields, such as classification and denoising. Additionally, it outlines the Audio Weaver platform that facilitates faster audio experience design through a modular approach for product developers.
“Comparing ML-Based Audio with ML-Based Vision: An Introduction to ML Audio for ML Vision Engineers,” a Presentation from DSP Concepts
- 1. Comparing ML-Based Audio with ML-Based Vision: An Introduction to ML Audio for ML Vision Engineers Josh Morris Engineering Manager, Machine Learning DSP Concepts
- 2. The Audio of Things Approach DSP Concepts helps product makers deliver remarkable Audio Experience through a flexible and modular approach within a design platform environment. This system makes the entire workflow faster and easier across prototyping, design, debugging, tuning, production, and even over-the-air updates. The Audio Experience Design platform that accelerates feature development and makes audio innovation easy Audio IP • Building blocks to product-ready solutions Prototyping Hardware • Test and verify performance Debug Tools • Pinpoint issues and reduce risk Tuning Tools • Optimize playback and voice control Expertise • Algorithm, software, tuning, testing Worldwide Support Santa Clara, Stuttgart, Taipei, Seoul © 2022 DSP Concepts 2
- 3. Processing at the edge is getting more and more powerful making it possible to do things that were reserved for the cloud Audio is becoming increasingly popular • Standalone applications • Smart assistant • Voice control • Denoising • Multi-modal applications • Industrial sensing • Anomaly detection Motivation for Talk © 2022 DSP Concepts 3
- 4. • Feature engineering • How is audio different from vision? • How is audio like vision? • Implementation • Similarities • Differences • Common problems in vision and their audio analogues • Classification • Sequence decoding • Restoration/denoising Agenda © 2022 DSP Concepts 4
- 5. Features
- 6. • Audio is a sequence of samples • Somewhere between video/images • Inherent left to right structure to data • Sample rate • Bit depth How Are Audio Features Different from Vision Features? © 2022 DSP Concepts https://en.wikipedia.org/wiki/Sampling_(signal_processing) https://en.wikipedia.org/wiki/File:Mike_Austin_Sequence.JPG 6
- 7. A lot of techniques employed for ML audio-based solutions borrow techniques from vision. This means we need to take a 1D sequence of samples and make them look like an image. Manipulating Audio from Time Domain to Frequency Domain © 2022 DSP Concepts https://en.wikipedia.org/wiki/Sampling_(signal_processing) https://upload.wikimedia.org/wikipedia/commons/c/c5/Spectrogram-19thC.png 7
- 8. • Fast Fourier transform (FFT) • Shows frequency over time • Linearly spaced frequency bins • Can apply processing in the frequency domain and then use an inverse fast Fourier transform (IFFT) to get time domain audio Fast Fourier Transform © 2022 DSP Concepts https://commons.wikimedia.org/wiki/File:FFT-Time-Frequency-View.png 8
- 9. • Built from short time Fourier transform • Repeat Fourier transform with set window and hop size • short-time Fourier transform (STFT) • Take magnitude squared of frequency bins • Common for classification tasks Power Spectrogram © 2022 DSP Concepts https://upload.wikimedia.org/wikipedia/commons/9/99/Mount_Rainier_soundscape.jpg 9
- 10. • Constant Q transform (CQT) • Logarithmically spaced frequency bins • Popular for musical applications • More computationally efficient since fewer bins are needed to cover a frequency range Constant Q Transform © 2022 DSP Concepts https://en.wikipedia.org/wiki/Constant-Q_transform#/media/File:CQT-piano-chord.png 10
- 11. • Mel spectrogram • Triangular frequency windows • Filter banks that attempt to approximate human hearing • Humans struggle to hear frequencies that are close together. This anomaly is known as masking. Mel Spectrogram © 2022 DSP Concepts http://www.ifp.illinois.edu/~minhdo/teaching/speaker_recognition/speaker_recognition.html 11
- 12. • Feature normalization • Multi-channel features for complex audio • Color channels in audio • Real and imaginary components in frequency time How Are Audio Features the Same as Vision? © 2022 DSP Concepts https://en.wikipedia.org/wiki/Fourier_transform#/media/File:Rising_circular.gif https://en.wikipedia.org/wiki/Grayscale#/media/File:Beyoglu_4671_tricolor.png 12
- 13. Implementation
- 14. • Take audio and create an image-like input with fixed dimensions. • Take the input signal into the frequency domain • Take a window of feature vectors • Slide window with a hop, usually smaller than the input dimension of the model How Do Implementations of Audio Solutions Differ from Vision Solutions? https://www.jonnor.com/2021/12/audio-classification-with-machine-learning-europython-2019/ © 2022 DSP Concepts 14
- 15. • Use the same layers • Convolutional layers • Conventional • Depth-wise separable • Residual blocks • RNN • Use the same training tricks How Are Implementations of Audio Solutions Like Vision Solutions? © 2022 DSP Concepts https://www.semanticscholar.org/paper/Singing-Voice-Separation-with-Deep-U-Net- Networks-Jansson-Humphrey/83ea11b45cba0fc7ee5d60f608edae9c1443861d 15
- 16. • Some popular vision model architectures show up • EfficientNet • Similar concepts • Encoder-decoder network • Transfer learning How Are Implementations of Audio Solutions Like Vision Solutions? https://ai.googleblog.com/2021/08/soundstream-end-to-end-neural-audio.html © 2022 DSP Concepts 16
- 17. Common Problems in Vision and Their Audio Analogues
- 18. • Very similar • Convolution layers pull out structural information • Trained on frequency domain features • In audio, we usually have a sliding window • Like working with a camera stream • Can be important for energy savings in complex systems • Motion detection • Voice activity detection Classification https://www.tensorflow.org/tutorials/audio/simple_audo © 2022 DSP Concepts 18
- 19. • Vision • Optical character recognition • Audio • Automatic speech recognition • Both look for structural information in their inputs and decode them to a character sequence • Similar architectures • RCNN • Transformer Sequence Decoding © 2022 DSP Concepts https://distill.pub/2017/ctc/ 19
- 20. • Vision • Directly regress the image • Audio • Regress a gain mask which is applied to audio stream • Applied in a streaming fashion • Window and hop Restoration/Denoising © 2022 DSP Concepts https://www.mathworks.com/help/audio/ug/denoise-speech-using-deep- learning-networks.html 20
- 21. • Feature engineering • After some preprocessing things are more similar than not • Implementation • Windowing with a set stride and hop allows us to deal with streams of data • Common problems in vision and their audio analogues • Classification • Sequence decoding • Restoration/denoising Conclusion © 2022 DSP Concepts 21
- 22. Resources Getting Started with Audio Audio Classification using Transfer Learning https://www.tensorflow.org/tutorials/audio/transfer _learning_audio Speech Command Recognition https://www.tensorflow.org/tutorials/audio/simple_ audio Get a 30 Day Trial of Audio Weaver https://w.dspconcepts.com/audio-weaver © 2022 DSP Concepts 22
- 23. Audio Weaver accelerates audio feature development and enables collaboration across product teams. With over 550 optimized processing modules, audio designs can be developed and implemented on hardware without writing any DSP code. The Audio Weaver Framework: Overview © 2022 DSP Concepts AWE Designer Windows-based graphical design environment Standard Edition: Design GUI Pro Edition: Works with MathWorks® MATLAB® platform Audio IP Modules Building blocks for product developers From low-level primitives to complete designs From DSP Concepts and our third-party partners AWE Core The embedded processing engine Optimized target-specific libraries Available for multiple processors Supports multicore and multi-instance implementation 23