Teaching Machines to Listen: An Introduction to Automatic Speech Recognition
0 likes44 views
This document provides an overview of automatic speech recognition systems and their components. It discusses: - The main components of ASR systems including preprocessing/feature extraction, acoustic models, and language models. - Unique challenges of working with speech data like data volume, quality, and annotation. - Common modeling approaches used in ASR systems have historically included hidden Markov models and n-gram language models, but more recent approaches use end-to-end neural networks like deep speech, wav2vec, conformer and whisper models.
Teaching Machines to Listen: An Introduction to Automatic Speech Recognition
- 1. An Introduction to Automatic Speech Recognition Teaching Machines to Listen
- 2. ● Introduction ● Considerations for Working with Speech Data ● Overview of Automatic Speech Recognition (ASR) Systems ● Modeling Approaches Outline
- 3. Introduction
- 4. • Why should you listen to me tell you about ASR? whoami
- 5. What we will cover • Main components of speech recognition systems • Unique challenges working with speech data • Some common modeling approaches Expectations
- 6. What we will cover • Main components of speech recognition systems • Unique challenges working with speech data • Some common modeling approaches Expectations What we will not cover • Deep dive into specific model architectures • Comprehensive coverage of all aspects of working with speech • Code
- 7. Consider this an orientation to the field of speech recognition for those familiar with other types of Data Science! What we will cover • Main components of speech recognition systems • Unique challenges working with speech data • Some common modeling approaches Expectations What we will not cover • Deep dive into specific model architectures • Comprehensive coverage of all aspects of working with speech • Code
- 8. • Speech recognition is the task of recognizing speech within audio and converting it into text • Active field since 1950’s! • Very active research field with substantial recent advances driven largely by novel neural network architectures Introduction
- 9. • 1950s and 1960s: Focus on limited use cases; digits, phonemes, single speakers • 1980s and 1990s: Shift to focus on statistical approaches (HMMs, etc.) • 2000s and 2010s: Wider availability of speech recognition toolkits • Late 2010s: end-to-end modeling approaches (Selected) History
- 10. • Data volume Specific Challenges for Speech Recognition
- 11. • Data Quality & Characteristics Specific Challenges for Speech Recognition
- 12. • Annotation Specific Challenges for Speech Recognition
- 13. Considerations for Working with Speech Data
- 14. • Machine learning algorithms want to deal with vectors (or tensors) of floating point numbers • Different data types have different preprocessing requirements, as well as different vectorization strategies Preparing Data for Machine Learning
- 15. • Tabular data is often a mix of different field types, likely captured from various systems of reference in a wide variety of ways • “Messiness” often comes from interpreting business context for individual fields in a given dataset Common Characteristics of Tabular Data
- 16. • Preprocessing often includes categorical/numerical standardization and identifying and addressing relationships between columns Preprocessing and Vectorizing Tabular Data
- 17. • Text carries it own set of unique preprocessing challenges and data characteristics • Characteristics • Language • Text Encoding • Typos and word-level errors Common Characteristics of Text Data
- 18. • Preprocessing • Case and punctuation normalization • Frequency analysis • Usually map to fixed lexicon • Vectorization • Tokenization: Word / subword / character • Vectorization: Count-based vs. embedding Preprocessing and Vectorizing Text Data
- 19. • Technical Characteristics • Size and resolution • Color space • Format / compression Common Characteristics of Image Data
- 20. • Preprocessing • Type Conversion • Resampling • Re-scaling (centering) • Cropping • Vectorization • Preprocessing produces matrices and tensors of machine-readable data! Preprocessing and Vectorizing Image Data
- 21. • Technical Characteristics • Sample rate • Bit rate • Compression / Encoding • Audio channels • Content Characteristics • Number / demographics of speakers • Acoustic Environment • Accent • Continuous vs. discrete speech • Language(s) • Dialect and vocabulary Common Characteristics of Audio Data
- 22. Overview of Automatic Speech Recognition (ASR) Systems
- 23. • Input is raw audio waveform Shape of Automatic Speech Recognition Task • Output is discrete text tokens automatic speech recognition is the coolest area of machine learning
- 24. • Input is raw audio waveform Shape of Automatic Speech Recognition Task • Output is discrete text tokens automatic speech recognition is the coolest area of machine learning Framed as a sequence to sequence machine learning task
- 25. 10,000 ft. View of ASR System Components automatic speech recognition is the coolest area of machine learning Preprocessing/ Feature Extraction Acoustic Model Language Model
- 26. Preprocessing and Feature Extraction automatic speech recognition is the coolest area of machine learning Preprocessing/ Feature Extraction Acoustic Model Language Model
- 27. • Resampling • Normalization • Splitting / Chunking • Noise detection / cleaning • Language identification / cleaning Preprocessing Audio Data
- 28. • Resampling • Normalization • Splitting / Chunking • Noise detection / cleaning • Language identification / cleaning Preprocessing Audio Data
- 29. • Resampling • Normalization • Splitting / Chunking • Noise detection / cleaning • Language identification / cleaning Preprocessing Audio Data
- 30. • Resampling • Normalization • Splitting / Chunking • Noise detection / parsing • Language identification / cleaning Preprocessing Audio Data
- 31. • Resampling • Normalization • Splitting / Chunking • Noise detection / parsing • Language identification / cleaning Preprocessing Audio Data
- 32. Preprocessing Audio Text Data Automatic speach recognition is the coolest area of making learning!!! automatic speech recognition is the coolest area of machine learning • Furthermore, the target text data (transcripts) need to be preprocessed, normalized, etc • Many speech recognition map to a fixed output vocabulary • Using this vocabulary, we can define the output lexicon for our downstream modeling tasks
- 33. • Much of the useful information for downstream tasks isn’t directly accessible from the raw audio signal • Most modern frameworks leverage features derived from the frequency representation of the audio signal Feature Extraction and Vectorization
- 34. • The human ear is more sensitive to particular frequency ranges than others • That domain knowledge is leveraged for computing sets of features that capture features most relevant for speech processing Feature Extraction and Vectorization
- 35. • A common approach for extracting relevant features incorporating this domain knowledge is the computation of Mel Frequency Cepstral Coefficients, or MFCCs Feature Extraction and Vectorization
- 36. • The result is now a series of discrete feature vectors which can then be fed as input to the next step in the process Feature Extraction and Vectorization
- 37. Acoustic Model automatic speech recognition is the coolest area of machine learning Preprocessing/ Feature Extraction Acoustic Model Language Model
- 38. • An acoustic model provides a mapping from the feature space to an intermediate acoustic representation of discrete phonemes Acoustic Model
- 39. Acoustic Model • Given a discrete input of feature vectors, the acoustic model provides probabilities that a given (sequence of) features corresponds to a particular output phoneme
- 40. Acoustic Model • As we’re ultimately interested in word output and not sequences of phonemes, an acoustic model • A lexicon is leveraged by several methods to directly map sequences of phonemes to a word token hypothesis
- 41. Language Model automatic speech recognition is the coolest area of machine learning Preprocessing/ Feature Extraction Acoustic Model Language Model
- 42. • Output tokens from an acoustic model are largely unconstrained in the sense of logical ordering • A language model attempts to introduce probabilistic constraints on the raw output of an acoustic model to provide more likely sequences of words Language Model
- 44. • Historically, Hidden Markov Models (HMM) have found great success in use as acoustic models for ASR systems Acoustic Models as Hidden Markov Models
- 45. • N-gram language models model the probability of the next token in a sequence by computing statistical transition probabilities between n-grams for a large representative corpus • Even complex, modern approaches still rely on n-gram language models to constrain the output text to a given domain! N-Gram Language Models
- 46. • Time-delay neural networks seek to directly incorporate the surrounding temporal context features into the classification of each frame into a corresponding phoneme Time-delay Neural Network
- 47. • Much research over the past decade has focused on approaching speech recognition as a single task • Many approaches borrow ideas or architectural design from the speech and signal processing domain to maximize the useful information signal through the model End to end approaches
- 48. • Deep speech (2014) was a popular architecture that directly modeled recursive relationships in the input signal End to end approaches
- 49. • Wav2vec(2) (2019/2020) leverages vector product quantization to encode discrete speech representations • This strategy, along with large scale unsupervised model pretraining, allowed for markedly performant models when fine-tuned on relatively small data sets End to end approaches
- 50. • Conformer (2020) leverages both convolutional layers and attention-based transformer layers to model both localized and longer term dependencies in input speech data End to end approaches
- 51. • More recently, the Whisper (2022) model from Open AI (of ChatGPT fame) made a splash for providing a very robust model trained on over 680,000 hours of audio from captioned video • This model uses an encoder-decoder architecture accommodated by a multitask training framework End to end approaches
- 52. 10,000 ft. View of ASR System Components automatic speech recognition is the coolest area of machine learning The newest awesome end-to-end model