Building a Pipeline for State-of-the-Art Natural Language Processing Using Hugging Face Tools
3 likes1,239 views
The document summarizes a presentation about state-of-the-art natural language processing (NLP) techniques. It discusses how transformer networks have achieved state-of-the-art results in many NLP tasks using transfer learning from large pre-trained models. It also describes how Hugging Face's Transformers library and Tokenizers library provide tools for tokenization and using pre-trained transformer models through a simple interface.
![Language Modeling
Objectives - MLM
['The', 'pipeline', 'for', 'State', '-', 'of', '-', 'the', '-', 'Art', 'Natural', 'Language', 'Process', '##ing']
The pipeline for State-of-the-Art Natural Language Processing
['The', ‘pipeline’ 'for', 'State', '-', 'of', '-', 'the', '-', 'Art', [MASK], 'Language', 'Process', '##ing']
Tokenization
Masking
['The', ‘pipeline’, 'for', 'State', '-', 'of', '-', 'the', '-', 'Art', [MASK], 'Language', 'Process', '##ing']
‘Natural’
‘Artificial’
‘Machine’
‘Processing’
‘Speech’
Prediction](https://image.slidesharecdn.com/661lysandredebutanthonymoi-200629032103/85/Building-a-Pipeline-for-State-of-the-Art-Natural-Language-Processing-Using-Hugging-Face-Tools-22-320.jpg)
![Language Modeling
Objectives - CLM
['The', 'pipeline', 'for', 'State', '-', 'of', '-', 'the', '-', 'Art', 'Natural', 'Language', 'Process', '##ing']
The pipeline for State-of-the-Art Natural Language Processing
Tokenization
Prediction
['Process', '##ing', '(', 'NL', '##P', ')', 'software', 'which', 'will', 'allow', 'a', 'user', 'to', 'develop']](https://image.slidesharecdn.com/661lysandredebutanthonymoi-200629032103/85/Building-a-Pipeline-for-State-of-the-Art-Natural-Language-Processing-Using-Hugging-Face-Tools-23-320.jpg)
![The tokenization pipeline
Inner workings
Normalization Pre-tokenization Tokenization Post-processing
▪ Add special tokens: for example [CLS], [SEP] with BERT
▪ Truncate to match the maximum length of the model
▪ Pad all sequence in a batch to the same length
▪ ...](https://image.slidesharecdn.com/661lysandredebutanthonymoi-200629032103/85/Building-a-Pipeline-for-State-of-the-Art-Natural-Language-Processing-Using-Hugging-Face-Tools-38-320.jpg)
![Transformers
Tokenization to prediction
transformers.PreTrainedTokenizer
transformers.PreTrainedModel
The pipeline for State-of-the-Art Natural Language Processing
[[464, 11523, 329, 1812, 12, ..., 15417, 28403]]
Tensor(batch_size, sequence_length, hidden_size) Task-specific prediction
Base model With task-specific head](https://image.slidesharecdn.com/661lysandredebutanthonymoi-200629032103/85/Building-a-Pipeline-for-State-of-the-Art-Natural-Language-Processing-Using-Hugging-Face-Tools-49-320.jpg)
Building a Pipeline for State-of-the-Art Natural Language Processing Using Hugging Face Tools
- 2. The pipeline for State-of-the-Art NLP Hugging Face
- 3. Agenda Lysandre DEBUT Machine Learning Engineer @ Hugging Face, maintainer and core contributor of huggingface/transformers Anthony MOI Technical Lead @ Hugging Face, maintainer and core contributor of huggingface/tokenizers Some slides were adapted from previous HuggingFace talk by Thomas Wolf, Victor Sanh and Morgan Funtowicz
- 4. Feedback Your feedback is important to us. Don’t forget to rate and review the sessions.
- 5. Hugging Face
- 6. Hugging Face Most popular open source NLP library ▪ 1,000+ Research paper mentions ▪ Used in production by 1000+ companies
- 7. Hugging Face
- 8. Today’s Menu
- 9. Subjects we’ll dive in today ● NLP: Transfer learning, transformer networks ● Tokenizers: from text to tokens ● Transformers: from tokens to predictions
- 10. Transfer Learning - Transformer networks One big training to rule them all
- 11. NLP took a turn in 2018 Self-supervised Training & Transfer Learning Large Text Datasets Compute Power The arrival of the transformer architecture
- 12. Transfer learning In a few diagrams
- 13. Sequential transfer learning Learn on one task/dataset, transfer to another task/dataset word2vec GloVe skip-thought InferSent ELMo ULMFiT GPT BERT DistilBERT Text classification Word labeling Question-Answering .... Pre-training Adaptation Computationally intensive step General purpose model
- 14. Transformer Networks Very large models - State of the Art in several tasks
- 15. Transformer Networks ● Very large networks ● Can be trained on very big datasets ● Better than previous architectures at maintaining long-term dependencies ● Require a lot of compute to be trained Source: BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova.In NACCL, 2019.
- 16. Transformer Networks Pre-training Base model Pre-trained language model Very large corpus $$$ in compute Days of training
- 17. Transformer Networks Fine-tuning Pre-trained language model Fine-tuned language model Training can be done on single GPU Small dataset Easily reproducible
- 18. Model Sharing Reduced compute, cost, energy footprint From 🏎 Smaller, faster, cheaper, lighter: Introducing DistilBERT, a distilled version of BERT, by Victor Sanh
- 19. A deeper look at the inner mechanisms Pipeline, pre-training, fine-tuning
- 20. Adaptation Head Pre-trained model Tokenizer Transfer Learning pipeline in NLP From text to tokens, from tokens to prediction Jim Henson was a puppet ##eer 11067 5567 245 120 7756 9908 1.2 2.7 0.6 -0.2 3.7 9.1 -2.1 3.1 1.5 -4.7 2.4 6.7 6.1 2.4 7.3 -0.6 -3.1 2.5 1.9 -0.1 0.7 2.1 4.2 -3.1 True 0.7886 False 0.223 Jim Henson was a puppeteer Tokenization Convert to vocabulary indices Pre-trained model Task-specificmodel
- 21. Pre-training Many currently successful pre-training approaches are based on language modeling: learning to predict Pϴ (text) or Pϴ (text | other text) Advantages: - Doesn’t require human annotation - self-supervised - Many languages have enough text to learn high capacity models - Versatile - can be used to learn both sentence and word representations with a variety of objective functions The rise of language modeling pre-training
- 22. Language Modeling Objectives - MLM ['The', 'pipeline', 'for', 'State', '-', 'of', '-', 'the', '-', 'Art', 'Natural', 'Language', 'Process', '##ing'] The pipeline for State-of-the-Art Natural Language Processing ['The', ‘pipeline’ 'for', 'State', '-', 'of', '-', 'the', '-', 'Art', [MASK], 'Language', 'Process', '##ing'] Tokenization Masking ['The', ‘pipeline’, 'for', 'State', '-', 'of', '-', 'the', '-', 'Art', [MASK], 'Language', 'Process', '##ing'] ‘Natural’ ‘Artificial’ ‘Machine’ ‘Processing’ ‘Speech’ Prediction
- 23. Language Modeling Objectives - CLM ['The', 'pipeline', 'for', 'State', '-', 'of', '-', 'the', '-', 'Art', 'Natural', 'Language', 'Process', '##ing'] The pipeline for State-of-the-Art Natural Language Processing Tokenization Prediction ['Process', '##ing', '(', 'NL', '##P', ')', 'software', 'which', 'will', 'allow', 'a', 'user', 'to', 'develop']
- 24. Tokenization It doesn’t have to be slow
- 25. Tokenization - Convert input strings to a set of numbers Its role in the pipeline Jim Henson was a puppet ##eer 11067 5567 245 120 7756 9908 Jim Henson was a puppeteer - Goal: Find the most meaningful and smallest possible representation
- 26. Some examples Let’s dive in the nitty-gritty
- 27. Word-based Word by word tokenization Let’s do tokenization! Let ‘s do tokenization ! Split on punctuation: Split on spaces: ▪ Split on spaces, or following specific rules to obtain words ▪ What to do with punctuation? ▪ Requires large vocabularies: dog != dogs, run != running ▪ Out-of-vocabulary (aka <UNK>) tokens for unknown words
- 28. Character Character by character tokenization ▪ Split on characters individually ▪ Do we include spaces or not? ▪ Smaller vocabularies ▪ But lack of meaning -> Characters don’t necessarily have a meaning separately ▪ End up with a huge amount of tokens to be processed by the model L e t ‘ s d o t o k e n i z a t i o n !
- 29. Byte Pair Encoding Welcome subword tokenization ▪ First introduced by Philip Gage in 1994, as a compression algorithm ▪ Applied to NLP by Rico Sennrich et al. in “Neural Machine Translation of Rare Words with Subwords Units”. ACL 2016.
- 30. Byte Pair Encoding Welcome subword tokenization A B C ... a b c ... ? ! ... Initial alphabet: ▪ Start with a base vocabulary using Unicode characters seen in the data ▪ Most frequent pairs get merged to a new token: 1. T + h => Th 2. Th + e => The
- 31. Byte Pair Encoding Welcome subword tokenization ▪ Less out-of-vocabulary tokens ▪ Smaller vocabularies Let’s</w> do</w> token ization</w> !</w>
- 32. And a lot more So many algorithms... ▪ Byte-level BPE as used in GPT-2 (Alec Radford et al. OpenAI) ▪ WordPiece as used in BERT (Jacob Devlin et al. Google) ▪ SentencePiece (Unigram model) (Taku Kudo et al. Google)
- 33. Tokenizers Why did we build it? ▪ Performance ▪ One API for all the different tokenizers ▪ Easy to share and reproduce your work ▪ Easy to use any tokenizer, and re-train it on a new language/dataset
- 34. The tokenization pipeline Inner workings Normalization Pre-tokenization Tokenization Post-processing
- 35. The tokenization pipeline Inner workings Normalization Pre-tokenization Tokenization Post-processing ▪ Strip ▪ Lowercase ▪ Removing diacritics ▪ Deduplication ▪ Unicode normalization (NFD, NFC, NFKC, NFKD)
- 36. The tokenization pipeline Inner workings Normalization Pre-tokenization Tokenization Post-processing ▪ Set of rules to split: - Whitespace use - Punctuation use - Something else?
- 37. The tokenization pipeline Inner workings Normalization Pre-tokenization Tokenization Post-processing ▪ Actual tokenization algorithm: - BPE - Unigram - Word level
- 38. The tokenization pipeline Inner workings Normalization Pre-tokenization Tokenization Post-processing ▪ Add special tokens: for example [CLS], [SEP] with BERT ▪ Truncate to match the maximum length of the model ▪ Pad all sequence in a batch to the same length ▪ ...
- 39. Tokenizers Let’s see some code!
- 40. Tokenizers Let’s see some code!
- 41. Tokenizers Let’s see some code!
- 42. Tokenizers Let’s see some code!
- 43. Tokenizers Let’s see some code!
- 44. Tokenizers Let’s see some code!
- 45. Tokenizers How to install it?
- 46. Transformers Using complex models shouldn’t be complicated
- 47. Transformers An explosion of Transformer architectures ▪ Wordpiece tokenization ▪ MLM & NSP BERT ALBERT GPT-2 ▪ SentencePiece tokenization ▪ MLM & SOP ▪ Repeating layers ▪ Byte-level BPE tokenization ▪ CLM Same API
- 48. Transformers As flexible as possible Runs and trains on: ▪ CPU ▪ GPU ▪ TPU With optimizations: ▪ XLA ▪ TorchScript ▪ Half-precision ▪ Others All models BERT & RoBERTa More to come!
- 49. Transformers Tokenization to prediction transformers.PreTrainedTokenizer transformers.PreTrainedModel The pipeline for State-of-the-Art Natural Language Processing [[464, 11523, 329, 1812, 12, ..., 15417, 28403]] Tensor(batch_size, sequence_length, hidden_size) Task-specific prediction Base model With task-specific head
- 50. Transformers Available pre-trained models transformers.PreTrainedTokenizer transformers.PreTrainedModel ▪ We publicly host pre-trained tokenizer vocabularies and model weights ▪ 1611 model/tokenizer pairs at the time of writing
- 51. Transformers Pipelines transformers.Pipeline ▪ Pipelines handle both the tokenization and prediction ▪ Reasonable defaults ▪ SOTA models ▪ Customizable
- 52. A few use-cases That’s where it gets interesting
- 53. Transformers Sentiment analysis/Sequence classification (pipeline)
- 55. Transformers Causal language modeling/Text generation
- 56. Transformers Sequence Classification - Under the hood
- 57. Transformers Sequence Classification - Under the hood
- 58. Transformers Sequence Classification - Under the hood
- 59. Transformers Sequence Classification - Under the hood
- 60. Transformers Sequence Classification - Under the hood
- 61. Transformers Sequence Classification - Under the hood
- 62. Transformers Training models Example scripts (TensorFlow & PyTorch) - Named Entity Recognition - Sequence Classification - Question Answering - Language modeling (fine-tuning & from scratch) - Multiple Choice Trains on TPU, CPU, GPU Example scripts for PyTorch Lightning
- 63. Transformers Just grazed the surface The transformers library covers a lot more ground: - ELECTRA - Reformer - Longformer - Encoder-decoder architectures - Translation & Summarization
- 64. Transformers + Tokenizers The full pipeline? Data Tokenization Prediction 🤗 nlp Tokenizers Transformers Metrics 🤗 nlp
- 65. Feedback Your feedback is important to us. Don’t forget to rate and review the sessions.