A Light Introduction to Transfer Learning for NLP
Download as PPTX, PDF1 like174 views
The document provides an introduction to transfer learning in natural language processing (NLP), explaining its relevance and significance, especially through examples like ULMFiT, ELMo, and Glomo. It discusses the evolution and impact of transfer learning methodologies, particularly the transformer architecture from Google, which has become foundational in modern NLP. Additionally, it highlights the differences in training and test distributions as well as the variety of text types and characteristics that affect NLP tasks.
A Light Introduction to Transfer Learning for NLP
- 1. A LIGHT INTRODUCTION TO TRANSFER LEARNING FOR NLP
- 2. NATURAL LANGUAGE PROCESSING • NLP INVOLVES MACHINE OR ROBOTS TO UNDERSTAND • IT PROCESS THE LANGUAGE THAT HUMAN SPEAK.
- 3. TRANSFER LEARNING • TRANSFER LEARNING IS THE PROCESS OF TRAINING A MODEL ON A LARGE- SCALE DATASET AND THEN USING THAT PRETRAINED MODEL TO CONDUCT LEARNING FOR ANOTHER DOWNSTREAM TASK (I.E., TARGET TASK) • TRANSFER LEARNING WAS POPULARIZED IN THE FIELD OF COMPUTER VISION THANKS TO THE IMAGENET DATASET
- 4. TRANSFER LEARNING FOR NLP • TRAINING AND TEST DISTRIBUTIONS ARE DIFFERENT • DIFFERENT TEXT TYPES. • DIFFERENT ACCENTS/AGES. • DIFFERENT TOPICS/CATEGORIES.
- 5. PRETRAINED NLP MODELS • ULMFIT • ELMO • GLOMO • OPENAI TRANSFORMER
- 6. ULMFIT • ULMFIT WAS PROPOSED AND DESIGNED BY FAST.AI’S JEREMY HOWARD AND DEEPMIND’S SEBASTIAN RUDER. YOU COULD SAY THAT ULMFIT WAS THE RELEASE THAT GOT THE TRANSFER LEARNING PARTY STARTED LAST YEAR. • UMFIT STANDS FOR UNIVERSAL LANGUAGE MODEL FINE TUNING FOR TEXT CLASSIFICATION
- 7. ELMO • ELMO WORD REPRESENTATIONS, OR EMBEDDINGS FROM LANGUAGE MODELS • PRETRAINING THE ENTIRE MODEL WITH DEEP CONTEXTUALIZED REPRESENTATIONS THROUGH STACKED NEURAL LAYERS • ELMO IS A NOVEL WAY OF REPRESENTING WORDS IN VECTORS AND EMBEDDINGS. THESE ELMO WORD EMBEDDINGS HELP US ACHIEVE STATE-OF-THE-ART RESULTS ON MULTIPLE NLP TASKS
- 8. GLOMO • UNSUPERVISED LEARNED RELATIONAL GRAPHS AS TRANSFERABLE REPRESENT0ATIONS • MODERN DEEP TRANSFER LEARNING APPROACHES HAVE MAINLY FOCUSED ON LEARNING GENERIC FEATURE VECTORS FROM ONE TASK THAT ARE TRANSFERABLE TO OTHER TASKS, SUCH AS WORD EMBEDDINGS IN LANGUAGE AND PRETRAINED CONVOLUTIONAL FEATURES IN VISION. • THESE APPROACHES USUALLY TRANSFER UNARY FEATURES AND LARGELY IGNORE MORE STRUCTURED GRAPHICAL REPRESENTATIONS. • THIS WORK EXPLORES THE POSSIBILITY OF LEARNING GENERIC LATENT RELATIONAL GRAPHS THAT CAPTURE DEPENDENCIES BETWEEN PAIRS OF DATA UNITS (E.G., WORDS OR PIXELS)
- 9. OPEN AI TRANSFORMER • IMPROVING LANGUAGE UNDERSTANDING WITH UNSUPERVISED LEARNING • THE TRANSFORMER ARCHITECTURE IS AT THE CORE OF ALMOST ALL THE RECENT MAJOR DEVELOPMENTS IN NLP. IT WAS INTRODUCED IN 2017 BY GOOGLE. BACK THEN, RECURRENT NEURAL NETWORKS (RNN) WERE BEING USED FOR LANGUAGE TASKS, LIKE MACHINE TRANSLATION AND QUESTION ANSWERING SYSTEMS. • GOOGLE RELEASED AN IMPROVED VERSION OF TRANSFORMER LAST YEAR CALLED UNIVERSAL TRANSFORMER.