Neural word embedding and language modelling
- 1. Neural Word Embedding And Language Modelling A Survey Paper Presentation by Riddhi Jain Graduate Student Computer Engineering, SJSU (014600716)
- 2. Paper: A survey on Neural Word Embeddings Authors: ERHAN SEZERER SELMA TEKIR Article Link: https://arxiv.org/pdf/2110.01804.pdf Published Date: 2021 About
- 3. What is VSM? The vector space model is an algebraic model that represents objects (like text) as vectors. This makes it easy to determine the similarity between words or the relevance between a search query and document. Cosine similarity is often used to determine similarity between vectors.
- 4. Neural Word Embeddings Neural network architecture is constructed to predict the next word given the set of neighboring words in the sequence in neural language modeling.
- 5. Word Embeddings with Improved Language Models ● Early Word Embeddings ● Embeddings Target Specific Semantic Relations ● Sense Embeddings ● Morpheme Embeddings
- 6. Early Word Embeddings Word2vec is the first neural word embedding model that efficiently computes representations to leverage the context of target words Two word2vec variants: ● CBOW (Continuous Bag of words) ○ Example - “nature is pleased with simplicity” ● Skip-gram
- 7. Embeddings Target Specific Semantic Relations Example - “She took a sip of hot coffee” and “He is taking a sip of cold water” Types of algorithms - SGNS - GloVe - ATTRACT and REPEL
- 8. Sense Embeddings ● Early word embeddings unite all the senses of a word into one representation. ● In reality, a word gets meaning in its use and can mean different things in varying contexts. ● When the issue becomes labeling those sense groups, the task becomes a supervised one
- 9. Morpheme Embeddings Proposes several ways to target morphological information in order to obtain sub-word information for solving the rare/unknown word problem of earlier word embedding methods and also to have better representations of words for morphologically rich languages. Two ways - Training Morphological Embeddings from Scratch - Adjusting the Existing Embeddings
- 10. Datasets ● Similarity Tasks ● Analogy Task Google Analogy Task with 8869 semantic and 10675 syntactic ● Synonym Selection Tasks ● Downstream Tasks GLUE benchmark dataset
- 11. Human-level language understanding is one of the oldest challenges in computer science. Pre-trained language models’ knowledge has been transferred to fine- tuned task-specific models. Multi-modal language models are based on human language acquisition, where learning starts with concrete concepts through images early on. Conclusion
Editor's Notes
- #2: Hello
- #4: Before neural representation learning, representations of words or documents have been computed using the vector space model. In VSM [120], frequencies of words in documents are considered to form a term-document matrix. Although these count-based representations are proved helpful in addressing semantics, they are the bag of words approaches and are not able to capture both syntactical and semantic features at the same time, which is required for performing well in NLP tasks.
- #5: Neural network architecture is constructed to predict the next word given the set of neighboring words in the sequence in neural language modeling. In the iterative processing of this prediction over a large corpus, the learned weights in the hidden layers serve as neural embeddings for words.
- #6: Although the initial word embedding models successfully identified semantic and syntactic similarities of words, they still need to be improved to address specific semantic relations among words. consider the sentences "She took a sip of hot coffee" and "He is taking a sip of cold water." The antonyms "cold" and "hot" are deemed to be similar since their context is similar. Therefore, it becomes an issue to differentiate the synonyms "warm" and "hot" from the antonyms "cold" and "hot" considering they have similar contexts in most occurrences. Early word embeddings unite all the senses of a word into one representation. In reality, a word gets meaning in its use and can mean different things in varying context. Sense Embeddings are used to get word sense discrimination as the decomposition of a word’s occurrences into same sense groups
- #7: Word2vec is the first neural word embedding model that efficiently computes representations to leverage the context of target words. It can be considered as the initiator of early word embeddings. Word2vec has two variants: Continuous bag of words model (CBOW) and Skip-gram model. In CBOW - middle word is predicted given the context, set of neighbouring left and right. When the input sentence "nature is pleased with simplicity" is processed, the system predicts the middle word "pleased" given the left and right context In Skip-gram system predicts the most probable context words for a given input word. In terms of a language model, while CBOW predicts an individual word’s probability, Skip-gram outputs the probabilities of a set of words, defined by a given context size.
- #8: Although the initial word embedding models successfully identified semantic and syntactic similarities of words, they still need to be improved to address specific semantic relations among words. consider the sentences "She took a sip of hot coffee" and "He is taking a sip of cold water." The antonyms "cold" and "hot" are deemed to be similar since their context is similar. Therefore, it becomes an issue to differentiate the synonyms "warm" and "hot" from the antonyms "cold" and "hot" considering they have similar contexts in most occurrences.
- #9: Early word embeddings unite all the senses of a word into one representation. In reality, a word gets meaning in its use and can mean different things in varying context. Sense Embeddings are used to get word sense discrimination as the decomposition of a word’s occurrences into same sense groups
- #10: The quest for morphological representations is a result of two important limitations of earlier word embedding models. The first point is, words are not the smallest units of meaning in language. Even if a model does not see the word unpleasant in the training it should be able to deduce that it is the negative form of pleasant. Word embedding methods that don’t take morphological information into account can not produce any results in such a situation. The second limitation is the data scarcity problem of morphologically rich languages and agglutinative languages. Unlike English, morphologically rich languages have many more noun and/or verb forms inflected by gender, case, or number, which may not exist in the training corpora. The same thing is also valid for agglutinative languages in which words can have many forms according to the suffix(es) they take. Therefore, models that take morphemes/lexemes into account needed.