NLP and LSA getting started
Download as PPTX, PDF11 likes7,205 views
Latent semantic analysis (LSA) is a technique used in natural language processing to analyze relationships between documents and terms by producing concepts related to them. LSA assumes words with similar meanings will occur in similar texts, and uses a documents-terms matrix and singular value decomposition to discover hidden concepts and represent words and documents as vectors in a semantic vector space. Apache OpenNLP is a machine learning toolkit that can be used for various natural language processing tasks like part-of-speech tagging and parsing, and LSA can be seen as part of natural language processing.
NLP and LSA getting started
- 1. Latent semantic analysis (LSA) is a technique in natural language processing, in particular in vectorial semantics, of analyzing relationships between a set of documents and the terms they contain by producing a set of concepts related to the documents and terms. Wikipedia Latent semantic analysis Getting started
- 2. Natural language processing (NLP) is a field of computer science, artificial intelligence, and linguistics concerned with the interactions between computers and human (natural) languages. Wikipedia Natural language processing could be divided in 4 phases: Grammar analysis Lexical analysis Semantic analysis Syntactic analysis Apache OpenNLP Machine learning based toolkit for the processing of natural language text. http://opennlp.apache.org/ LSA LSA could be seen as a part of NLP
- 3. Apache OpenNLP usage examples: Lexical analysis Grammar analysis Syntactic analysis Part-of-speech tagging Tokenization Chunker - Parser NOTE: Before the lexical analysis is possible to use a sentences analysis tool: sentence detector (Apache OpenNLP).
- 4. Supervised machine learning concepts INPUT DATA (ex: wikipedia corpus) Humans produce a finite set of couples (INPUT,OUTPUT). It represents the training set. It can be seen as discrete function. Machine learning algorithm (ex:linear regretion, maximum entropy, perceptron) MODEL OUTPUT DATA (ex:corpus POSTagged) Machine produces a model. It can be seen as a continuous function. INPUT DATA (ex: just a document) OUTPUT DATA (that document POSTagged) Input data are taken from an infinte set. Machine, using model and input, produces the expected output.
- 5. LSA assumes that words that are close in meaning will occur in similar pieces of text. LSA is a method for discovering hidden concepts in document data. LSA key concepts Doc 2 Doc 3 Doc 4 Doc 1 Set of documents, each document contains several words. LSA algorithm takes docs and words and evaluates vectors in a semantic vectorial space using: • A documents/words matrix • Singular value decomposition (SVD) word1word2 doc1 doc2 doc3 doc4 Semantic vectorial space. Word1 and word2 are close, it means that their (latent) meaning is related.
- 6. Example: Doc 2 Doc 3 Doc 4 Doc 1 Doc1 Doc2 Doc3 Doc4 Word1 1 0 1 0 Word2 1 0 1 1 Word3 0 1 0 1 … Words/document matrix 1: there are occurrences of the i-word in the j-doc. 0: there are not occurrences of the i-word in the j-doc. The matrix dimension is very big (thousands of words, hundreds of documents). Matrix SVD decomposition To reduce the matrix dimension Semantic Vector or JLSI libraries: • SVD decomposition. • Build the vectorial semantic space. word1word2 doc1 doc2 doc4 UIMA to manage the solution
- 7. Online references: http://opennlp.apache.org/documentation/manual/opennlp.html https://code.google.com/p/semanticvectors/ http://hlt.fbk.eu/en/technology/jlsi http://uima.apache.org/ http://en.wikipedia.org/wiki/Singular_value_decomposition http://en.wikipedia.org/wiki/Eigenvalues_and_eigenvectors Coursera video references: http://www.coursera.org/course/nlangp http://www.coursera.org/course/ml
- 8. Some snipptes and console commands OpenNLP has a command line tool which is used to train the models. Trained Model
- 9. Models and document to manage This snippet takes as inputs 4 files and it evaluates a new file sentence detected, tokenized and POSTtaggered. Sentences tokens tags Document that is sentence detected, tokenized and POSTaggered, and that could be, for example, indexed in a search engine like Apache Solr.
- 10. Note that the lucene-core is a hierarchical dependency. .bat file to load the classpath SemanticVectors has two main functions: 1. Building wordSpace models. To build the wordSpace model Semantic Vector needs indexes created by Apache Lucene. 2. Searching through the vectors in such models. Es: Bible chapter Indexed by Lucene
- 11. 1. Building wordSpace models using pitt.search.semanticvectors.LSA class from the index created by Apache Lucene (from a bible chapter). In this example the Bible chapter contains 29 documents, and in total there are 2460 terms. Semantic Vector builds: 1. 29 vectors that represent the documents (docvector.bin) 2. 2460 vectors that represent the terms (termvector.bin) This two files represent the wordSpace. Note that could be also possible to use pitt.search.semanticvectors.BuildIndex class that use Random Projection instead of LSA to reduce the dimensional representation.
- 12. 2. Searching through docVector and termVector 2.1 Searching for Documents using Terms Search for document vectors closest to the vector ”Abraham”:
- 13. 2.2 Using a document file as a source of queries Find terms most closely related to Chapter 1 of Chronicles:
- 14. 2.3 Search a general word Find terms most closely related to “Abraham”.
- 15. 2.4 Comparing words Compare “abraham” with “Isaac”. Compare “abraham” with “massimo”.