Random Indexing
Download as PPT, PDF4 likes749 views
The document discusses the concept of random indexing as a method for representing words as vectors in a high-dimensional space to capture their meanings based on their contexts of use. It notes that random indexing maps each word to a sparse random index vector, allowing words with similar meanings to be clustered closer together in the vector space. This distributional representation of words addresses issues like synonyms and polysemy by representing the latent semantic meaning behind words. The method is also described as being computationally efficient and easily parallelizable.
Random Indexing
- 1. Random indexing: On space and meaning Simon Belak
- 2. Order of the day Meaning Philosophy Neuroscience Computer science Space Words as points in space On dimensionality Random indexing
- 3. What’s the meaning of meaning ?
- 4. Philosophers say: “ Meaning just is use.” – Wittgenstein
- 5. Neuroscientists say: Episodic memory semantic memory (concrete event abstract concept) Hebbian process
- 6. Computer scientists say: LSA semantic networks HAL TLC SAM ACT-R ontology
- 7. Projecting meaning into space
- 8. Adjacent words closely related
- 9. Movement Co-occurrences Hebbian process Self-organisation Clustering Evolution of language Coach ( Kocs carriage train car)
- 10. Problem: homonym s Table 1. a. An article of furniture supported by one or more vertical legs and having a flat horizontal surface. b. The objects laid out for a meal on this article of furniture. 2. The food and drink served at meals; fare: kept an excellent table. 3. The company of people assembled around a table, as for a meal. 4 A plateau or tableland. 5 . a. A flat facet cut across the top of a precious stone. b . A stone or gem cut in this fashion. 6 . Music a. The front part of the body of a stringed instrument. b. The sounding board of a harp. 7 . Architecture a. A raised or sunken rectangular panel on a wall. b. A raised horizontal surface or continuous band on an exterior wall; a stringcourse. 8 . A part of the human palm framed by four lines, analyzed in palmistry. 9 . An orderly arrangement of data, especially one in which the data are arranged in columns and rows in an essentially rectangular form. 1 0 . An abbreviated list, as of contents; a synopsis. 1 1 . An engraved slab or tablet bearing an inscription or a device. 1 2 . Anatomy The inner or outer flat layer of bones of the skull separated by the dipole.
- 11. Solution: high dimensionality One dimension per word Table extends into food , furniture , music ,... dimensions
- 12. Problem: synonyms amazing , stupefying , staggering , awesome , awful , awe-inspiring , awing , astonishing , astounding
- 13. Solution: latent meaning Reduced dimensionality Closely related words fold into one “ Higher-order” meaning
- 14. Random indexing
- 15. The idea Word is the sum of it’s contexts Context is the sum of it’s words Grounding?
- 16. The algorithm Take a context of words Generate a context index vector Add index to all the word vectors Go to 1) Episodic memory (2) + Hebbian process (3)
- 17. Dimensionality reduction Sparse high-dimensional ternary index (a small number of randomly distributed +1s and -1s) N early orthogonal Distances approximately preserved
- 18. The good Fast, scalable Trivially parallelised Per word Addition is associative, commutative Stable Words are independent Integer arithmetics Incremental
- 19. The bad Memory hungry Caching (Zipf’s law)
- 20. Uses Comparing words to words Query expnasion Comparing documents to documents Clustering Search Recomendations Comparing documents to words Keyword extraction
- 21. Key points Meaning is use Words in space Multiple meanings, multiple dimensions Random indexing Cognitive rationale Simple Fast, scalable
- 22. Questions?
- 23. References http://www.sics.se/~mange/papers/KarlgrenSahlgren2001.pdf http://www.kfs.org/~jonathan/witt/tlph.html http://www.mtsu.edu/~sschmidt/Cognitive/semantic/semantic.html http://memory.syr.edu/marc/papers/HowaAddiJingKaha-LSAChap-doc.pdf http://memory.psych.upenn.edu/research/research_episodic_memory.php