Self-Attentive Associative Memory
- 3. Background 3
- 5. Current associative memory is item memory, weak at recognizing relationships Item Memory • Store and retrieve individual items • Relate pair of items of the same time step • Fail to relate temporally distant items 5 Source: Zeithamova et al., "The hippocampus and inferential reasoning: building memories to navigate future decisions. " Frontiers in human neuroscience 6 (2012).
- 6. Current attention is dot product NTM DNC Transformer RMCALSTM 6 Represent a relationship as a scalar Source: Santoro et al., "Relational recurrent neural networks.“ In Advances in neural information processing systems. 2018.
- 7. Limitations of current memory systems • The relational representation is often computed without storing • Few works that manage both items and the relationships in a single memory • The memory-memory relationship is coarse 7
- 9. Neuropathologists say there exist 2 memory systems in human brain 9 • Store items • Simple, low-order Relational Memory • Store relationships between items • Complicated, high-order Item Memory Howard Eichenbaum, Memory, amnesia, and the hippocampal system (MIT press, 1993). Alex Konkel and Neal J Cohen, "Relational memory and the hippocampus: representations and methods", Frontiers in neuroscience 3 (2009).
- 15. Self-attentive associative memory (SAM) 15 • Build upon OPA • Differentiable • Convert 2d matrix to 3d tensor • Convert item memory to relational memory Extract items Associate items
- 29. Limitations and future directions • OPA is powerful but complicated • The current SAM uses only 1 layer of OPA. Is deep SAM possible? • Can we do relational reasoning with visual associated features? E.g., reasoning over graph with each node associated with an image. 29 5 km 2 km 1 km Human: Shortest path from to is -> ->
- 30. Meet the authors Hung Le Associate Research Fellow Applied Artificial Intelligence Institute Truyen Tran Associate Professor Applied Artificial Intelligence Institute Svetha Venkatesh Co-Director, Applied Artificial Intelligence Institute (A2I2) Alfred Deakin Professor, Australian Laureate Fellow 30
- 31. Thank you thai.le@deakin.edu.au A²I² Deakin University Geelong Waurn Ponds Campus, Geelong, VIC 3220 Hung Le 31