Locations in the Neocortex: A Theory of Sensorimotor Prediction Using Cortical Grid Cells
Download as PPTX, PDF1 like954 views
This document summarizes a talk on how cortical grid cells may represent the structure of objects. [1] Cortical grid cells in the entorhinal cortex and within cortical columns can represent locations within environments and on objects. As an animal or sensor moves, grid cell modules update locations using path integration. [2] A network model shows how grid cell representations of location can be used to build predictive models of objects by associating locations with sensory features. Simulations demonstrate the network can recognize objects through sequences of movements and sensations. [3] There is suggestive anatomical and physiological evidence mapping the proposed model to biology, including grid cell signatures in human cortex and sensorimotor prediction in primary sensory areas.
Locations in the Neocortex: A Theory of Sensorimotor Prediction Using Cortical Grid Cells
- 1. Workshop on Integrative Theories of Cortical Function July 18, 2018 Subutai Ahmad sahmad@numenta.com @SubutaiAhmad Locations in the Neocortex: A Theory of Sensorimotor Prediction Using Cortical Grid Cells Collaborators: -Jeff Hawkins, Marcus Lewis, Scott Purdy, Mirko Klukas
- 2. Observation: The neocortex is constantly predicting its inputs. “the most important and also the most neglected problem of cerebral physiology” (Lashley, 1951) How can networks of neurons learn predictive models of the world and objects through movement? Question: The neocortex uses “cortical grid cells” and path integration to model an object’s structure. Proposal:
- 3. 1) Properties of grid cells 3) Mapping to biology - Suggestive biological evidence - Cortical columns revisited - Model predictions - Encoding location using grid cells - Path integration and location spaces - Sensory cues and recalling locations - Learns predictive models of objects using locations and sensory cues - Simulation results 2) A network model of sensorimotor prediction using cortical grid cells
- 4. Single Grid Cell (Moser & Moser, 2013)
- 5. Grid Cell Module Contains Multiple Cells That Are Offset
- 6. Grid Cell Module Contains Multiple Cells That Are Offset
- 7. Multiple Modules At Different Scales and Orientations Environment
- 8. Multiple Modules Can Represent Locations Uniquely (Fiete et al, 2008; Sreenivasan and Fiete, 2011)
- 9. Path Integration (Hafting et al, 2005; McNaughton et al., 2006; Ocko et al., 2018) - As animal moves, grid cells update their location - This can happen in the dark, using efference copy of motion signals - Path integration: regardless of path trajectory, the same location in environment will activate a consistent grid code - Imprecise, so sensory cues are used to “anchor” grid cells
- 10. Unique Location Spaces For Each Environment - Each module randomly initialized in a new room, codes for each room will be unique (e.g. 20 modules, 100 cells each = 10020 possible codes) - Initial point implicitly defines a location space for each environment - Modules update independently, so path integration still works - Sensory cues can be used to anchor or re-activate the location space (Rowland & Moser, 2014)
- 11. Summary: Grid Cells Represent the Structure of Environments Entorhinal Cortex Body in environments Location - Encoded by grid cells - Unique to location in room AND room - Location is updated by movement A Room is: - A set of locations that are connected by movement (via path integration). - Some locations have associated features.
- 12. Proposal: Cortical Grid Cells Represent the Structure of Objects Location - Encoded by grid cells - Unique to location in room AND room - Location is updated by movement A Room is: - A set of locations that are connected by movement (via path integration). - Some locations have associated features. Entorhinal Cortex Body in environments Cortical Column Sensor patch relative to objects Location - Encoded by grid-like cells - Unique to location on object AND object - Location is updated by movement An Object is: - A set of locations that are connected by movement (via path integration). - Some locations have associated features.
- 13. 1) Properties of grid cells 3) Mapping to biology - Suggestive biological evidence - Cortical columns revisited - Model predictions - Encoding location using grid cells - Path integration and location spaces - Sensory cues and recalling locations - Learns predictive models of objects using locations and sensory cues - Simulation results 2) A network model of sensorimotor prediction using cortical grid cells
- 14. Network Model Grid cell modules use movement to update their location Sensory predictions (forward model)Predictions + sensation => sensory representation - Sequence of discrete time steps - Each time step consists of 4 stages representing a movement followed by a sensation New sensation updates location
- 15. Updating Grid Cell Module Based On Motion - Given a motion 𝒅 𝑡 , phase is shifted according to: - Threshold activity to get a binary location representation - Cells in each module represented by 2D phase Φ - Activity at time t is a bump centered around a phase Φ 𝑡 𝑖 - Each module has a different scale and orientation, represented by a transform matrix: 𝑴𝑖 = 𝑠𝑖 cos 𝜃𝑖 − 𝑠𝑖 sin 𝜃𝑖 𝑠𝑖 sin 𝜃𝑖 𝑠𝑖 cos 𝜃𝑖 −1 Φ 𝑡,move 𝑖 = 𝜑 + 𝑴𝑖 𝒅 𝑡 mod 1.0 𝜑 ∈ Φ 𝑡−1 𝑖
- 16. Location Layer Forms Predictions in Sensory Layer Sensory predictions (forward model)
- 17. Sensory Layer is a Sequence Memory Layer - Neurons in a mini-column learn same FF receptive field. - Distal dendritic segments form connections to cells in location layer. - Active segments act as predictions and bias cells. - With sensory input these cells fire first, and inhibit other cells within mini-column. (Hawkins & Ahmad, 2016; Hawkins et al, 2017) No prediction t=0 t=1 Sensory input Predicted input t=0 Predicted cells inhibit neighbors t=1 Sensory input Sensory layer Very specific sparse representation that encodes the current sensory input at the current location. Dense representation that activates the codes for this sensory input at any location.
- 18. Multiple Simultaneous Locations Represents Uncertainty - Sensory representation activates grid cell locations - If this sensory representation is not unique, we activate a union of grid cells in each module - With sufficiently large modules, the union can represent several locations without confusion - Movement shifts all active grid cells (Ahmad & Hawkins, 2016)
- 19. Learning - For a new object, we first activate a random cell in each module and move to first feature - Select random subset of active sensory cells for this sensory input. Store location representation on independent dendritic segment in sensory cells. - Store this sensory representation on an independent dendritic segment in each active location cell - Move to next feature and repeat - This process invokes a new unique location space and sequentially stores specific location representation with sensory cues, and specific sensory code with locations.
- 20. An Example Recognition Walkthrough Sense f1
- 21. An Example Recognition Walkthrough Move to f2
- 22. An Example Recognition Walkthrough Sense f2
- 23. An Example Recognition Walkthrough Move to f1
- 25. Network Convergence Improves With More Modules 100 objects with 10 points each, pool of 100 unique features
- 26. Network Convergence vs Ideal Observer 100 objects, 10 unique features, 20 modules
- 27. Capacity of Single Cortical Column 100 unique features, 20 modules
- 28. Two Layer Network Model For Sensorimotor Inference - Structure of objects are defined by the relative locations of sensory features - Grid cell code enables a powerful predictive sensorimotor network - Objects are recognized through a sequence of movements and sensations - Simulation results demonstrate convergence and capacity
- 29. Sensorimotor Inference With Multiple Columns And Long- Range Lateral Connections (Hawkins et al, 2017)
- 30. Sensorimotor Inference With Multiple Columns And Long- Range Lateral Connections - Each column has partial knowledge of object. - Columns use lateral context and vote through long range lateral connections. - Inference is much faster with multiple columns. (Hawkins et al, 2017)
- 31. 1) Properties of grid cells 3) Mapping to biology - Suggestive biological evidence - Cortical columns revisited - Model predictions - Encoding location using grid cells - Path integration and location spaces - Sensory cues and recalling locations - Learns predictive models of objects using locations and sensory cues - Simulation results 2) A network model of sensorimotor prediction using cortical grid cells
- 32. L2/3 L4 L6 L5 Input Anatomy of a Cortical Column 1) Cortical columns are really complex! The function of a cortical column must also be complex. 2) Large reciprocal projections from L6a to L4 3) Motor signals drive activity in L6 Simple Output L2 L3a L3b L4 L6a L6b L6 ip L6 mp L6 bp L5 tt L5 cc L5 cc-ns L5: Calloway et. al, 2015 L6: Zhang and Deschenes, 1997 Binzegger et al., 2004 Nelson et al., 2013 Output, via thalamus 50%10% Cortex Thalamus Output, direct L5 CTC: Guillery, 1995 Constantinople and Bruno, 2013 Long range lateral connections
- 33. 1) Border ownership cells: Cells fire only if feature is present at object-centric location on object. Detected even in primary sensory areas (V1 and V2). (Zhou et al., 2000; Willford & von der Heydt, 2015) 2) Grid cell signatures in cortex: Cortical areas in humans show grid cell like signatures (fMRI and single cell recordings) Seen while subjects navigate conceptual object spaces and virtual environments. (Doeller et al., 2010; Jacobs et al. 2013; Constantinescu et al., 2016; ) 3) Sensorimotor prediction in sensory regions: Cells predict their activity before a saccade. Predictions during saccades are important for invariant object recognition. (Duhamel et al., 1992; Nakamura and Colby, 2002; Li and DiCarlo, 2008) 4) Hippocampal functionality may have been conserved in neocortex: Six-layer neocortex evolved by stacking 3-layer hippocampus and piriform cortex (Jarvis et al., 2005; Luzatti, 2015) Other Biological Evidence 33
- 34. Experimentally Testable Predictions 34 1) Object coding: Every sensory region will contain layers that are stable while sensing a familiar object. The set of cells will be sparse but specific to object identity. Ambiguous information will lead to denser activity in upper layers. Each region will contain cells tuned to locations of features in the object’s reference frame. (Zhou et al., 2000; Zheng & Kwon, 2018) 2) Cortical columns: Cortical cols can learn complete object models Complexity of objects tied to span of long-range lateral connections Activity within stable layers will converge slower with long–range connections disabled Subgranular layers of primary sensory regions (Layer 6) will be driven by motor signals Grid-like cells in Layer 6a (Nelson et al., 2013; Sutter et Shepherd, 2015; Lee et al, 2008; Leinweber et al, 2017)
- 35. Summary: Cortical Grid Cells Represent the Structure of Objects Entorhinal Cortex Body in environments Cortical Column Sensor patch relative to objects 1. Grid cell code provides a neural mechanism for representing locations 2. Cortical grid cells can build predictive sensorimotor models of objects 3. Suggestive anatomical and physiological mapping to biology
- 36. Numenta Team Jeff Hawkins Marcus Lewis Contact: sahmad@numenta.com Scott Purdy Mirko Klukas Luiz Scheinkman Max Schwarzer