Computational Motor Control: State Space Models for Motor Adaptation (JAIST summer course)

Computational Motor
Control Summer School
03: State space models
for motor adaptation.
Hirokazu Tanaka
School of Information Science
Japan Institute of Science and Technology

State-space modeling of motor adaptation.
In this lecture, we will learn:
• Motor adaptation paradigms
• Continuous-time state-space models
• Discrete-time state-space models
• Controllability
• Observability
• State-space description for motor adaptation
• Multi-rate models
• Motor memory of errors
• Mirror reversal (non-error based learning)

Motor adaptation paradigms to dynamical perturbations: Force-field adaptation.
Shadmehr & Mussa-Ivaldi (1994) J Neurosci
Baseline (no field) Initial exposures
adaptation catch trials

Motor adaptation paradigms to kinematical perturbations: Visuomotor rotation.
Krakauer et al. (2000) J Neurosc; Krakauer (2009) Progress in Motor Control

Adaptation to prism displacements.
Martin et al. (1996) Brain ;Kitazawa et al. (1995) J Neurosci

Adaptation to prism displacements.
Kitazawa et al. (1995) J Neurosci
1 1n n ne e ke  1
1
1
n
in
i
e e k e


  

Continuous-time state-space models.
F ma mx 
x v
Fv a
m


 
x
v
 
  
 
x
Newton’s equation of dynamics
0 1 0
0 0 1/
x x
F
v v m
      
          
      
x Ax Bu
 x Ax Bu
State-space representation
A x B u
State-space vector

Discrete-time state-space models.
Discrete-time representation
 k k t x x
 
 
 
1 ( 1)k
k k
k k k
k k
k t
t
t
t t
   
  
   
    
x x
x x
x Ax Bu
I A x B u
1
ˆ ˆ
k k k  x Ax Bu  
 
2
2ˆ
ˆ t
t
t t
te t
e
t


  
  
 


 
A
A
A I A
B BB
t
Δt 2Δt 3Δt (k-1)Δt kΔt (k+1)Δt0
k-1 k k+10 1 2 3
time (continuous)
time steps
(discrete)

Deterministic and stochastic state-space models.
1k k k
k k
  


x Ax Bu
z Cx
1
k
kk k
kk
k   

 
w
v
x Ax Bu
z Cx
Deterministic Stochastic
XkXk-1 Xk+1
zk-1 zk zk+1
uk-1 uk uk+1

Linear time-variant and time-invariant state-space models.
1k k k
k k
k k
k
  


x x u
xCz
A B
Time-variant model
1k k k
k k
  


x x u
xCz
A B
Time-invariant model
Throughout these lectures, we will use linear time-invariant (LTI) models
for mathematical simplicity.

State-space models in an explicit component form.
1k k k  x Ax Bu
k kz Cx
1, 1
2, 1
, 1
k
k
N k
x
x
x



1,
2,
,
k
k
N k
x
x
x
11 12 1
21 21
11 1
N
N
a a a
a a
a a
11
21
1
1
L
N NL
b b
b
b b
1
L
u
u
= +
1,
2,
,
k
k
N k
x
x
x
1
M
z
z
11 12 1
1 112
N
MM
c c c
c c c
=
Process equation
Measurement equation
N vector N×N matrix N vector N×L matrix L vector
M vector M×N matrix N vector

Controllability: the ability of driving a system into desired final state.
1k k k  x Ax Bu
, , ,N L N N
k k
N L 
   x u A B
Controllability is the ability of external inputs {uk} to drive a state from any
initial condition to any final condition in a finite time. A state-space model
is controllable if the N×NL controllability matrix has full row rank:
2 1n
   B AB A B A B
Sketch of proof:
0
1 1
2
2 2 1
1
1
0
2
N N N
N N
NN
N
N
N
 
  


 
  
 
 
       
 
 
x Ax Bu
A x ABu Bu
u
u
A x B AB A B
u
Kalman (1963) SIAM J Contr

Observability: determining hidden state from measurements.
k kz Cx
, ,N M N
k
M
k

  x z C
Observability is the ability to determine a (latent) state from a sequence
of measurements {zk}. A state space model is called observable if the
MN×N observability has full rank N:
Kalman (1963) SIAM J Contr
1N 
 
 
 
 



 
C
CA
C A

State-space models for dynamic (force-field) motor adaptation.
Thoroughman & Shadmehr (2000) Nature; Donchin et al. (2003) J Neurosci
1n n n
n n n
  
 
x Ax Bu
z Cx Du

State-space models for kinematic (visual rotation) motor adaptation.
Tanaka et al. (2006) J Neurophysiol
T
1k k k k
k k k
z
z
   

x Ax BH
H x

Trial-by-trial generalization width reflects directional tuning width.
   
 
 
 
1
1i i N
i
N
g
g
g

 

 
 
   
  
r r r r Rg
 T
k k k  rR g
         1 1 1
T
k k k k k k k k        R R g R g gr g
Suppose that, for target direction θ, the motor output is a weighted sum of
population activity {gi(θ)} multiplied with preferred directions {ri}:
A gradient descent learning rule specifies the change of preferred directions
according to the movement error Δrk and the population activity {g(θk)} :
This change affects the motor output at the next trial as:

Two-rate model of motor adaptation: fast and slow learners.
Smith et al. (2006) PLoS Biol
1n n nu  x Ax B
 
 
 
 
 
 
 
 
f ff f
1
s ss s
1
0
0
n n
n
n n
x xa b
u
x xa b


      
       
      
      
n nz  Cx
 
 
 
   
f
f s1
1 1s
1
1 1 n
n n n
n
x
z x x
x

 

 
   
 
 
       f s s f
,a a b b 
There are two learners in the brain; the fast learner (x(f)) learns
quickly but forgets quickly, while slow learner (x(s)) learns slowly
but maintains its memory longer.
Motor output is a sum of the fast
and slow learners.
State vector consists of fast
(x(f)) and slow learners (x(s)).

The model explains savings, spontaneous recovery.
Savings Spontaneous recovery

The prediction of spontaneous recovery is confirmed in humans.

The slow process contributes to motor memory consolidation.
Joiner & Smith (2008) J Neurophysiol
The slow process, but not the fast process, contributes to motor memory consolidation.

Explicit (strategic) and implicit (error-based) learning.
Mazzoni & Krakauer (2006) J Neurosci
Strategy (aiming the adjacent target) cancels the “error” without
any adaptation!

Explicit (strategic) and implicit (error-based) learning.
Mazzoni & Krakauer (2006) J Neurosci

What is “motor error?”: Aiming error and target error.
Taylor & Ivry (2011) PLoS Comp Biol; Taylor & Ivry (2014) Prog Brain Res

State-space model for strategic and error-based learning.
yn: target direction
rn: rotation angle
xn: adaptation variable
sn: strategy variable
yn
sn
sn-rn+xn
 aiming
n n n nn n ne s s r x r x    
     target
n n n n n n nn ne y s r x y sx r     
aiming
netarget
ne

State-space model for strategic and error-based learning.
yn
sn
sn-rn+xn
 aiming
n n n nn n ne s s r x r x    
     target
n n n n n n nn ne y s r x y sx r     
aiming
netarget
ne
aiming
targ
1
e
1
t
n n n
nn n
x ax be
s cs de


 
 
a=0.99, b=0.015,
c=0.999, d=0.022

Steepest descent learning rule for optimization.
Lecture 6, in Neural Networks for Machine Learning, Geoff Hinton
E
( 1) ( )n n E
 
 

w w
w
optimum
E
w
E
w
Descent learning rule:
RPROP: Adjustment of learning rate.
E
w
gradient
learning rate
     
1  1 

Motor memory of experienced errors.
Herzfeld et al. (2014) Science
( ) ( ) ( )
( 1) ( ) ( ) ( )
ˆ
ˆ ˆ
n n n
n n n n
e y y
x ax e
  

 
( )
( )
( )
( )
( )
: perturbation
ˆ : estimated perturbation ("belief")
: sensory consequence
ˆ : predicted sensory consequence
: control signal
n
n
n
n
n
x
x
y
y
u
State-space model: memory of environments
Population-coding model: memory of errors
 ( ) ( )n n
i i
i
w g e    
 
2
2
exp
2
i
i
e e
g e

 
  
 
 
 
 
   
( 1)
( 1) ( 1) ( 1) ( )
T ( 1) ( 1)
sgn
n
n n n n
n n
e
e e
e e


  
 
 
g
w w
g g error
activity
w1 w2 w3 wn
η

Motor memory of experienced errors.
Herzfeld et al. (2014) Science

Displacement and left-right reversal: Why so different?
Martin et al. (1996) Brain; Sekiyama et al. (2000) Nature
Displacement prism
… takes only few dozen trials. … takes a few weeks.
Left-right reversed prism
Day 3
Day 34

Mirror reversal: a distinct form of motor adaptation?
Taglen et al. (2014) J Neurosci; Lilicrap et al. (2013) Exp Brain Res
Movement number Movement number
Absoluteerror
Absoluteerror
Visual rotation Mirror reversal

Summary
• A state-space model consists of a process equation
(temporal transition) and an observation equation
(measurement).
• Humans are flexible to a novel environment, known as
motor adaptation, such as perturbations of force fields
and visual transformation.
• State-space modeling has been very successful in
describing trial-by-trial adaptation processes in humans.

References
• Thoroughman, K. A., & Shadmehr, R. (2000). Learning of action through adaptive combination of motor primitives. Nature,
407(6805), 742-747.
• Donchin, O., Francis, J. T., & Shadmehr, R. (2003). Quantifying generalization from trial-by-trial behavior of adaptive systems
that learn with basis functions: theory and experiments in human motor control. The Journal of Neuroscience, 23(27),
9032-9045.
• Tanaka, H., Sejnowski, T. J., & Krakauer, J. W. (2009). Adaptation to visuomotor rotation through interaction between
posterior parietal and motor cortical areas. Journal of Neurophysiology, 102(5), 2921-2932.
• Smith, M. A., Ghazizadeh, A., & Shadmehr, R. (2006). Interacting adaptive processes with different timescales underlie
short-term motor learning. PLoS Biol, 4(6), e179.
• Joiner, W. M., & Smith, M. A. (2008). Long-term retention explained by a model of short-term learning in the adaptive
control of reaching. Journal of Neurophysiology, 100(5), 2948-2955.
• Inoue, M., Uchimura, M., Karibe, A., O'Shea, J., Rossetti, Y., & Kitazawa, S. (2015). Three timescales in prism adaptation.
Journal of Neurophysiology, 113(1), 328-338.
• Mazzoni, P., & Krakauer, J. W. (2006). An implicit plan overrides an explicit strategy during visuomotor adaptation. The
Journal of Neuroscience, 26(14), 3642-3645.
• Taylor, J. A., & Ivry, R. B. (2011). Flexible cognitive strategies during motor learning. PLoS Comput Biol, 7(3), e1001096-
e1001096.
• Taylor, J. A., & Ivry, R. B. (2014). Cerebellar and prefrontal cortex contributions to adaptation, strategies, and reinforcement
learning. Progress in Brain Research, 210, 217.
• Taylor, J. A., Krakauer, J. W., & Ivry, R. B. (2014). Explicit and implicit contributions to learning in a sensorimotor adaptation
task. The Journal of Neuroscience, 34(8), 3023-3032.
• Telgen, S., Parvin, D., & Diedrichsen, J. (2014). Telgen, S., Parvin, D., & Diedrichsen, J. (2014). Mirror reversal and visual
rotation are learned and consolidated via separate mechanisms: Recalibrating or learning de novo?. The Journal of
Neuroscience, 34(41), 13768-13779.?. The Journal of Neuroscience, 34(41), 13768-13779.
• Lillicrap, T. P., Moreno-Briseño, P., Diaz, R., Tweed, D. B., Troje, N. F., & Fernandez-Ruiz, J. (2013). Adapting to inversion of
the visual field: a new twist on an old problem. Experimental Brain Research, 228(3), 327-339.
• Ostry, D. J., Darainy, M., Mattar, A. A., Wong, J., & Gribble, P. L. (2010). Somatosensory plasticity and motor learning. The
Journal of Neuroscience, 30(15), 5384-5393.

Exercise
• Simulate the state-space model proposed by Taylor and
Ivry.
• Mirror reversal is different from most adaptation
paradigms in that learning from error worsens the
performance. Can we consider a state-space model for
mirror reversal?

Computational Motor Control: State Space Models for Motor Adaptation (JAIST summer course)

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to Computational Motor Control: State Space Models for Motor Adaptation (JAIST summer course) (20)

More from hirokazutanaka (9)

Recently uploaded (20)

Computational Motor Control: State Space Models for Motor Adaptation (JAIST summer course)