Markov chain
- 1. Markov Chain and Its Use in Economic Modelling Markov process Transition matrix Convergence Likelihood function Expected values and Policy Decision Ecocomic Modelling: PG 1
- 2. A stochastic process { } t x has the Markov process if for all k ³ 2 and all t ( t t t t k ) ( t t ) Pr ob x / x , x ,...., x prob x / x +1 -1 - +1 = A Markov process is characterised by three elements: 1) an N dimensional vector of all possible values of the state of the system. 2) a t transition matrix P, that shows possibility of moving from one state. to another 3) the probability of being in each state i at time 0. Ecocomic Modelling: PG 2
- 3. A typical Transition matrix p p p p . 01 02 03 0 p p p p . 11 12 13 1 p p p p . 21 22 23 2 . . . . . p p p p Ecocomic Modelling: PG 3 ù ú ú ú ú ú ú û é ê ê ê ê ê ê ë = N N N N N N N N i j P 1 2 3 . , . 1 , = åN j i j p 1 0, = åN i i p
- 4. Chapman-Kolmogorov Equations i ( i ) = ob x = x 0, 0 p Pr ( ) i j t j t i = ob x = x x = x , +1 p Pr n ( ) ( ) 2 t j t h t h t i P P P x x x x ob x x x x ob = = = = = = å å= p ' = =p Ecocomic Modelling: PG 4 , , , 1 2 1 1 Pr Pr / i j h i h h j n h + + + ( ) k t k j t i i j ob x x x x P, Pr = = = + ( ) P x prob '0 1 '1 p = =p ( ) 2 '0 2 '2 p = prob x =p P ( ) k k k P x prob '0
- 5. Likelihood Function for a Markov Chain ( ) L ob x x x x x º - - Pr , , ,.... , i T i T i T i i , , 1 , 2 ,1 ,0 P P P P p - - - - - = 1 2 1 3 2 0 1 ... i i i i i i i i i T , T T , T T , T , 0, ni j L =p PP P , 0, i i j i , j q Two uses of likelihood function to study alternative histories of a Markov Chain to estimate the parameter q Ecocomic Modelling: PG 5
- 6. Convergence of Markov Process with Finite States ù 3 4 1 4 ù 5 8 3 8 2 é 9 16 7 16 3 1 2 1 2 é lim n n 17 32 15 32 4 úû ù Reference: Stokey and Lucas (page 321) é Ecocomic Modelling: PG 6 úû é êë P = 1 4 3 4 úû êë P = 3 8 5 8 ù úû êë P = 7 16 9 16 úû é êë P = 15 32 17 32 ù êë P = ®¥ 1 2 1 2 A Markov Process Converges when each element of the of the transition matrix approaches to a limit like this. { } t x Process is stationary in this example.
- 7. Recurrent or absorbing State or Transient State in a Markov Chain S1 is the recurrent state whenever the process ù 1 g g 2 g 2 0 1 2 1 2 Ecocomic Modelling: PG 7 { } t x leaves, re-enters in it and stays there forever. It is transient when it does not return to S1 when it leaves it. ú ú ú û é - ê ê ê ë P = 0 1 2 1 2 Here S1 is the recurrent state whenever the process leaves, re-enters in it. S2 and S3 are transient.
- 8. Converging and Non-converging Sequences ù 1 2 2 0 1 2 1 2 é 0 1 2 1 2 0 1 2 1 2 g Î(0,1) n é é P ù ù 3 4 1 4 3 4 1 4 0 0 é 1 4 3 4 0 0 0 0 3 4 1/ 4 ù 1/ 2 1 2 0 0 1 2 1 2 0 0 0 0 1/ 2 1/ 2 é 0 0 1/ 2 1/ 2 0 0 1/ 2 1/ 2 1/ 2 1/ 2 0 0 Ecocomic Modelling: PG 8 ( ) ú ú ú û é - ê ê ê ë P = ®¥ 0 1 2 1 2 lim n n n n n g d d ( )n n d = 1- 1-g ù ú ú ú û ê ê ê ë P = ®¥ 0 1 2 1 2 lim n ù úû é P êë P P = 0 0 2 ù 1 úû êë P P úû ù P P = êë P úû êë é P P PP = 1 2 1 2 2 1 2 1 0 0 0 0 0 0 úû é êë P = P = 1 4 3 4 1 2 ù ú ú ú ú û ê ê ê ê ë P = 0 0 1/ 4 3/ 4 ú ú ú ú û é ê ê ê ê ë P = 0 0 1/ 2 1/ 2 2n ù ú ú ú ú û ê ê ê ê ë P + = 1/ 2 1/ 2 0 0 2n 1 Even Odd
- 9. One Example of Markov Chain Stochastic life cycle optimisation model (preliminary version of Bhattarai and Perroni) w t Z d w t Z - + + + + + ì 1 1, 1, 1 1, t z t z U = + 1 + p High income Low income 2 Ecocomic Modelling: PG 9 t s t t t t t t t t z d t Z t Z d t z t Z t Z d w w U w w U C s s s r r + + - + + + + + + + + ü ïþ ïý ïî ïí ïþ ïý ü ïî ïí ì + + + 1 1 1 1 1, 1, 1, 1, 1, 1 , , 1 t Z t Z t z t z E r W C V , , , , + × = + t Z d t z t z W W V 1, t , , = + + + T Z T z W V , , = - Probability of recurrent state p Prob of Transient state (1-p ) If transient 2 p Probability of being in Ambiguous state m
- 10. Impact of Risk Aversion and Ambiguity in Expected Wealth with Markov Process Expected nonhuman wealth with increasing risk aversion (1-3) SC1 SC2 SC3 SC4 SC5 T2 0.872 0.914 0.956 0.996 1.032 T3 1.580 1.654 1.727 1.796 1.862 T4 1.960 2.053 2.144 2.230 2.312 T5 1.659 1.740 1.819 1.894 1.965 Expected nonhuman wealth with increasing ambiguity (0.2-0.8) SC1 SC2 SC3 SC4 SC5 T2 0.872 0.906 0.938 0.968 0.995 T3 1.580 1.646 1.709 1.768 1.825 T4 1.960 2.050 2.135 2.216 2.293 T5 1.659 1.742 1.820 1.895 1.967 Ecocomic Modelling: PG 10
- 11. Markov Decision problem (refer Ross (187)). Let there be a sequence of action , ,…., corresponding to states and the reward for this be given be . Policy makers problem with the Markov process is: Subject to 1. for all i and a. 2. 3. Optimal policy is Ecocomic Modelling: PG 11
- 12. Use of Markov Chain in analysis of Duopoly Sargent and Ljungqvist (133) ( )2 , , , 1 , 0.5 i t t i t i t i t R = p y - d y - y + ( ) t t t p A A y y0 1 1, 1 2 = - + + ( )2 R = A y - A y - A y y - 0.5 d y - y i , t 0 i , t 1 i t 1 i , t + 1 j , t i , t + 1 i , t ( , ) max { ( , )} , , , , 1 , 1 2, i i t j t y i t i i t j t v y y R v y y + + = + + , 1 Ecocomic Modelling: PG 12 i t b ( ) j t j i t j t y f y y , 1 , , = , + Markov perfect equilibrium is the pair of value functions and a pair of policy functions for i=1,2 that satisfies the above Bellman equation. Equilibrium is computed by backward induction and he optimising behaviours of firms by iterating forward for all conceivable future states.
- 13. Other Application of Markov Process • Regime -Switch analysis in economic time series (Hamilton pp. 677-699; Harvey (285)) • Industry investment under uncertainty (SL chap 10) • Stochastic dynamic programming (SL chapter 8,9) • Weak and strong convergence analysis (SLChap 11-13) • Arrow Securities (Ljungqvist and Sargent Chapter 7). • Life cycle consumption and saving: An example • Precautionary saving Ecocomic Modelling: PG 13
- 14. References: Dreze Jacques (2003) Advances in Macroeconomic Theory, Palgrave. Hamilton JD. (1994) Time Series Analysis Princeton University Press. Harvey A. C. (1993) Time Series Models Harvester Wheatsheaf. Ljungqvist L and T.J. Sargent (2000), Recursive Macroeconomic theory, MIT Press Ross Sheldon (1993) Probability Models, Academic Press. Sargent TJ (1987) Macroeconomic Theory, Harvard University Press. Sargent TJ (1987) Dynamic Macroeconomic Theory, Chapter 1, Harvard University Press. Stokey, N. L. and R.E. Lucas (1989) Recursive Methods in Economic Dynamics, Harvard UP, Cambridge, MA. Wang Wang Peijie (2003) Financial Econometrics, Routledge Advanced Texts. Bianchi and Zoega (1998) Unemployment Persistence: Does the Size of the Shock Matter, Journal of Applied Econometrics, 13:283-304 (1998). Ecocomic Modelling: PG 14
- 15. Markov Chain Example in GAMS *retire1.gms $title model with Knightian uncertainty scalar pi transition probability /0.33/ mu cond probability of ambiguous state /0/ beta pure rate of time preference /0.02/ r interest rate /0.05/ rho relative risk aversion /4.0/ eh high earnings /2.0/ el low earnings /0.5/; Ecocomic Modelling: PG 15 option iterlim = 1000000000; option reslim = 1000000000; set t /t1*t5/ z /s1*s16/; alias(t,tt); alias(z,zz); * card(z) = 2**(card(t)-1) beta = (1+beta)**(50/card(t))-1; r = (1+r)**(50/card(t))-1; parameter act(t,z) a tree generator d(t) remaining states l(z) odd number generator nlst(t) non-last period prob(t,z) probability of occurence weight(t,z) weight with ambiguity e(t,z) earnings trans(t,z) transition index sex(t) discount factor ; act(t,z) = round(ord(z) - trunc(ord(z)/(card(z)/(2**(ord(t)-1))))*card(z)/(2**(ord(t)-1))); act(t,z) = 1$((act(t,z) eq 1) or (ord(t) eq card (t))); d(t) = round(2**(card(t)-ord(t))) ; l(z) = round(ord(z)- trunc(ord(z)/2)*2); nlst(t) = 1$(ord(t) ne card(t)); sex(t) = sum(tt$(ord(tt) gt ord(t)), 1/(1+beta)**(ord(tt)-ord(t)));
- 16. e("t1","s1") = eh; loop((t,z)$(act(t,z) and nlst(t)), e(t+1,z) = eh; e(t+1,z+d(t+1)) = el; ); trans(t,z) = 0; trans("t1","s1") = 0; loop((t,z)$(act(t,z) and nlst(t)), trans(t+1,z) = 1$(e(t+1,z) ne e(t,z)); trans(t+1,z+d(t+1)) = 1$(e(t+1,z+d(t+1)) ne e(t,z)); ); prob("t1","s1") = 1; loop((t,z)$(act(t,z) and nlst(t)), prob(t+1,z) = prob(t,z)*( (1-pi+pi/2)$(trans(t+1,z) eq 0) +(pi/2) $(trans(t+1,z) eq 1) ); prob(t+1,z+d(t+1)) = prob(t,z)*( (1-pi+pi/2)$(trans(t+1,z+d(t+1)) eq 0) +(pi/2) $(trans(t+1,z+d(t+1)) eq 1) ); ); weight("t1","s1") = 1; loop((t,z)$(act(t,z) and nlst(t)), weight(t+1,z) = weight(t,z)*( ( (1-pi+(1-mu)*pi/2) $(trans(t+1,z) eq 0) +(1-(1-pi+(1-mu)*pi/2))$(trans(t+1,z) eq 1) )$(e(t,z) eq eh) +( (1-(1-mu)*pi/2) $(trans(t+1,z) eq 0) +( (1-mu)*pi/2) $(trans(t+1,z) eq 1) )$(e(t,z) eq el) Ecocomic Modelling: PG 16 ); weight(t+1,z+d(t+1)) = weight(t,z)*( ( (1-pi+(1-mu)*pi/2) $(trans(t+1,z+d(t+1)) eq 0) +(1-(1-pi+(1-mu)*pi/2))$(trans(t+1,z+d(t+1)) eq 1) )$(e(t,z) eq eh) +( (1-(1-mu)*pi/2) $(trans(t+1,z+d(t+1)) eq 0) +( (1-mu)*pi/2) $(trans(t+1,z+d(t+1)) eq 1) )$(e(t,z) eq el) ); ); parameter checkp(t), checkw(t); Markov Chain Example in GAMS
- 17. Markov Chain Example in GAMS: Model Equations defu(t,z)$act(t,z).. u(t,z) =e= (( c(t,z)**(1/(1+sex(t))) * ( (( weight(t+1,z) /(weight(t+1,z)+weight(t+1,z+d(t+1))) *u(t+1,z) ** (1-rho) +weight(t+1,z+d(t+1))/(weight(t+1,z)+weight(t+1,z+d(t+1))) *u(t+1,z+d(t+1)) ** (1-rho) ) ** (1/(1-rho)) )$(rho ne 1) + ( u(t+1,z) **(weight(t+1,z) /(weight(t+1,z)+weight(t+1,z+d(t+1)))) * u(t+1,z+d(t+1))**(weight(t+1,z+d(t+1))/(weight(t+1,z)+weight(t+1,z+d(t+1)))) )$(rho eq 1) ) **(sex(t)/(1+sex(t))) )$nlst(t) +c(t,z)$(nlst(t) eq 0) )$(insure eq 0) + (( c(t,z)**(1/(1+sex(t))) * ( weight(t+1,z) /(weight(t+1,z)+weight(t+1,z+d(t+1))) *u(t+1,z) +weight(t+1,z+d(t+1))/(weight(t+1,z)+weight(t+1,z+d(t+1))) *u(t+1,z+d(t+1)) )**(sex(t)/(1+sex(t))) )$nlst(t) Ecocomic Modelling: PG 17 +c(t,z)$(nlst(t) eq 0) )$(insure eq 1); defc(t,z)$act(t,z).. c(t,z) + v(t,z) =e= e(t,z)+ w(t,z)*r; defwl(t,z)$(act(t,z) and nlst(t)).. w(t+1,z) =e= w(t,z) + v(t,z); defwn(t,z)$(act(t,z) and nlst(t)).. w(t+1,z+d(t+1)) =e= w(t,z) + v(t,z); defwt(t,z)$(act(t,z) and (nlst(t) eq 0)).. w(t,z) =e= -v(t,z); dobj.. obj =e= u("t1","s1"); model lc /all/; variables u(t,z) c(t,z) v(t,z) w(t,z) obj; equations defu(t,z) defc(t,z) defwl(t,z) defwn(t,z) defwt(t,z) dobj;
- 18. loop(cases, mu = 0; weight("t1","s1") = 1; loop((t,z)$(act(t,z) and nlst(t)), weight(t+1,z) = weight(t,z)*( ( (1-pi+(1-mu)*pi/2) $(trans(t+1,z) eq 0) +(1-(1-pi+(1-mu)*pi/2))$(trans(t+1,z) eq 1) )$(e(t,z) eq eh) +( (1-(1-mu)*pi/2) $(trans(t+1,z) eq 0) +( (1-mu)*pi/2) $(trans(t+1,z) eq 1) )$(e(t,z) eq el) ); weight(t+1,z+d(t+1)) = weight(t,z)*( ( (1-pi+(1-mu)*pi/2) $(trans(t+1,z+d(t+1)) eq 0) +(1-(1-pi+(1-mu)*pi/2))$(trans(t+1,z+d(t+1)) eq 1) )$(e(t,z) eq eh) +( (1-(1-mu)*pi/2) $(trans(t+1,z+d(t+1)) eq 0) +( (1-mu)*pi/2) $(trans(t+1,z+d(t+1)) eq 1) )$(e(t,z) eq el) ); Ecocomic Modelling: PG 18 ); Calculation of Weight Among Various States