SlideShare a Scribd company logo

12-markov-chains and dynamical systems.pdf

M
Mohammad732983

This document covers linear dynamical systems and Markov chains, emphasizing their properties and the connection between them through stochastic matrices. It discusses the analysis of state vectors, steady-state vectors, and the evolution function of dynamical systems, illustrating the concepts using examples such as population dynamics. Key topics include the existence and uniqueness of steady-state vectors, convergence of states under matrix transformations, and the use of matrices to model changes in population distributions over time.

Science
1 of 126
Download to read offline
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
CAS CS 132 Markov Chains Geometric Algorithms Lecture 12
Objectives 1. Motivate linear dynamical systems 2. Analyze Markov chains and their properties 3. Learn to solve for steady-states of Markov chains 4. Connect this to graphs and random walks
Keywords linear dynamical systems recurrence relations linear difference equations state vector probability vector stochastic matrix Markov chain steady-state vector random walk state diagram
Motivation
Change (or Waxing Poetic)
Change (or Waxing Poetic) Things change.
Change (or Waxing Poetic) Things change. Things change from one state of affairs to another state of affairs.
Change (or Waxing Poetic) Things change. Things change from one state of affairs to another state of affairs. Things change often in unpredictable ways.
Change (or Waxing Poetic) Things change. Things change from one state of affairs to another state of affairs. Things change often in unpredictable ways. If something changes unpredictably, what can we say about it?
Dynamical Systems
Dynamical Systems Definition (Informal). A dynamical system is a thing (typically with interacting parts) that changes over time.
Dynamical Systems Definition (Informal). A dynamical system is a thing (typically with interacting parts) that changes over time. A dynamical system has possible states which it can be in as time elapses and its behavior is defined by a evolution function.
Dynamical Systems Definition (Informal). A dynamical system is a thing (typically with interacting parts) that changes over time. A dynamical system has possible states which it can be in as time elapses and its behavior is defined by a evolution function. Examples. » economics (stocks) » physical/chemical systems » populations » weather
An Aside: Chaos Theory https://commons.wikimedia.org/wiki/File:Fr137.jpg
An Aside: Chaos Theory Complex systems like the weather or the economy look nearly random. https://commons.wikimedia.org/wiki/File:Fr137.jpg
An Aside: Chaos Theory Complex systems like the weather or the economy look nearly random. But even in chaotic systems there are underlying patterns and repeated structures. https://commons.wikimedia.org/wiki/File:Fr137.jpg
An Aside: Chaos Theory Complex systems like the weather or the economy look nearly random. But even in chaotic systems there are underlying patterns and repeated structures. Often it's useful to consider chaotic systems in terms of global properties. https://commons.wikimedia.org/wiki/File:Fr137.jpg
Motivating Questions
Motivating Questions What does a dynamical system look like "in the long view?"
Motivating Questions What does a dynamical system look like "in the long view?" Does it reach a kind of equilibrium? (think heat diffusion)
Motivating Questions What does a dynamical system look like "in the long view?" Does it reach a kind of equilibrium? (think heat diffusion) Or does some part of the system dominate over time? (think the population of rabbits without a predator)
(Linear) Dynamical Systems
Linear Dynamical Systems
Linear Dynamical Systems Definition. A (discrete time) linear dynamical system is a described a matrix . It's evolution function is the matrix transformation . n × n A x ↦ Ax
Linear Dynamical Systems Definition. A (discrete time) linear dynamical system is a described a matrix . It's evolution function is the matrix transformation . n × n A x ↦ Ax The possible states of the system are vectors in . ℝn
Linear Dynamical Systems Definition. A (discrete time) linear dynamical system is a described a matrix . It's evolution function is the matrix transformation . n × n A x ↦ Ax The possible states of the system are vectors in . ℝn Given an initial state vector , we can determine the state vector of the system after time steps: v0 i vi = Avi−1
Linear Dynamical Systems Definition. A (discrete time) linear dynamical system is a described a matrix . It's evolution function is the matrix transformation . n × n A x ↦ Ax The possible states of the system are vectors in . ℝn Given an initial state vector , we can determine the state vector of the system after time steps: v0 i vi = Avi−1 tells us how our system evolves over time. A
State Vectors The state vector tells us what the system looks like after a number time steps. This is also called a recurrence relation or a linear difference function. vk k v1 = Av0 v2 = Av1 = A(Av0) v3 = Av2 = A(AAv0) v4 = Av3 = A(AAAv0) v5 = Av4 = A(AAAAv0) ⋮
How to: Determining State Vectors Question. Determine the state vector for the linear dynamical system with matrix given the initial state vector . Solution. Compute vi A v0 vi = Ai v0
Matrix Powers in NumPy numpy.linalg.matrix_power(a)
Matrix Powers in NumPy There is a function in NumPy for doing matrix powers. numpy.linalg.matrix_power(a)
Matrix Powers in NumPy There is a function in NumPy for doing matrix powers. Use can use this when you need to take a large power of a matrix. numpy.linalg.matrix_power(a)
Matrix Powers in NumPy There is a function in NumPy for doing matrix powers. Use can use this when you need to take a large power of a matrix. It's much faster than doing each multiplication individually because it uses the "repeated squaring" trick numpy.linalg.matrix_power(a)
Matrix Powers in NumPy There is a function in NumPy for doing matrix powers. Use can use this when you need to take a large power of a matrix. It's much faster than doing each multiplication individually because it uses the "repeated squaring" trick But be cautious of floating-point error. numpy.linalg.matrix_power(a)
Warm up: Population Dynamics
The Setup
The Setup We're working for the census. We have 2023 population measurements for a city and a suburb which are geographically coincident.
The Setup We're working for the census. We have 2023 population measurements for a city and a suburb which are geographically coincident. We find by analyzing previous data that each year: » 5% of the population moves from city suburb → » 3% of the population moves from suburb city →
Fundamental Question Can we make any predictions about the population of the city and suburb in 2043? Note: No immigration, emigration, birth, death, etc. The overall population stays fixed.
Setting up Linear Equations
Setting up Linear Equations If = 2023 city pop. = 600,000 and = 2023 suburb pop. = 400,000 𝖼 𝗂 𝗍 𝗒 0 𝗌 𝗎 𝖻 𝗎 𝗋 𝖻 0
Setting up Linear Equations If = 2023 city pop. = 600,000 and = 2023 suburb pop. = 400,000 𝖼 𝗂 𝗍 𝗒 0 𝗌 𝗎 𝖻 𝗎 𝗋 𝖻 0 then the pop. in 2024 are given by: 𝖼 𝗂 𝗍 𝗒 1 = (0.95) 𝖼 𝗂 𝗍 𝗒 0 + (0.03) 𝗌 𝗎 𝖻 𝗎 𝗋 𝖻 0 𝗌 𝗎 𝖻 𝗎 𝗋 𝖻 1 = (0.05) 𝖼 𝗂 𝗍 𝗒 0 + (0.97) 𝗌 𝗎 𝖻 𝗎 𝗋 𝖻 0
Setting up Linear Equations If = 2023 city pop. = 600,000 and = 2023 suburb pop. = 400,000 𝖼 𝗂 𝗍 𝗒 0 𝗌 𝗎 𝖻 𝗎 𝗋 𝖻 0 then the pop. in 2024 are given by: 𝖼 𝗂 𝗍 𝗒 1 = (0.95) 𝖼 𝗂 𝗍 𝗒 0 + (0.03) 𝗌 𝗎 𝖻 𝗎 𝗋 𝖻 0 𝗌 𝗎 𝖻 𝗎 𝗋 𝖻 1 = (0.05) 𝖼 𝗂 𝗍 𝗒 0 + (0.97) 𝗌 𝗎 𝖻 𝗎 𝗋 𝖻 0 people who stayed people who left
Setting up a Matrix In 2024, we expect the population of the city to decrease. [ 𝖼 𝗂 𝗍 𝗒 1 𝗌 𝗎 𝖻 𝗎 𝗋 𝖻 1 ] = [ 0.95 0.3 0.05 0.97] [ 𝖼 𝗂 𝗍 𝗒 0 𝗌 𝗎 𝖻 𝗎 𝗋 𝖻 0 ] = [ 582,000 418,000]
Setting up a Matrix In 2025, we expect the population of the city to continue to decrease. Will it decrease indefinitely? [ 𝖼 𝗂 𝗍 𝗒 2 𝗌 𝗎 𝖻 𝗎 𝗋 𝖻 2 ] = [ 0.95 0.3 0.05 0.97] [ 𝖼 𝗂 𝗍 𝗒 1 𝗌 𝗎 𝖻 𝗎 𝗋 𝖻 1 ] = [ 565,440 434,560]
Setting up a Matrix This is a linear dynamical system. So we want to guess what the population will look like in 20 years, we need to compute [ 0.95 0.03 0.05 0.97] 20 [ 𝖼 𝗂 𝗍 𝗒 0 𝗌 𝗎 𝖻 𝗎 𝗋 𝖻 0 ] [ 𝖼 𝗂 𝗍 𝗒 k 𝗌 𝗎 𝖻 𝗎 𝗋 𝖻 k ] = [ 0.95 0.3 0.05 0.97] [ 𝖼 𝗂 𝗍 𝗒 k−1 𝗌 𝗎 𝖻 𝗎 𝗋 𝖻 k−1 ]
demo
Markov Chains
Stochastic Matrices What's special about this matrix? » Its entries are nonnegative. » Its columns sum to 1. This should make us think probability. [ 0.95 0.03 0.05 0.97]
Stochastic Matrices Definition. A matrix is stochastic if its entries are nonnegative and its columns sum to 1. Example. n × n [ 0.7 0.1 0.3 0.2 0.8 0.3 0.1 0.1 0.4]
Markov Chains Definition. A Markov chain is a linear dynamical system whose evolution function is given by a stochastic matrix. (We can construct a "chain" of state vectors, where each state vector only depends on the one before it.)
Key Property of Stochastic Matrices
Key Property of Stochastic Matrices Stochastic matrices redistribute the "stuff" in a vector.
Key Property of Stochastic Matrices Stochastic matrices redistribute the "stuff" in a vector. Theorem. For a stochastic matrix and a vector , A v sum of entries of v sum of entries of Av =
Key Property of Stochastic Matrices The sum of the entries of can be computed as So the previous statement can be written v 1T v = ⟨1, v⟩ 1T (Av) = 1T v
Key Property of Stochastic Matrices Let's verify this: (I'll leave it as an exercise) 1T (Av) = 1T v is stochastic A
More General Solutions
More General Solutions In our example, we analyzed the dynamics of a particular population.
More General Solutions In our example, we analyzed the dynamics of a particular population. What if we're interested more generally in the behavior of the process for any population?
More General Solutions In our example, we analyzed the dynamics of a particular population. What if we're interested more generally in the behavior of the process for any population? We need to shift from a population vector to a population distribution vector.
Returning to the Example [ 𝖼 𝗂 𝗍 𝗒 k 𝗌 𝗎 𝖻 𝗎 𝗋 𝖻 k ] = [ 0.95 0.3 0.05 0.97] [ 𝖼 𝗂 𝗍 𝗒 k−1 𝗌 𝗎 𝖻 𝗎 𝗋 𝖻 k−1 ]
Returning to the Example [ 𝖼 𝗂 𝗍 𝗒 k 𝗌 𝗎 𝖻 𝗎 𝗋 𝖻 k ] = [ 0.95 0.3 0.05 0.97] k [ 𝖼 𝗂 𝗍 𝗒 0 𝗌 𝗎 𝖻 𝗎 𝗋 𝖻 0 ]
Returning to the Example [ 𝖼 𝗂 𝗍 𝗒 k 𝗌 𝗎 𝖻 𝗎 𝗋 𝖻 k ] = [ 0.95 0.3 0.05 0.97] k [ 600,000 400,000]
Returning to the Example But what if we start of with a different population? [ 𝖼 𝗂 𝗍 𝗒 k 𝗌 𝗎 𝖻 𝗎 𝗋 𝖻 k ] = [ 0.95 0.3 0.05 0.97] k [ 600,000 400,000]
Returning to the Example But what if we start of with a different population? Do we have to do all our work over again? [ 𝖼 𝗂 𝗍 𝗒 k 𝗌 𝗎 𝖻 𝗎 𝗋 𝖻 k ] = [ 0.95 0.3 0.05 0.97] k [ 600,000 400,000]
Returning to the Example Not really. But rather than thinking in terms of populations, we need to think about how the population is distributed. [ 𝖼 𝗂 𝗍 𝗒 k 𝗌 𝗎 𝖻 𝗎 𝗋 𝖻 k ] = [ 0.95 0.3 0.05 0.97] k [ 0.6 0.4] 60% of pop. in city 40% of pop. in suburb
Probability Vectors
Probability Vectors Definition. A probability vector is a vector whose entries sum to 1.
Probability Vectors Definition. A probability vector is a vector whose entries sum to 1. They represent » discrete probability distributions » distributions of collections of things
Probability Vectors Definition. A probability vector is a vector whose entries sum to 1. They represent » discrete probability distributions » distributions of collections of things These are really the same thing.
Probability Vectors (Example) The vector represents the distribution where we choose: with probability with probability with probability 1/3 1/6 1/2 1 1/3 2 1/6 3 1/2
Probability Vectors (Example) The vector represented the distribution of the population, but we can also think of this as: If we choose a random person from the population we'll get someone: in the city with probability in the suburbs with probability [ 0.6 0.4] 0.6 0.4
The point
The point We'll be interested in the dynamics of Markov chains on probability vectors.
The point We'll be interested in the dynamics of Markov chains on probability vectors. Since stochastic matrices preserve , they transform one distribution into another. 1T v
The point We'll be interested in the dynamics of Markov chains on probability vectors. Since stochastic matrices preserve , they transform one distribution into another. 1T v Can we say something about how the distribution changes in the long run?
Steady-State Vectors
Steady-State Vectors Definition. A steady-state vector for a stochastic matrix is a probability vector such that A steady-state vector is not changed by the stochastic matrix. They describe equilibrium distributions. A q Aq = q
Returning to the Example
Returning to the Example How do we interpret a steady-state vector for our example?
Returning to the Example How do we interpret a steady-state vector for our example? The populations in the city and the suburb stay the same over time.
Returning to the Example How do we interpret a steady-state vector for our example? The populations in the city and the suburb stay the same over time. The same number of people are moving into and out of the city each year.
Fundamental Questions Do steady states exist? Are they unique? How do we find them?
Finding Steady-State Vectors Let's solve this equation for . q Aq = q
Finding Steady-State Vectors Let's solve this equation for . q Aq − q = 0
Finding Steady-State Vectors Let's solve this equation for . q Aq − Iq = 0
Finding Steady-State Vectors Let's solve this equation for . q (A − I)q = 0
Finding Steady-State Vectors Let's solve this equation for . q (A − I)q = 0 This is a matrix equation. So we know how to solve it.
How to: Steady-State Vectors
How to: Steady-State Vectors Question. Determine if the Markov chain with stochastic matrix has a steady-state vector. If it does, find such a vector. A
How to: Steady-State Vectors Question. Determine if the Markov chain with stochastic matrix has a steady-state vector. If it does, find such a vector. A Solution. Solve the equation and find a solution whose entries sum to 1 (this will be possible given a free variable). (A − I)x = 0
How to: Steady-State Vectors Question. Determine if the Markov chain with stochastic matrix has a steady-state vector. If it does, find such a vector. A Solution. Solve the equation and find a solution whose entries sum to 1 (this will be possible given a free variable). (A − I)x = 0 If there is no such solution, the system does not have a steady state.
demo
Existence vs Convergence If infinitely many solutions, then it has a stable state. This does not mean: » the stable state is unique » the system converges to this state (A − I)x = 0
Convergence
Convergence Definition. For a Markov chain with stochastic matrix , an initial state converges to the state if . A v0 v lim k→∞ Ak v0 = v
Convergence Definition. For a Markov chain with stochastic matrix , an initial state converges to the state if . A v0 v lim k→∞ Ak v0 = v As we repeatedly multiply by , we get closer and closer to (in the limit). v0 A v
Convergence Definition. For a Markov chain with stochastic matrix , an initial state converges to the state if . A v0 v lim k→∞ Ak v0 = v As we repeatedly multiply by , we get closer and closer to (in the limit). v0 A v
Example of Non-Convergence
Example of Non-Convergence Non-Example. is a stochastic matrix and I Iv = v for any choice of . v
Example of Non-Convergence Non-Example. is a stochastic matrix and I Iv = v for any choice of . v So this system does not have a unique steady state.
Example of Non-Convergence Non-Example. is a stochastic matrix and I Iv = v for any choice of . v So this system does not have a unique steady state. And no vectors converge to the same stable state.
Regular Stochastic Matrices
Regular Stochastic Matrices Definition. A stochastic matrix is regular if has all positive entries for some nonnegative . A Ak k
Regular Stochastic Matrices Definition. A stochastic matrix is regular if has all positive entries for some nonnegative . A Ak k Theorem. A regular stochastic matrix has a unique steady state, and P every probability vector converges to it
Mixing This process of converging to a unique steady state is called "mixing." This theorem says, after some amount of mixing, we'll be close to the stable state, no matter where we started.
How to: Regular Stochastic Matrices Question. Show that is regular, and then find it's unique steady state. Solution. Find a power of which has all positive entries, then solve the equation as before. A A (A − I)x = 0
Random Walks
Recall: Adjacency Matrices Let be an undirected unweighted graph with its nodes labeled by numbers through . We can create the adjacency matrix for as follows. G 1 n A G 0 1 0 0 1 0 1 0 1 0 1 0 0 1 0 1 0 0 0 0 1 0 1 1 1 1 0 1 0 0 0 0 0 1 0 0 A12 A21 A43 A34 A64 A46 Aij = { 1 there is an edge between i and j 0 otherwise
Random Walk
Random Walk A random walk on an undirected unweighted starting at is the following process: G v
Random Walk A random walk on an undirected unweighted starting at is the following process: G v » if is connected to nodes, roll a -sided die v k k
Random Walk A random walk on an undirected unweighted starting at is the following process: G v » if is connected to nodes, roll a -sided die v k k » go to the th vertex according to some order k
Random Walk A random walk on an undirected unweighted starting at is the following process: G v » if is connected to nodes, roll a -sided die v k k » go to the th vertex according to some order k » repeat
Applications of Random Walks Brownian Motion is a random walk in 3D space. Random walks are to simulate complex systems in physics and in economics. They are also used to design algorithms. https://commons.wikimedia.org/wiki/File:Wiener_process_3d.png
General Adjacency Matrices We can extend the notion of an adjacency matrix to directed and weighted graphs. Example. Aij = { wji there is an edge from j to i 0 otherwise 0 0 1 −2 0 −3 0 5 0 1 -2 -3 5 1 2 3
State Diagrams Definition. A state diagram is a directed weighted graph whose adjacency matrix is stochastic. Example. 𝖼 𝗂 𝗍 𝗒 𝗌 𝗎 𝖻 𝗎 𝗋 𝖻 0.95 0.03 0.05 0.97
Naming Convention Clash The nodes of a state diagram are often called states. The vectors which are dynamically updated according to a linear dynamical system are called state vectors. This is an unfortunate naming clash.
Example: Computer System Imagine a computer system in which tasks request service from disk, network or CPU. In the long term, which device is busiest? This is about finding a stable state.
How To: State Diagram Question. Given a state diagram, find the stable state for the corresponding linear dynamical system. Solution. Find the adjacency matrix for the state diagram and go from there.
Random Walks as Linear Dynamical Systems Once we have a stochastic matrix, we can reason about random walks as linear dynamical systems. What are its steady states? How do we interpret these steady states?
Random Walks on State Diagrams A random walk on a state diagram starting at is the following process: » choose a node is connected to according to the distribution given by the edge weights » go to that node » repeat v v
Random Walks on State Diagrams A random walk on a state diagram starting at is the following process: » choose a node is connected to according to the distribution given by the edge weights » go to that node » repeat v v Stable states of linear dynamical systems are stable states of random walks on state diagrams.
Steady-States of Random Walks Theorem (Advanced). Let be the stochastic matrix for the graph . The probability that a random walk starting at of length ends on node is the th entry of the vector A G i k j (Ak ei)j j Ak ei transforms a distribution for length walks to length walks. A k k + 1
Steady States of Random Walks If a random walk goes on for a sufficiently long time, then the probability that we end up in a particular place becomes fixed. If you wander for a sufficiently long time, it doesn't matter where you started.
Summary Markov chains allow us to reason about dynamical systems that are dictated by some amount of randomness. Stable states represent global equilibrium. We can think of Markov chains as random walks on state diagrams.

More Related Content

PPTX
Advance control theory
SHIMI S L
 
PDF
Fundamentos de la cadena de markov - Libro
Nelson Salinas
 
PPTX
Spacey random walks CAM Colloquium
Austin Benson
 
PPTX
State space analysis.pptx
RaviMuthamala1
 
PPTX
Eigenvectors & Eigenvalues: The Road to Diagonalisation
Christopher Gratton
 
PPTX
Quantum mechanics an overview
SkillBridge360 - Home Tutoring & Digital Services
 
PPTX
Spacey random walks CMStatistics 2017
Austin Benson
 
PDF
Modern Control System (BE)
PRABHAHARAN429
 
Advance control theory
SHIMI S L
 
Fundamentos de la cadena de markov - Libro
Nelson Salinas
 
Spacey random walks CAM Colloquium
Austin Benson
 
State space analysis.pptx
RaviMuthamala1
 
Eigenvectors & Eigenvalues: The Road to Diagonalisation
Christopher Gratton
 
Quantum mechanics an overview
SkillBridge360 - Home Tutoring & Digital Services
 
Spacey random walks CMStatistics 2017
Austin Benson
 
Modern Control System (BE)
PRABHAHARAN429
 

Similar to 12-markov-chains and dynamical systems.pdf (20)

PPTX
Jacobian | velocity and static forces
Abrar Mohamed
 
PPTX
Unit 5: All
Hector Zenil
 
PDF
By BIRASA FABRICE
BIRASA FABRICE NYIRIMANA
 
DOCX
4 forced vibration of damped
Jayesh Chopade
 
PDF
Eigen value and eigen vectors shwetak
MrsShwetaBanait1
 
PPTX
Discrete state space model 9th &10th lecture
Khalaf Gaeid Alshammery
 
PDF
solver (1)
Raj Mitra
 
PDF
Control Systems (K-Wiki_State Variable Analysis).pdf
JaiGanesh838418
 
PPTX
Eigenstates of 2D Random Walk with Multiple Absorbing States
Kyle Poe
 
PDF
Lecture 4 sapienza 2017
Franco Bontempi Org Didattica
 
PPT
2-Mathematical Modeling of Systems.ppt
RizaMuhida1
 
PDF
Ingeniería de control: Tema 3. El método del espacio de estados
SANTIAGO PABLO ALBERTO
 
PPTX
Aplicaciones y subespacios y subespacios vectoriales en la
emojose107
 
PPT
CMU Advanced Controls M60AC Lesson #2.ppt
DrMiltonRichardson
 
PPT
CMU Advanced Controls M60AC Lesson #2.ppt
DrMiltonRichardson
 
PPTX
Unit 3-State-Space Analysis_all.pptx
datamboli
 
PDF
PPTChapter12.pdf
Vara Prasad
 
PPTX
4852014.pptx
ssuser72b8e8
 
PPTX
Hmm and neural networks
Janani Ramasamy
 
PPTX
Linear Control Systems unit 5 complete d
Ravi821178
 
Jacobian | velocity and static forces
Abrar Mohamed
 
Unit 5: All
Hector Zenil
 
By BIRASA FABRICE
BIRASA FABRICE NYIRIMANA
 
4 forced vibration of damped
Jayesh Chopade
 
Eigen value and eigen vectors shwetak
MrsShwetaBanait1
 
Discrete state space model 9th &10th lecture
Khalaf Gaeid Alshammery
 
solver (1)
Raj Mitra
 
Control Systems (K-Wiki_State Variable Analysis).pdf
JaiGanesh838418
 
Eigenstates of 2D Random Walk with Multiple Absorbing States
Kyle Poe
 
Lecture 4 sapienza 2017
Franco Bontempi Org Didattica
 
2-Mathematical Modeling of Systems.ppt
RizaMuhida1
 
Ingeniería de control: Tema 3. El método del espacio de estados
SANTIAGO PABLO ALBERTO
 
Aplicaciones y subespacios y subespacios vectoriales en la
emojose107
 
CMU Advanced Controls M60AC Lesson #2.ppt
DrMiltonRichardson
 
CMU Advanced Controls M60AC Lesson #2.ppt
DrMiltonRichardson
 
Unit 3-State-Space Analysis_all.pptx
datamboli
 
PPTChapter12.pdf
Vara Prasad
 
4852014.pptx
ssuser72b8e8
 
Hmm and neural networks
Janani Ramasamy
 
Linear Control Systems unit 5 complete d
Ravi821178
 
Ad

More from Mohammad732983 (9)

PDF
محاضرة 6 التوزيع الاحتمالي في حالة التباين مجهول.pdf
Mohammad732983
 
PDF
محاضرة التوزيع الاحتمالي لتباين ونسبة العينة5.pdf
Mohammad732983
 
PDF
محاضرة نظرية النهاية المركزية والتحليل الاحصائي4.pdf
Mohammad732983
 
PDF
----المحاضرة الاولى محاضرة بحوث عمليات.pdf
Mohammad732983
 
PPT
Basics and fundamentals of Game theory Lecture 1.ppt
Mohammad732983
 
PDF
presentation lecture of game theory .pdf
Mohammad732983
 
PDF
presentation lecture 2 of game theory.pdf
Mohammad732983
 
PDF
Lecture 2 presentation of game theory.pdf
Mohammad732983
 
PDF
Lecture 6-1. normal and extensive form of game theory
Mohammad732983
 
محاضرة 6 التوزيع الاحتمالي في حالة التباين مجهول.pdf
Mohammad732983
 
محاضرة التوزيع الاحتمالي لتباين ونسبة العينة5.pdf
Mohammad732983
 
محاضرة نظرية النهاية المركزية والتحليل الاحصائي4.pdf
Mohammad732983
 
----المحاضرة الاولى محاضرة بحوث عمليات.pdf
Mohammad732983
 
Basics and fundamentals of Game theory Lecture 1.ppt
Mohammad732983
 
presentation lecture of game theory .pdf
Mohammad732983
 
presentation lecture 2 of game theory.pdf
Mohammad732983
 
Lecture 2 presentation of game theory.pdf
Mohammad732983
 
Lecture 6-1. normal and extensive form of game theory
Mohammad732983
 
Ad

Recently uploaded (20)

PDF
. Radiology Case Scenariosssssssssssssss
MiraMusleh
 
PDF
Mastering Bioreactors and Media Sterilization: A Complete Guide to Sterile Fe...
Sabina Sharma
 
PDF
Assessment of environmental effects of quarrying in Kitengela subcountyof Kaj...
mohamedmahmoudSebbab1
 
PPTX
Microbiology with diagram medical studies .pptx
mohammedabbusiddiq
 
PPTX
neck nodes and dissection types and lymph nodes levels
DipuSarma
 
PPTX
Vitamins & Minerals: Complete Guide to Functions, Food Sources, Deficiency Si...
Muhammad Sajid Afridi
 
PDF
lecture 2026 of Sjogren's syndrome l .pdf
ssuserf13226
 
PDF
Sciences of Europe No 170 (2025)
Sciences of Europe
 
PDF
The scientific heritage No 166 (166) (2025)
The scientific heritage
 
PPT
The World of Physical Science, • Labs: Safety Simulation, Measurement Practice
joybejerano
 
PPTX
BIOMOLECULES PPT........................
vachieagrawal1221
 
PPTX
cpcsea ppt.pptxssssssssssssssjjdjdndndddd
amolpawasesng123
 
PDF
Phytochemical Investigation of Miliusa longipes.pdf
IrfanShahirSharafi
 
PPTX
The KM-GBF monitoring framework – status & key messages.pptx
pensoftservices
 
PPTX
ANEMIA WITH LEUKOPENIA MDS 07_25.pptx htggtftgt fredrctvg
vedvyas5668
 
PPTX
Pharmacology of Autonomic nervous system
omkarkumar17
 
PPTX
TOTAL hIP ARTHROPLASTY Presentation.pptx
hararghemssc
 
PDF
CAPERS-LRD-z9:AGas-enshroudedLittleRedDotHostingaBroad-lineActive GalacticNuc...
Sérgio Sacani
 
PPTX
Introduction to Fisheries Biotechnology_Lesson 1.pptx
Esther Pongcol
 
PDF
Looking into the jet cone of the neutrino-associated very high-energy blazar ...
Sérgio Sacani
 
. Radiology Case Scenariosssssssssssssss
MiraMusleh
 
Mastering Bioreactors and Media Sterilization: A Complete Guide to Sterile Fe...
Sabina Sharma
 
Assessment of environmental effects of quarrying in Kitengela subcountyof Kaj...
mohamedmahmoudSebbab1
 
Microbiology with diagram medical studies .pptx
mohammedabbusiddiq
 
neck nodes and dissection types and lymph nodes levels
DipuSarma
 
Vitamins & Minerals: Complete Guide to Functions, Food Sources, Deficiency Si...
Muhammad Sajid Afridi
 
lecture 2026 of Sjogren's syndrome l .pdf
ssuserf13226
 
Sciences of Europe No 170 (2025)
Sciences of Europe
 
The scientific heritage No 166 (166) (2025)
The scientific heritage
 
The World of Physical Science, • Labs: Safety Simulation, Measurement Practice
joybejerano
 
BIOMOLECULES PPT........................
vachieagrawal1221
 
cpcsea ppt.pptxssssssssssssssjjdjdndndddd
amolpawasesng123
 
Phytochemical Investigation of Miliusa longipes.pdf
IrfanShahirSharafi
 
The KM-GBF monitoring framework – status & key messages.pptx
pensoftservices
 
ANEMIA WITH LEUKOPENIA MDS 07_25.pptx htggtftgt fredrctvg
vedvyas5668
 
Pharmacology of Autonomic nervous system
omkarkumar17
 
TOTAL hIP ARTHROPLASTY Presentation.pptx
hararghemssc
 
CAPERS-LRD-z9:AGas-enshroudedLittleRedDotHostingaBroad-lineActive GalacticNuc...
Sérgio Sacani
 
Introduction to Fisheries Biotechnology_Lesson 1.pptx
Esther Pongcol
 
Looking into the jet cone of the neutrino-associated very high-energy blazar ...
Sérgio Sacani
 

12-markov-chains and dynamical systems.pdf

  • 1. CAS CS 132 Markov Chains Geometric Algorithms Lecture 12
  • 2. Objectives 1. Motivate linear dynamical systems 2. Analyze Markov chains and their properties 3. Learn to solve for steady-states of Markov chains 4. Connect this to graphs and random walks
  • 3. Keywords linear dynamical systems recurrence relations linear difference equations state vector probability vector stochastic matrix Markov chain steady-state vector random walk state diagram
  • 6. Change (or Waxing Poetic) Things change.
  • 7. Change (or Waxing Poetic) Things change. Things change from one state of affairs to another state of affairs.
  • 8. Change (or Waxing Poetic) Things change. Things change from one state of affairs to another state of affairs. Things change often in unpredictable ways.
  • 9. Change (or Waxing Poetic) Things change. Things change from one state of affairs to another state of affairs. Things change often in unpredictable ways. If something changes unpredictably, what can we say about it?
  • 11. Dynamical Systems Definition (Informal). A dynamical system is a thing (typically with interacting parts) that changes over time.
  • 12. Dynamical Systems Definition (Informal). A dynamical system is a thing (typically with interacting parts) that changes over time. A dynamical system has possible states which it can be in as time elapses and its behavior is defined by a evolution function.
  • 13. Dynamical Systems Definition (Informal). A dynamical system is a thing (typically with interacting parts) that changes over time. A dynamical system has possible states which it can be in as time elapses and its behavior is defined by a evolution function. Examples. » economics (stocks) » physical/chemical systems » populations » weather
  • 14. An Aside: Chaos Theory https://commons.wikimedia.org/wiki/File:Fr137.jpg
  • 15. An Aside: Chaos Theory Complex systems like the weather or the economy look nearly random. https://commons.wikimedia.org/wiki/File:Fr137.jpg
  • 16. An Aside: Chaos Theory Complex systems like the weather or the economy look nearly random. But even in chaotic systems there are underlying patterns and repeated structures. https://commons.wikimedia.org/wiki/File:Fr137.jpg
  • 17. An Aside: Chaos Theory Complex systems like the weather or the economy look nearly random. But even in chaotic systems there are underlying patterns and repeated structures. Often it's useful to consider chaotic systems in terms of global properties. https://commons.wikimedia.org/wiki/File:Fr137.jpg
  • 19. Motivating Questions What does a dynamical system look like "in the long view?"
  • 20. Motivating Questions What does a dynamical system look like "in the long view?" Does it reach a kind of equilibrium? (think heat diffusion)
  • 21. Motivating Questions What does a dynamical system look like "in the long view?" Does it reach a kind of equilibrium? (think heat diffusion) Or does some part of the system dominate over time? (think the population of rabbits without a predator)
  • 24. Linear Dynamical Systems Definition. A (discrete time) linear dynamical system is a described a matrix . It's evolution function is the matrix transformation . n × n A x ↦ Ax
  • 25. Linear Dynamical Systems Definition. A (discrete time) linear dynamical system is a described a matrix . It's evolution function is the matrix transformation . n × n A x ↦ Ax The possible states of the system are vectors in . ℝn
  • 26. Linear Dynamical Systems Definition. A (discrete time) linear dynamical system is a described a matrix . It's evolution function is the matrix transformation . n × n A x ↦ Ax The possible states of the system are vectors in . ℝn Given an initial state vector , we can determine the state vector of the system after time steps: v0 i vi = Avi−1
  • 27. Linear Dynamical Systems Definition. A (discrete time) linear dynamical system is a described a matrix . It's evolution function is the matrix transformation . n × n A x ↦ Ax The possible states of the system are vectors in . ℝn Given an initial state vector , we can determine the state vector of the system after time steps: v0 i vi = Avi−1 tells us how our system evolves over time. A
  • 28. State Vectors The state vector tells us what the system looks like after a number time steps. This is also called a recurrence relation or a linear difference function. vk k v1 = Av0 v2 = Av1 = A(Av0) v3 = Av2 = A(AAv0) v4 = Av3 = A(AAAv0) v5 = Av4 = A(AAAAv0) ⋮
  • 29. How to: Determining State Vectors Question. Determine the state vector for the linear dynamical system with matrix given the initial state vector . Solution. Compute vi A v0 vi = Ai v0
  • 30. Matrix Powers in NumPy numpy.linalg.matrix_power(a)
  • 31. Matrix Powers in NumPy There is a function in NumPy for doing matrix powers. numpy.linalg.matrix_power(a)
  • 32. Matrix Powers in NumPy There is a function in NumPy for doing matrix powers. Use can use this when you need to take a large power of a matrix. numpy.linalg.matrix_power(a)
  • 33. Matrix Powers in NumPy There is a function in NumPy for doing matrix powers. Use can use this when you need to take a large power of a matrix. It's much faster than doing each multiplication individually because it uses the "repeated squaring" trick numpy.linalg.matrix_power(a)
  • 34. Matrix Powers in NumPy There is a function in NumPy for doing matrix powers. Use can use this when you need to take a large power of a matrix. It's much faster than doing each multiplication individually because it uses the "repeated squaring" trick But be cautious of floating-point error. numpy.linalg.matrix_power(a)
  • 37. The Setup We're working for the census. We have 2023 population measurements for a city and a suburb which are geographically coincident.
  • 38. The Setup We're working for the census. We have 2023 population measurements for a city and a suburb which are geographically coincident. We find by analyzing previous data that each year: » 5% of the population moves from city suburb → » 3% of the population moves from suburb city →
  • 39. Fundamental Question Can we make any predictions about the population of the city and suburb in 2043? Note: No immigration, emigration, birth, death, etc. The overall population stays fixed.
  • 40. Setting up Linear Equations
  • 41. Setting up Linear Equations If = 2023 city pop. = 600,000 and = 2023 suburb pop. = 400,000 𝖼 𝗂 𝗍 𝗒 0 𝗌 𝗎 𝖻 𝗎 𝗋 𝖻 0
  • 42. Setting up Linear Equations If = 2023 city pop. = 600,000 and = 2023 suburb pop. = 400,000 𝖼 𝗂 𝗍 𝗒 0 𝗌 𝗎 𝖻 𝗎 𝗋 𝖻 0 then the pop. in 2024 are given by: 𝖼 𝗂 𝗍 𝗒 1 = (0.95) 𝖼 𝗂 𝗍 𝗒 0 + (0.03) 𝗌 𝗎 𝖻 𝗎 𝗋 𝖻 0 𝗌 𝗎 𝖻 𝗎 𝗋 𝖻 1 = (0.05) 𝖼 𝗂 𝗍 𝗒 0 + (0.97) 𝗌 𝗎 𝖻 𝗎 𝗋 𝖻 0
  • 43. Setting up Linear Equations If = 2023 city pop. = 600,000 and = 2023 suburb pop. = 400,000 𝖼 𝗂 𝗍 𝗒 0 𝗌 𝗎 𝖻 𝗎 𝗋 𝖻 0 then the pop. in 2024 are given by: 𝖼 𝗂 𝗍 𝗒 1 = (0.95) 𝖼 𝗂 𝗍 𝗒 0 + (0.03) 𝗌 𝗎 𝖻 𝗎 𝗋 𝖻 0 𝗌 𝗎 𝖻 𝗎 𝗋 𝖻 1 = (0.05) 𝖼 𝗂 𝗍 𝗒 0 + (0.97) 𝗌 𝗎 𝖻 𝗎 𝗋 𝖻 0 people who stayed people who left
  • 44. Setting up a Matrix In 2024, we expect the population of the city to decrease. [ 𝖼 𝗂 𝗍 𝗒 1 𝗌 𝗎 𝖻 𝗎 𝗋 𝖻 1 ] = [ 0.95 0.3 0.05 0.97] [ 𝖼 𝗂 𝗍 𝗒 0 𝗌 𝗎 𝖻 𝗎 𝗋 𝖻 0 ] = [ 582,000 418,000]
  • 45. Setting up a Matrix In 2025, we expect the population of the city to continue to decrease. Will it decrease indefinitely? [ 𝖼 𝗂 𝗍 𝗒 2 𝗌 𝗎 𝖻 𝗎 𝗋 𝖻 2 ] = [ 0.95 0.3 0.05 0.97] [ 𝖼 𝗂 𝗍 𝗒 1 𝗌 𝗎 𝖻 𝗎 𝗋 𝖻 1 ] = [ 565,440 434,560]
  • 46. Setting up a Matrix This is a linear dynamical system. So we want to guess what the population will look like in 20 years, we need to compute [ 0.95 0.03 0.05 0.97] 20 [ 𝖼 𝗂 𝗍 𝗒 0 𝗌 𝗎 𝖻 𝗎 𝗋 𝖻 0 ] [ 𝖼 𝗂 𝗍 𝗒 k 𝗌 𝗎 𝖻 𝗎 𝗋 𝖻 k ] = [ 0.95 0.3 0.05 0.97] [ 𝖼 𝗂 𝗍 𝗒 k−1 𝗌 𝗎 𝖻 𝗎 𝗋 𝖻 k−1 ]
  • 47. demo
  • 49. Stochastic Matrices What's special about this matrix? » Its entries are nonnegative. » Its columns sum to 1. This should make us think probability. [ 0.95 0.03 0.05 0.97]
  • 50. Stochastic Matrices Definition. A matrix is stochastic if its entries are nonnegative and its columns sum to 1. Example. n × n [ 0.7 0.1 0.3 0.2 0.8 0.3 0.1 0.1 0.4]
  • 51. Markov Chains Definition. A Markov chain is a linear dynamical system whose evolution function is given by a stochastic matrix. (We can construct a "chain" of state vectors, where each state vector only depends on the one before it.)
  • 52. Key Property of Stochastic Matrices
  • 53. Key Property of Stochastic Matrices Stochastic matrices redistribute the "stuff" in a vector.
  • 54. Key Property of Stochastic Matrices Stochastic matrices redistribute the "stuff" in a vector. Theorem. For a stochastic matrix and a vector , A v sum of entries of v sum of entries of Av =
  • 55. Key Property of Stochastic Matrices The sum of the entries of can be computed as So the previous statement can be written v 1T v = ⟨1, v⟩ 1T (Av) = 1T v
  • 56. Key Property of Stochastic Matrices Let's verify this: (I'll leave it as an exercise) 1T (Av) = 1T v is stochastic A
  • 58. More General Solutions In our example, we analyzed the dynamics of a particular population.
  • 59. More General Solutions In our example, we analyzed the dynamics of a particular population. What if we're interested more generally in the behavior of the process for any population?
  • 60. More General Solutions In our example, we analyzed the dynamics of a particular population. What if we're interested more generally in the behavior of the process for any population? We need to shift from a population vector to a population distribution vector.
  • 61. Returning to the Example [ 𝖼 𝗂 𝗍 𝗒 k 𝗌 𝗎 𝖻 𝗎 𝗋 𝖻 k ] = [ 0.95 0.3 0.05 0.97] [ 𝖼 𝗂 𝗍 𝗒 k−1 𝗌 𝗎 𝖻 𝗎 𝗋 𝖻 k−1 ]
  • 62. Returning to the Example [ 𝖼 𝗂 𝗍 𝗒 k 𝗌 𝗎 𝖻 𝗎 𝗋 𝖻 k ] = [ 0.95 0.3 0.05 0.97] k [ 𝖼 𝗂 𝗍 𝗒 0 𝗌 𝗎 𝖻 𝗎 𝗋 𝖻 0 ]
  • 63. Returning to the Example [ 𝖼 𝗂 𝗍 𝗒 k 𝗌 𝗎 𝖻 𝗎 𝗋 𝖻 k ] = [ 0.95 0.3 0.05 0.97] k [ 600,000 400,000]
  • 64. Returning to the Example But what if we start of with a different population? [ 𝖼 𝗂 𝗍 𝗒 k 𝗌 𝗎 𝖻 𝗎 𝗋 𝖻 k ] = [ 0.95 0.3 0.05 0.97] k [ 600,000 400,000]
  • 65. Returning to the Example But what if we start of with a different population? Do we have to do all our work over again? [ 𝖼 𝗂 𝗍 𝗒 k 𝗌 𝗎 𝖻 𝗎 𝗋 𝖻 k ] = [ 0.95 0.3 0.05 0.97] k [ 600,000 400,000]
  • 66. Returning to the Example Not really. But rather than thinking in terms of populations, we need to think about how the population is distributed. [ 𝖼 𝗂 𝗍 𝗒 k 𝗌 𝗎 𝖻 𝗎 𝗋 𝖻 k ] = [ 0.95 0.3 0.05 0.97] k [ 0.6 0.4] 60% of pop. in city 40% of pop. in suburb
  • 68. Probability Vectors Definition. A probability vector is a vector whose entries sum to 1.
  • 69. Probability Vectors Definition. A probability vector is a vector whose entries sum to 1. They represent » discrete probability distributions » distributions of collections of things
  • 70. Probability Vectors Definition. A probability vector is a vector whose entries sum to 1. They represent » discrete probability distributions » distributions of collections of things These are really the same thing.
  • 71. Probability Vectors (Example) The vector represents the distribution where we choose: with probability with probability with probability 1/3 1/6 1/2 1 1/3 2 1/6 3 1/2
  • 72. Probability Vectors (Example) The vector represented the distribution of the population, but we can also think of this as: If we choose a random person from the population we'll get someone: in the city with probability in the suburbs with probability [ 0.6 0.4] 0.6 0.4
  • 74. The point We'll be interested in the dynamics of Markov chains on probability vectors.
  • 75. The point We'll be interested in the dynamics of Markov chains on probability vectors. Since stochastic matrices preserve , they transform one distribution into another. 1T v
  • 76. The point We'll be interested in the dynamics of Markov chains on probability vectors. Since stochastic matrices preserve , they transform one distribution into another. 1T v Can we say something about how the distribution changes in the long run?
  • 78. Steady-State Vectors Definition. A steady-state vector for a stochastic matrix is a probability vector such that A steady-state vector is not changed by the stochastic matrix. They describe equilibrium distributions. A q Aq = q
  • 79. Returning to the Example
  • 80. Returning to the Example How do we interpret a steady-state vector for our example?
  • 81. Returning to the Example How do we interpret a steady-state vector for our example? The populations in the city and the suburb stay the same over time.
  • 82. Returning to the Example How do we interpret a steady-state vector for our example? The populations in the city and the suburb stay the same over time. The same number of people are moving into and out of the city each year.
  • 83. Fundamental Questions Do steady states exist? Are they unique? How do we find them?
  • 84. Finding Steady-State Vectors Let's solve this equation for . q Aq = q
  • 85. Finding Steady-State Vectors Let's solve this equation for . q Aq − q = 0
  • 86. Finding Steady-State Vectors Let's solve this equation for . q Aq − Iq = 0
  • 87. Finding Steady-State Vectors Let's solve this equation for . q (A − I)q = 0
  • 88. Finding Steady-State Vectors Let's solve this equation for . q (A − I)q = 0 This is a matrix equation. So we know how to solve it.
  • 90. How to: Steady-State Vectors Question. Determine if the Markov chain with stochastic matrix has a steady-state vector. If it does, find such a vector. A
  • 91. How to: Steady-State Vectors Question. Determine if the Markov chain with stochastic matrix has a steady-state vector. If it does, find such a vector. A Solution. Solve the equation and find a solution whose entries sum to 1 (this will be possible given a free variable). (A − I)x = 0
  • 92. How to: Steady-State Vectors Question. Determine if the Markov chain with stochastic matrix has a steady-state vector. If it does, find such a vector. A Solution. Solve the equation and find a solution whose entries sum to 1 (this will be possible given a free variable). (A − I)x = 0 If there is no such solution, the system does not have a steady state.
  • 93. demo
  • 94. Existence vs Convergence If infinitely many solutions, then it has a stable state. This does not mean: » the stable state is unique » the system converges to this state (A − I)x = 0
  • 96. Convergence Definition. For a Markov chain with stochastic matrix , an initial state converges to the state if . A v0 v lim k→∞ Ak v0 = v
  • 97. Convergence Definition. For a Markov chain with stochastic matrix , an initial state converges to the state if . A v0 v lim k→∞ Ak v0 = v As we repeatedly multiply by , we get closer and closer to (in the limit). v0 A v
  • 98. Convergence Definition. For a Markov chain with stochastic matrix , an initial state converges to the state if . A v0 v lim k→∞ Ak v0 = v As we repeatedly multiply by , we get closer and closer to (in the limit). v0 A v
  • 100. Example of Non-Convergence Non-Example. is a stochastic matrix and I Iv = v for any choice of . v
  • 101. Example of Non-Convergence Non-Example. is a stochastic matrix and I Iv = v for any choice of . v So this system does not have a unique steady state.
  • 102. Example of Non-Convergence Non-Example. is a stochastic matrix and I Iv = v for any choice of . v So this system does not have a unique steady state. And no vectors converge to the same stable state.
  • 104. Regular Stochastic Matrices Definition. A stochastic matrix is regular if has all positive entries for some nonnegative . A Ak k
  • 105. Regular Stochastic Matrices Definition. A stochastic matrix is regular if has all positive entries for some nonnegative . A Ak k Theorem. A regular stochastic matrix has a unique steady state, and P every probability vector converges to it
  • 106. Mixing This process of converging to a unique steady state is called "mixing." This theorem says, after some amount of mixing, we'll be close to the stable state, no matter where we started.
  • 107. How to: Regular Stochastic Matrices Question. Show that is regular, and then find it's unique steady state. Solution. Find a power of which has all positive entries, then solve the equation as before. A A (A − I)x = 0
  • 109. Recall: Adjacency Matrices Let be an undirected unweighted graph with its nodes labeled by numbers through . We can create the adjacency matrix for as follows. G 1 n A G 0 1 0 0 1 0 1 0 1 0 1 0 0 1 0 1 0 0 0 0 1 0 1 1 1 1 0 1 0 0 0 0 0 1 0 0 A12 A21 A43 A34 A64 A46 Aij = { 1 there is an edge between i and j 0 otherwise
  • 111. Random Walk A random walk on an undirected unweighted starting at is the following process: G v
  • 112. Random Walk A random walk on an undirected unweighted starting at is the following process: G v » if is connected to nodes, roll a -sided die v k k
  • 113. Random Walk A random walk on an undirected unweighted starting at is the following process: G v » if is connected to nodes, roll a -sided die v k k » go to the th vertex according to some order k
  • 114. Random Walk A random walk on an undirected unweighted starting at is the following process: G v » if is connected to nodes, roll a -sided die v k k » go to the th vertex according to some order k » repeat
  • 115. Applications of Random Walks Brownian Motion is a random walk in 3D space. Random walks are to simulate complex systems in physics and in economics. They are also used to design algorithms. https://commons.wikimedia.org/wiki/File:Wiener_process_3d.png
  • 116. General Adjacency Matrices We can extend the notion of an adjacency matrix to directed and weighted graphs. Example. Aij = { wji there is an edge from j to i 0 otherwise 0 0 1 −2 0 −3 0 5 0 1 -2 -3 5 1 2 3
  • 117. State Diagrams Definition. A state diagram is a directed weighted graph whose adjacency matrix is stochastic. Example. 𝖼 𝗂 𝗍 𝗒 𝗌 𝗎 𝖻 𝗎 𝗋 𝖻 0.95 0.03 0.05 0.97
  • 118. Naming Convention Clash The nodes of a state diagram are often called states. The vectors which are dynamically updated according to a linear dynamical system are called state vectors. This is an unfortunate naming clash.
  • 119. Example: Computer System Imagine a computer system in which tasks request service from disk, network or CPU. In the long term, which device is busiest? This is about finding a stable state.
  • 120. How To: State Diagram Question. Given a state diagram, find the stable state for the corresponding linear dynamical system. Solution. Find the adjacency matrix for the state diagram and go from there.
  • 121. Random Walks as Linear Dynamical Systems Once we have a stochastic matrix, we can reason about random walks as linear dynamical systems. What are its steady states? How do we interpret these steady states?
  • 122. Random Walks on State Diagrams A random walk on a state diagram starting at is the following process: » choose a node is connected to according to the distribution given by the edge weights » go to that node » repeat v v
  • 123. Random Walks on State Diagrams A random walk on a state diagram starting at is the following process: » choose a node is connected to according to the distribution given by the edge weights » go to that node » repeat v v Stable states of linear dynamical systems are stable states of random walks on state diagrams.
  • 124. Steady-States of Random Walks Theorem (Advanced). Let be the stochastic matrix for the graph . The probability that a random walk starting at of length ends on node is the th entry of the vector A G i k j (Ak ei)j j Ak ei transforms a distribution for length walks to length walks. A k k + 1
  • 125. Steady States of Random Walks If a random walk goes on for a sufficiently long time, then the probability that we end up in a particular place becomes fixed. If you wander for a sufficiently long time, it doesn't matter where you started.
  • 126. Summary Markov chains allow us to reason about dynamical systems that are dictated by some amount of randomness. Stable states represent global equilibrium. We can think of Markov chains as random walks on state diagrams.