A STUDY ON MARKOV CHAIN WITH TRANSITION DIAGRAM

International Journal of Technical Research and Applications e-ISSN: 2320-8163,
www.ijtra.com Volume 3, Issue 2 (Mar-Apr 2015), PP. 123-125
123 | P a g e
A STUDY ON MARKOV CHAIN WITH
TRANSITION DIAGRAM
M.Reni Sagayaraj1, A.Michael Raj2, G.Sathiyavani3
1,2,3
Sacred Heart College (Autonomous), Tirupattur,
Vellore, Tamilnadu, India.
reni.sagaya@gmail.com
Abstract—Stochastic processes have many useful applications
and are taught in several university programmers. In this paper
we are using stochastic process with complex concept on Markov
chains which uses a transition matrix to plot a transition diagram
and there are several examples which explains various type of
transition diagram. The concept behind this topic is simple and
easy to understand.
Keywords— Transition Diagram, Transition Matrix, Markov
Chain, First Passage Time, Persistent State, Transient State,
Periodic State, Inter-Communicating States.
I. INTRODUCTION
A Stochastic processes are important for modelling many
natural and social phenomena and have useful applications in
computer science, physics, biology, economics and finance. There are
many textbooks on stochastic processes, from introductory to
advanced ones [2] & [4]. Discrete Markov processes are the simplest
and most important class of stochastic processes.
There are only few publications on teaching stochastic processes
and Markov chains, such as [1] & [5] “More research needs to be
carried out on how to teach stochastic processes to researchers” [5].
“While trying to understand Markov chains models, students
usually encounter many obstacles and difficulties” [6]. Many
lecturers use visual displays such as sample paths and transition
diagrams to illustrate Markov chains. In this article we utilise
transition diagrams further for teaching several important concepts of
Markov chains. We explain in details how these concepts can be
defined in terms of transition diagrams (treated as directed weighted
graphs) and we accompany this with worked examples. Transition
diagrams provide a good techniques for solving some problems about
Markov chains, especially for students with poor mathematical
background.
II. TRANSITION DIAGRAM OF A MARKOV CHAIN:
DEFINITIONS
A homogeneous finite Markov chain is entirely defined
by its initial state distribution and its transition matrix S =
[pij], where pij = P(X1 = i|X0 = j) is the transition probability
from state j to state i.
The graphical representation of a Markov chain is a transition
diagram, which is equivalent to its transition matrix.
The transition diagram of a Markov chain X is a single weighted
directed graph, where each vertex represents a state of the Markov
chain and there is a directed edge from vertex j to vertex i if the
transition probability pij >0; this edge has the weight/probability of
pij.
Fig.1.The transition diagram of the Markov chain
Example1.
Example 1
A Markov chain has states 1, 2, 3, 4, 5, 6 and the following
transition matrix:
0 0 0 0 0 0.5
0.85 0 0 0.1 0 0
0 0.3 0.9 0 0 0
0 0.2 0.3 0 0 0
0.6 0.1 0 0.7 0 0
0 0 0 0 0.8 0
S
 
 
 
 
  
 
 
 
 
This is its transition diagram.
In the diagram in Fig. 1 the probability of each edge is shown
next to it. For example, the loop from state 3 to state 3 has probability
0.9 = p33 = P(X1 = 3 | X0 = 3) and the edge from state 2 to state 3 has
probability 0.3 = p32 = = P(X1 = 3 | X0 = 2).
In the graph terminology, an edge sequence of length n is an
ordered sequence of edges e1, e2, …, e n, where ei
and ei+1 are adjacent edges for all i = 1, 2,…, n-1.
A path is an edge sequence, where all edges are distinct. A simple
path is a path, where all vertices are distinct (except possibly the start
and end vertices). A cycle is a simple path there the start vertex and
the end vertex are the same.
In a transition diagram the probability of an edge sequence equals
a product of the probabilities of its edges.
III. PROPERTIES OF A MARKOV CHAIN IN TERMS OF
TRANSITION DIAGRAMS
A. N-Step Transition Probability:
An n-step transition probability is:
pij
(n)
= P  Xn = i | X0 = j .
It equals the probability of getting from state j to state i in
exactly n steps. It can be calculated as the corresponding element of
the matrix S(n)
but it is usually easier to find it from the transition
diagram as a sum of the probabilities of all edge sequences of length n
from j to i.
Example 2
In the chain from Example 1, the 3-step transition probability
from 2 to 1 equals:
p32
(3)
= a1 + a2

124 | P a g e
Where:
a1 = The probability of the path 2423
= = 0.006
a2 = The probability of the edge sequence 2333.
= 0.3 0.9 2
= 0.243
These probabilities are easy to find from the diagram
p32
3
 0.006  0.243  0.249.
p32
3
 0.249.
B. Probability of Visiting a State for the First Time:
Let us consider a random variable:
Ti = min {n ≥1 : Xn = i}.
It represents the number of steps to visit a state i for the
first time. It is called the first passage time of the state i. Related
probabilities are:
fij
(m)
= P  Ti = m | X0 = j and fij = P  Ti | X0 = j .
Clearly:
fij=
( )
1
m
ij
m
f



These probabilities can be interpreted as follows:
fij
(m)
= the probability to visit i on step m for the first time starting
from j;
fij
(m)
= the probability to visit i in finite number of starting from j;
In terms of transition diagrams, fij equals a sum of the
probabilities of all edge sequences from j to i that do not include the
vertex i between the start and end vertices.
fij
 m
equals a similar sum for the edge sequences of length m
only.
For finite Markov chains these probabilities are easier to find
from their transition diagrams than with other methods.
Example.3
From the transition diagram
we can calculate the following probabilities:
 f61
2
 as the probability of the path
156.
f61
n
for , f61
 For vertices 2 and 3 we have
f23

 
f23
2
 as the probability of the path 342.
f23
3
  as the probability of the path 3342.
and in general, for , for f23
n+1
n
 as the
probability of the edge sequence
( 1)
3...3 42
n times
with n loops
around 3.
So:
f23   
 
f23 = 0.3.
C. Persistent and Transient States
A state i of a Markov chain is called persistent if fii = 1 and
transient otherwise.
Thus, if the chain starts at a persistent state, it will return to
this state almost surely. If the chain starts at a transient state, there is
a positive probability of never returning to this state. From the
transition diagram we can evaluate the probability fii and therefore
determine whether the state i is persistent or transient.
Example 4
For each of the states 3 and 6 of the Markov chain in
Example 1 determine whether the state is persistent or transient.
Solution
 f66= =0.24 as the probability of the cycle 615.So the
state 6 is persistent.
 =0.9 the probability of the loop around 3
=0
=0.3 =0.009 as the probability of the
cycle 3423.
More generally, for any , =0 and 0.1 as
the probability of the edge sequence 342...423
ntimes
.
So:
f33  
 
Since f11=0.9 < 1, the state 3 is transistent.
Lemma 1
Suppose i and j are two different states of a Markov chain.
If pji > 0 and fij = 0, then the state i is transient. This lemma is
easily derived from the definition of fij. The lemma can be rephrased
in terms of transition diagrams: if the chain can reach state j from
state i in one step (pji > 0) but cannot come back (fij = 0), then the
state i is transient.
Lemma 1 gives a method of finding transient states from a
transition diagram without any calculations. For example, from Fig. 1
we can see that p16 = 0.3 > 0 and f61 = 0 because the chain cannot
return from state 6 to state 1. Therefore by Lemma 1 the state 1 is
transient. This is consistent with the result of Example 4.

125 | P a g e
D. Mean Recurrence Time:
The mean recurrence time of a persistent state i is
defined as
( )
1
m
i ii
m
mf


  .
If i is a transient state, i   by the definition.
Thus is the expected time of returning to the state i if thechain
states at i.
Example 5
For each of the states 3 and 6 of the Markov chain in Example 1 find
its mean recurrence time.
Solution
 Since the state 3 is transient =
 For the state 6, =0.24 and =0 for any n
So =3 =3 0.24=0.72.
E. Periodic States:
The period of a state i is the greatest common divisor of all n ≥ 1
with p(
ii
n)
> 0.
The state i is periodic if its period is greater than 1; otherwise it is
aperiodic.
In terms of transition diagrams, a state i has a period d if every
edge sequence from i to i has the length, which is a multiple of d.
Example 6
For each of the states 3 and 6 of the Markov chain in Example 1 find
its period and determine whether the state is periodic.
Solution
The transition diagram in Fig. 1 has a cycle 343 of length 2
and a cycle 3432 of length 3. The greatest common divisor of 2 and 3
equals 1. Therefore the period of the state 2 equals 1 and the state is
aperiodic.
Any edge sequence from 6 to 6 is a cycle 615 or its
repetition, so its length is a multiple of 3. Hence the state 6 is periodic
with period 3.
F. Communicating States
State i communicates with state j (notation i→j) if p
ji
n
> 0 for some n ≥ 0.
In terms of transition diagrams, a state i communicates with
state j if there is a path from i to j.
State i inter-communicates with state j (notation ij) if the
two states communicate with each other.
Theorem 1
(Grimmett and Stirzaker, 2001)
Suppose i and j are two states of a Markov chain and ij. Then:
 i and j have the same period; 
 i is persistent  j is persistent;
 i is transient  j is transient
Inter-communication is an equivalence relation on the set Q
of all states of a Markov chain. So the set Q can be partitioned into
equivalence classes;all states in one equivalence class share the same
properties, according to Theorem 1.
Example 7
Let us consider the Markov chain from Example 1 and its
transition diagram
Clearly, the states 2 and 4 inter-communicate. Also 2→3, since
p32 > 0 and 3→2, since there is a path 342 from 3 to 2.
Next, 6→1 but not 1→6 (there is no path from 6 to 1). States 1, 5
and 6 all inter-communicate.
Therefore, the equivalence class of 3 is:
[3] = {2,3,4}
and the equivalence class of 6 is:
[6] = {1,5,6}.
According to Theorem 1 and Examples 4 and 5, the states 2, 3 and 4
are all transient and aperiodic; the states 1, 5 and 6 are all persistent
and periodic with period 3.
IV. CONCLUSION
In this paper we have given some elementary definitions for
Markov chains. These definitions are utilized in plotting the transition
diagrams. Finally we have related the two components that are
Markov chains and transition matrix in the form of diagram.
REFERENCES
[1] Chang-Xing, L., 2009. Probe into the teaching of probability
theory and stochastic processes. Proceedings of the International
Conference on Computational Intelligence and Software
Engineering, Dec. 11-13, IEEE Xplore Press, Wuhan, pp: 1-4.
DOI: 10.1109/CISE.2009.5366432.I. S. Jacobs and C. P. Bean,
“Fine particles, thin films and exchange anisotropy,” in Magnetism, vol.
III, G. T. Rado and H. Suhl, Eds. New York: Academic, 1963, pp. 271–
350.
[2] Cinlar, E., 2013. Introduction to Stochastic Processes. 1st Edn.,
Elsevier, ISBN-10: 0486497976, pp: 416.
[3] Favida Kachapova “Representing Markov chains with transition
diagram” Journal of Mathematics and Statistics 9 (3): 149-154, 2013
ISSN: 1549-3644
[4] Grimmett, G. and D. Stirzaker, 2001. Probability and Random
Processes. 3rd Edn., Oxford University Press, New York, ISBN-
10: 0198572220, pp: 596.
[5] Wang, A.L. and S.H. Kon, 2003. Should simple Markov
processes be taught by mathematics teachers? International
Statistical Institute, 54th
Session, Berlin, pp:1-4.
[6] Wang, A.L., 2001b. Introducing Markov chains models to
undergraduates. International Statistical Institute, 53rd Session,
Seoul, pp:1-4.

A STUDY ON MARKOV CHAIN WITH TRANSITION DIAGRAM

More Related Content

What's hot (14)

Viewers also liked (20)

Similar to A STUDY ON MARKOV CHAIN WITH TRANSITION DIAGRAM (20)

More from International Journal of Technical Research & Application (20)

Recently uploaded (20)

A STUDY ON MARKOV CHAIN WITH TRANSITION DIAGRAM