Take the 2-minute tour ×
Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. It's 100% free, no registration required.

I just started learning about Markov processes and got the following homework question.

Classify all the states as recurrent or transient for the Markov chain below

$$\begin{matrix} {}&s_1&s_2&s_3&s_4&s_5\cr s_1&1/4 & 3/4 & 0 & 0 & 0\cr s_2&1/2 & 1/2 & 0 & 0 & 0\cr s_3&0 & 0 & 1 & 0 & 0\cr s_4&0 & 0 & 1/3 & 2/3 & 0\cr s_5&1 & 0 & 0 & 0 & 0 \end{matrix}$$

I worked through the first few powers of the matrix and found the following:

For n=2 $$\begin{matrix} {}&s_1&s_2&s_3&s_4&s_5\cr s_1&7/16 & 9/16 & 0 & 0 & 0\cr s_2&3/8 & 5/8 & 0 & 0 & 0\cr s_3&0 & 0 & 1 & 0 & 0\cr s_4&0 & 0 & 5/9 & 4/9 & 0\cr s_5&1/4 & 3/4 & 0 & 0 & 0 \end{matrix}$$

For n=3 $$\begin{matrix} {}&s_1&s_2&s_3&s_4&s_5\cr s_1&25/64 & 39/64 & 0 & 0 & 0\cr s_2&13/32 & 19/32 & 0 & 0 & 0\cr s_3&0 & 0 & 1 & 0 & 0\cr s_4&0 & 0 & 19/27 & 8/27 & 0\cr s_5&7/16 & 9/16 & 0 & 0 & 0 \end{matrix}$$

From the diagonals, I can figure out that state 3 is recurrent because $$\Sigma p_{33}(n)=\infty$$

By the same reasoning, states 4 and 5 are transient because $$\Sigma p_{44}(n)$$ and $$\Sigma p_{55}(n)$$ are convergent.

First of all, is this method of finding whether the states are recurrent or transient correct? Is there a simpler way to solve the problem? Second, I can't figure out whether states 1 and 2 are transient or recurrent using this method.

Would appreciate if someone can point me in the right direction. Many thanks!

share|improve this question
6  
Drawing a diagram of the Markov chain with vertices for the states and arrows between states x and y if p(x,y) is positive would yield the answer instantaneously. –  Did Jun 28 '11 at 21:07
    
Look at Didier Piau's comment. It fully answers the question. –  Michael Hardy Jun 28 '11 at 21:34

1 Answer 1

Drawing a diagram of the Markov chain with vertices for the states and arrows between states x and y if and only if p(x,y) is positive yields the answer instantaneously.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.