0
$\begingroup$

I have got a Markov Chain as below

$\begin{bmatrix}q & p & 0 & 0 &0\\0 & p &q & 0 &0\\0&p&0&q&0\\q&0&0&0&p\\0&0&0&0&1\end{bmatrix}$

I am asked to classify the class and if they are transient. The answer from my lecturer is {0,1,2,3} is a open class so that its transient; and {4} is closed so that its persistent. What i am struggling is, for the class {0,1,2,3}, for example taking state 1, it can get back to state 1 in the next step, otherwise it can go to state 2, then it either back to state 1 or go to state 3, then it either go to state 4 which the matrix will terminate, or it can go to state 0 which can go back to state 1. So i can conclude that if we start in state 1, there are possibilities that it will go back to state 1 so that {0,1,2,3} which include state 1 shouldn't be transient.

Are there any mistake in my argument or i have mess up the definition of a transient state?

Many thanks

$\endgroup$
1
$\begingroup$

A state is transient if there's a nonzero proability that, starting there, that state is never visited again. State 1 in your chain is transient, because from state 1 you can go (with probability $q$) to state 2, then (with probability $q$) to state 3, then (with probability $p$) to state 4, after which you will never return to state 1; so if you start at 1 the probability of never returning to 1 is at least $pq^2$ which is bigger than 0.

I think you may be thinking that "transient" means "nonzero probability of returning", but that's not right: "recurrent" means "zero probability of not returning" and "transient" means "nonzero probability of not returning" or "not-1 probability of returning".

| cite | improve this answer | |
$\endgroup$
0
$\begingroup$

For a finite state Markov chain, you can think in the following way:

1) If it is possible to reach $y$ from $x$ and it is possible to reach $x$ from $y$, then $x$ and $y$ are in the same group. From your line of reasoning, $0,1,2$ and $3$ are all inthe same group. Note that, although it is possible to reach $4$ from $3$, it is not possible to reach $3$ from $4$. This is why $4$ is not in the same group. As a result, you have the groups $\{0,1,2,3\}$ and $\{4\}$.

2) If there exists $x_1$ in a group $G_1$ and $x_2$ in a group $G_2$ such that it is possible to reach $x_2$ from $x_1$, then $G_1$ is transient. For example, it is possible to reach $4$ from $3$. Therefore, $\{0,1,2,3\}$ is transient.

3) If $G$ is a group and, for every state $x$ in $G$, it is only possible to reach a state in $G$ from x, then $G$ is recurrent. Since it is only possible to reach 4 from 4, $\{4\}$ is recurrent.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.