5
$\begingroup$

A Markov chain has states $0,1,2$ with transition probabilities $$P=\begin{pmatrix} 0.8 & 0.1 & 0.1 \\ 0.3 & 0.5 & 0.2 \\ 0.2 & 0.4 & 0.4 \end{pmatrix}.$$

I struggle to calculate the mean return time to state 1 given that we start at state 1.

Define $T=\min{\{n\geq1 :X_n=1\}}$ and $g_i=E(T|X_0=i)$. I seek $g_1$. By total probability and Markov property $$g_i=\sum_{j\in {1,2,3}}E(T|X_0=i,X_1=j)P(X_0=i,X_1=j))) =\sum_{j\in {1,2,3}} E(T+1|X_0=j)P(X_0=i,X_1=j))=\sum_{j\in {1,2,3}}g_ip_{ij}+1$$ So I get the system of equations \begin{cases} g_0 = 0.8g_0+0.1g_1+0.1g_2+1 \\ g_1 = 0.3g_0+0.5g_1+0.2g_2+1 \\ g_2 = 0.2g_0+0.4g_1+0.4g_2+1 \\ \end{cases} which has no solution. What is wrong with my equations?

$\endgroup$

1 Answer 1

7
$\begingroup$

For all states $j \ne 1$, if you enter state $j$, you have on average $g_j$ more steps before you get to state $1$. However, if you enter state $1$, you don't have on average $g_1$ more steps before you get to state $1$: if you enter state $1$, you're done!

In other words, $E(T \mid X_0 = i, X_1 = 1)$ should just simplify to $1$, unlike every other instance of $E(T \mid X_0 = i, X_1 = j)$, because if we know that $X_1 = 1$, we know that $T=1$.

So your system of equations should instead be: \begin{cases} g_0=0.8g_0+0.1\color{red}{\cdot 0}+0.1g_2+1 = 0.8g_0 + 0.1g_2 + 1\\ g_1=0.3g_0+0.5\color{red}{\cdot 0}+0.2g_2+1 = 0.3g_0 + 0.2g_2 + 1\\ g_2=0.2g_0+0.4\color{red}{\cdot 0}+0.4g_2+1 = 0.2g_0 + 0.4g_2 + 1\\ \end{cases}

$\endgroup$
5
  • $\begingroup$ How does the 1 become a 0? $\endgroup$
    – user30523
    Mar 4, 2018 at 20:28
  • 2
    $\begingroup$ It gets multiplied by the transition probability and becomes part of the $+1$ at the end. $\endgroup$ Mar 4, 2018 at 20:46
  • $\begingroup$ @MishaLavrov, let say we have found the systems' solution. What is a unit of mean return time here? $\endgroup$
    – Nick
    Jan 7, 2020 at 11:49
  • 1
    $\begingroup$ The units are in number of steps. (So, if you are in state $1$, it takes $g_1$ steps in expectation to return to state $1$ again.) $\endgroup$ Jan 7, 2020 at 15:18
  • $\begingroup$ It may help someone who couldn't understand the expressions at first like me, if we rewrite the expression of $g_0$ into $g_0=0.8(g_0+1)+0.1\cdot 1+0.1(g_2+1)= 0.8g_0 + 0.1g_2 + 1$. Similar rewriting for the expressions of $g_1, g_2$. $\endgroup$
    – Sam Wong
    Jul 20, 2023 at 4:05

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .