For a finite Markov chain with transition matrix given by: $P=\left[ \matrix{1\over 3 & 2\over 3&0&0 &0 &0 \\ 2\over 3&0&1\over 3&0 &0&0\\0&1\over 3&0&2 \over 3&0&0\\0&0&2\over 3 &0 &1\over 3&0\\0&0&0&1\over 3 &0&2\over 3\\0&0&0&0&2 \over 3&1\over 3}\right]$

I need to find:

  1. the probability of visiting state 6 before visiting state 1, given that my initial state is 2 (so that my initial distribution vector is $\mu_{0}=\left[ \matrix{0 & 1 &0 &0&0&0}\right]$).
  2. The mean number of visits at state 1, until returning to state 6, and assuming $X_0=6$.

My attempts didn't help so much.

As for part one: I tried solving this according to a similar question, by denoting $p_i$ as the probability of visiting state 6 if I'm currently at state $i\in\lbrace1,\dots ,6\rbrace$, but I got a system of linear equations where all $p_i=1$... and generally I had trouble formalizing this question in terms of the random variables of the states.

As for part two: Denoting $t^*_6=\mathbb E\left[T_6|X_0=6\right]$ as the mean recurrence time for state 6, I tried solving the system given by $t_i=1+\sum_{i\neq 6}p_{ij}t_{i}$ but it became a mess. Also since the stationary distribution is $\pi=\left[ \matrix{1\over 6 & \dots & 1\over 6}\right]$, and the chain is aperiodic and irreducable, it seems that $t_6^{8}=6$, but it still doesn't answer the question, as for how many times in average state 1 is visited.

Any help would be appreciated.


Let $p_i$ denote the probability that you reach state $6$ before state $1$ starting from state $i$. Then $p_2=\frac13p_3$ and $p_3=\frac13p_2+\frac23p_4$. By the symmetry of the problem, $p_4=1-p_3$. Substituting the first and third equations into the second yields $p_3=\frac19p_3+\frac23(1-p_3)$, with solution $p_3=\frac37$ and thus $p_2=\frac17$.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.