I've gotten stuck on this problem. Here's my transition matrix:

$$P=\begin{bmatrix} 0.4 & 0.1 & 0.5 \\ 0.4 & 0 & 0.6 \\ 0.6 & 0.4 & 0\end{bmatrix}$$ with state space $S$ = {1,2,3}.

I've actually been asked two questions here is the first one:

When starting in state 1, find the probability that state 3 is visited before state 2.

I realise that this question is similar to the one asked here, Markov Chain Reach One State Before Another, however as my transition matrix is not diagonisable I cannot use the method described in the answer. If anyone could provide a solution to this problem or an alternative method it would be massively appreciated.

The second problem is as follows:

When starting in state 1, find the probability that state 2 is never visited.

I don't really have any idea how to go ahead with this problem other that than I'd assume it will use a similar method to the first problem. Again I'd really appreciate any help with a method for solving this problem or a solution.

  • 1
    $\begingroup$ Your transition matrix is in fact diagonalizable, but the method that you reference is kind of overkill for this problem, anyway. You only have three states to consider, so try writing down the probabilities of entering the two “terminal” states in exactly one step, exactly two, and so on, and see where that leads. $\endgroup$ – amd Jan 16 '18 at 19:30
  • $\begingroup$ OK by finding the probability of reaching state 3 without visiting state 2 in 1,2,3,4,... steps and I found that it was a geometric series so I found the probability that we reach state 3 before state 2 to be $\frac {5}{6}$ is this correct? $\endgroup$ – B.K97 Jan 16 '18 at 19:52
  • $\begingroup$ Sounds right to me. You should write your work up as an answer once SE lets you. $\endgroup$ – amd Jan 16 '18 at 19:57
  • $\begingroup$ Yeah I will but I don't think I'm able to yet. Have you any tips for the second problem? $\endgroup$ – B.K97 Jan 16 '18 at 20:06
  • $\begingroup$ Never visited at all? The probability is zero: state 2 is reachable from every other state. For a finite number of steps, and to verify this, you can attack it the same way you did the first problem. $\endgroup$ – amd Jan 16 '18 at 20:12

Starting in state $1$, you may stay in that state for some number of steps, but with probability $1$ you will eventually leave. The first step that doesn't stay in state $1$ will decide the question of which of states $2$ and $3$ get visited first. The conditional probability of going to state $3$ from state $1$ in a step, given that you are not staying in state $1$, is $0.5/(0.1+0.5) = 5/6$, so that is the probability that state $3$ is visited before state $2$.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.