# What would be the value of $P$ in this example?

A Markov Chain $$(X_n)_n$$ has the following transition matrix:
$$P = \begin{bmatrix} 0.1 & 0.3 & 0.6\\ 0 & 0.4 & 0.6\\ 0.3&0.2&0.5 \end{bmatrix}$$ with initial distribution $$\alpha = (0.2, 0.3, 0.5)$$.

Find the following:
(a) $$P(X_7 = 3|X_6 = 2)$$
(b) $$P(X_9 = 2|X_1 = 2, X5 = 1, X_7 = 3)$$
(c) $$P(X_0 = 3|X_1 = 1)$$

What I understand according to the Example-$$7.16$$ of book by Symour Lipchuz (Page-$$132$$) that, $$P^n$$ is used to find the probabilities of change of states in exactly $$n$$ steps, and $$p^{(m)} \cdot P^n$$ is used to find the probability distribution of various states after $$m$$ steps.

According to my understanding, there are no uses of Initial Distribution in these three computations. Because, multiplication of a $$1X3$$ and $$3X3$$ matrices will give a $$1X3$$ matrix. In that case, the transition matrix won't make sense for three states.

So, my attempted solution is the following:

What is the probability that the system changes from state-$$2$$ to state-$$3$$ in exactly $$1$$ step?

So, the answer would be $$p(2, 3) = 0.6$$

What is the probability that the system changes from state-$$3$$ to state-$$2$$ in exactly $$2$$ steps?

$$P^2 = P \cdot P = \begin{bmatrix} 0.1 & 0.3 & 0.6\\ 0 & 0.4 & 0.6\\ 0.3&0.2&0.5 \end{bmatrix} \cdot \begin{bmatrix} 0.1 & 0.3 & 0.6\\ 0 & 0.4 & 0.6\\ 0.3&0.2&0.5 \end{bmatrix} = \begin{bmatrix} 0.19&0.27&0.54\\ 0.18&0.28&0.54\\ 0.18&0.27&0.55 \end{bmatrix}$$

So, the answer would be: $$p(3, 2) = 0.27$$

(c) I think, this question asks:

What is the probability that the system changes from state-$$1$$ to state-$$3$$ in exactly $$9$$ steps?

Is it? If yes, then it would be $$p(1,3)$$ from $$P^9$$.

Am I correct?

Edit:

$$P(X_0 = 3|X_1 = 1) = \frac{P(X_0 = 3, X_1 = 1)}{P(X_1 = 1)}$$
$$\Rightarrow P(X_0 = 3|X_1 = 1) = \frac{P(X_1 = 1, X_0 = 3)}{P(X_1 = 1)}$$
$$\Rightarrow P(X_0 = 3|X_1 = 1) = \frac{P(X_1 = 1| X_0 = 3)\cdot P(X_0=3)}{P(X_1 = 1)}$$

Now,

• from $$P$$, we have $$P(X_1=1|X_0=3) = 0.30$$
• from $$\alpha$$, we have $$P(X_0=3) = 0.50$$, and
• from $$\alpha P$$, we have $$P(X_1=1) = 0.17$$

So,

$$P(X_0 = 3|X_1 = 1) = \frac{0.30 \cdot 0.50}{0.17} \approx 0.88$$.

Is this a correct calculation?

• At the top, you have for (c) $P(X_0=3 \mid X_1=1)$, but later you talk about 9 steps. Presumably this means that you’re thinking about $X_9$ (in which case it’s really eight steps) or $X_{10}$. Which is it? – amd Mar 10 '19 at 23:17
• No, they’re not ”changing in reverse order.” The process only goes in one direction. It’s asking you about the probability that the system started in state 3 given that it’s in state 1 after a single step. Why then are you asking about nine steps of the process at the end of your question? – amd Mar 10 '19 at 23:20
• $X_0$ is the state at time $t=0$; $X_9$ is the state at time $t=9$. You can’t go back to $0$ from $9$. – amd Mar 10 '19 at 23:31
• It means exactly what I wrote in a previous comment. You’re being asked about the probability of a specific history of the system. – amd Mar 10 '19 at 23:39

You have part (a) and part (b) down. For part (c), we need that initial distribution.

What is the probability that the system is in state 1 after one step?
What is the probability that the system starts in state 3 and is then in state 1 after 1 step?
Can you use these to find the conditional probability this part asks for?

And no, there's no cyclic behavior here. The sequence of $$X_n$$ keeps going for all positive integers $$n$$, and it's not periodic.

[In response to the added material in the question]
Yes, that's a correct calculation. You have now fully solved the problem.

• Considering that your claimed answers imply the probability of starting in state 3 and then going to state 1 is less than the probability of going to state 1 regardless of where we started ... no, you've got that part wrong. – jmerry Mar 10 '19 at 23:36
• I'm not sure what you were trying to say there. Now, what I was trying to say with the bit that had the word "regardless"? The probability of starting in state 3 and then going to state 1 after one step must be $\le$ the probability of being in state 1 after one step. – jmerry Mar 11 '19 at 0:03