# Long term probability in Markov Chains

I was practicing some questions on transition probability matrices and I came up with this question.

You have 3 coins: A (Heads probability 0.2),B (Heads probability 0.4), C (Heads probability 0.6).Plan is to toss one of the 3 coins each minute. Start by tossing A. Subsequently if you toss Heads you coin A next minute. If you toss Tails, you chose at random with equal probability one of coins B or C for your next toss.

$X_n:$type of coin tossed on the nth minute.

Then the one steep transition probability matrix is: $$\begin{matrix} 0.2 & 0.4 & 0.4 \\ 0.4 & 0.3 & 0.3 \\ 0.6 & 0.2 & 0.2 \\ \end{matrix}$$

a) Determine the asymptotic proportion of minutes you toss heads.
b) how could you compute the probability that you toss Heads on tosses 3 and 10.
$\pi_A *0.2 +\pi_B *0.4 +\pi_c* 0.6$ where $\pi_A,\pi_B,\pi_C$ are stationary distribution.
But the given answer is $\pi_A$.Can some one please tell me what is wrong with the way I did it.

For b) Is it P(tossing coin A on 4th toss and tossing coin on 11 th toss)=$P_{AA}^4*P_{AA} ^{11}$ ?

• If the answer to a) is pi_A, this might indicate that a) asks for "the asymptotic proportion of minutes you toss" A, not "the asymptotic proportion of minutes you toss heads".
– Did
May 2 '15 at 9:05
• Both answers are OK. Accidentally $\pi_A *0.2 +\pi_B *0.4 +\pi_c* 0.6=\pi_A.$ See, my answer below.
– zoli
May 2 '15 at 13:54
• @Did, Surprisingly (not at all accidentally) the probability of heads equals the stationary probability of the "head" state however biased the coins are.
– zoli
May 2 '15 at 15:39

Determining the stable (long term) state probabilities

The solutions of the following system of equations are the stationary state probabilities:

$$\begin{matrix}[\pi_A\ \pi_B \ \pi_C]\\ \\ \\ \end{matrix} \begin{bmatrix} 0.2 & 0.4 & 0.4 \\ 0.4 & 0.3 & 0.3 \\ 0.6 & 0.2 & 0.2 \\ \end{bmatrix} \begin{matrix}=[\pi_A\ \pi_B \ \pi_C]\\ \\ \\\end{matrix}$$ and $$\pi_A+\pi_B+\pi_C=1 .$$

Note

Here $\pi_A\not=1$. $\pi_A$ is a hypothetical quantity an intuitive definition of which (together with that of $\pi_B$,$\pi_C$) is as follows: "If we started the system choosing a state randomly with probabilities $\pi_A,\pi_B, \pi_C$ then the probability that the next state would be $A$, $B$, or $C$ would be $\pi_A,\ \pi_B, \pi_C$, respectively." Such stationary probabilities do not necessarily exists. You can read about the details here.

Continued

Having performed the vector-matrix multiplication we arrive at a redundant system of equations. After omitting the redundant equation and after some rearrangement we have

$$\begin{matrix} -0.8\pi_A & +& 0.4\pi_B & +& 0.6\pi_C &=0 \\ \ \ 0.4\pi_A & -& 0.7\pi_B& +& 0.2\pi_C&=0 \\ \pi_A&+&\pi_B&+& \pi_C&=1& \end{matrix}$$

The solution, with the help of Alpha, is $$\pi_A=\frac{5}{13}, \ \pi_B=\frac{4}{13},\ \pi_C=\frac{4}{13} .$$

Question a.

We can toss heads in each states, so $$P(\text{heads})=P(\text{heads}|\text{in state} A)\pi_A+P(\text{heads}|\text{in state}B)\pi_B+P(\text{heads}|\text{in state}C)\pi_C=\frac{2}{10}\frac{5}{13}+\frac{4}{10}\frac{4}{13}+\frac{6}{10}\frac{4}{13}=\frac{5}{13}=\pi_A.$$ That is, the result of the tossing is a head in $\frac{5}{13}100$% of the time on the long run.

Remark

It seemed to be accidental that

$$P(\text{heads})=\pi_A.$$

I wondered, if it was? So I took $p_A, p_B, \text{ and }p_C$ as the probabilities of the heads in the states $A, B, \text{ and } C$, respectively.

In order to get the stationary probabilities we have to solve the following system of equations:

$$\begin{matrix} p_A\pi_A & +& p_B\pi_B & +& p_C\pi_C &=\pi_A \\ \frac{1-p_A}{2}\pi_A & +& \frac{1-p_B}{2}\pi_B& +& \frac{1-p_C}{2}\pi_C&=\pi_B \\ \pi_A&+&\pi_B&+& \pi_C&=1& \end{matrix}.$$ The solutions are

$$\begin{matrix} \pi_A=\frac{p_B+p_C}{-2p_A+p_B+p_C+2}\\ \pi_B=\frac{1-p_A}{-2p_A+p_B+p_C+2}\\ \pi_C=\frac{1-p_A}{-2p_A+p_B+p_C+2} \end{matrix}.$$ At this point, note that $p_A<1$. If $p_A$ was $1$ then the Markov chain would never get out of state $A$. Then, of course, $$1=P(\text{heads})=\pi_A.$$

If $p_A<1$ then the solutions above are valid and we may compute the probability of the heads in general. Watch this:

$$P(\text{heads})=\frac{(p_B+p_C)p_A+(1-p_A)p_B+(1-p_A)p_C}{-2p_A+p_B+p_C+2}=\pi_A \ !$$

Perhaps this is what the OOOOOP wanted to see.

Question b.

We have to go back to the initial state which is $A$. Now, the following vector-matrix product will give the probabilities that we end up in states $A$,$B$, or $C$ at the $\text{n}^{th}$ minute:

$$\begin{matrix}[1\ 0 \ 0]\\ \\ \\\end{matrix} \begin{bmatrix} 0.2 & 0.4 & 0.4 \\ 0.4 & 0.3 & 0.3 \\ 0.6 & 0.2 & 0.2 \\ \end{bmatrix}^n\begin{matrix}=[\pi_A^n\ \pi_B^n \ \pi_C^n]\\ \\ \\\end{matrix}$$

Finally, we have to repeat the steps performed during solution a. But this time we work with the results of the vector-matrix product above with $n=3$ and with $n=10$.