# Calculating probability in a Markov Chain

Suppose I have this Markov chain:

And suppose that:

$P_{AA} = 0.70$

$P_{AB} = 0.30$

$P_{BA} = 0.50$

$P_{BB} = 0.50$

I realize that $P_{AA} + P_{AB} = P_{BA} + P_{BB}$ but when I simulate I'm getting some percentage of A's and B's. I'm trying to calculate the probability (i.e. the predicted percentage of A's and B's) to check this data, so I want to calculate $P(A)$ and $P(B)$.

My stochastics is very rusty, so I'm sure someone with a basic understanding of probability and stochastics would know how to calculate this.

EDIT:

$\pi_A = 0.7 * \pi_A + 0.5 * \pi_B$

$\pi_A - 0.7 * \pi_A = 0.5 * \pi_B$

$\pi_A = \frac{0.5}{0.3} * \pi_B$

$\frac{0.5}{0.3} \pi_B + \pi_B = 1 \hspace{10 mm} since \hspace{5 mm} \pi_A + \pi_B = 1$

$\pi_B = 0.375 = \frac{3}{8}$

$\pi_A = 0.625 = 0.7*0.625 + 0.5*0.375$

$\pi_B = 0.375 = 0.3*0.625 + 0.5*0.375$

These helped a lot in solving this problem:

http://www.haverford.edu/econ/econ365/Note%20on%20Markov%20Chains.pdf

http://www.mast.queensu.ca/~stat455/lecturenotes/set3.pdf

-
what are you trying calculate? I don't understand... are you trying to calcuate the ratio of times the Markov chain stays at A and B? –  Lost1 Dec 30 '13 at 21:28
I'm trying to check on the percentages I'm getting. So, yes the ratio. The percent probability for each in $\{A,B\}$ –  stackuser Dec 30 '13 at 21:33

## 3 Answers

This is called the stationary dstribution and solves $\pi=\pi P$. Thus, $\pi_A=\pi_AP_{AA}+\pi_BP_{BA}$ and $\pi_B=\pi_AP_{AB}+\pi_BP_{BB}$, which, in your case, yields $\pi_A=\frac58$ and $\pi_B=\frac38$.

-
+1 that makes sense to my data. I had very little stochastics training, and am trying to understand $\pi_A=\pi_AP_{AA}+\pi_BP_{BA}$. I see the $P_{AA}$ and $P_{BA}$ but I'm not sure where you're getting $\pi_A$ and $\pi_B$ from, since it's on both the right and left sides of the equation. I took a beginning stochastics class a long time ago, but didn't go anywhere near this depth, nothing about Markov Chains, so I'm teaching it to myself now. –  stackuser Dec 30 '13 at 22:52
Yes the pies are on both sides of the equations but the equations yield 3pi_A=5pi_B, which, together with the normalization pi_A+pi_B=1, determines pi_A and pi_B. –  Did Dec 30 '13 at 22:57
Just want to make sure I understand you. Does my edit (with solution) look like what you're saying or is this not right? –  stackuser Dec 31 '13 at 0:06
Do you seriously think that I would approve any solution leading to a 3:2 ratio when my answer explains that the correct ratio is 5:3? By the way, $\pi_A$ and $\pi_B$ are the $P(A)$ and $P(B)$ you are asking about hence I fail to understand the computations in your edit. There is a theory behind all this, you know, and it tells you to solve $\pi=\pi P$. Did you consult the book by Norris (available online) that @Lost1 suggested? –  Did Dec 31 '13 at 10:07

This is really basic Markov Chain Theory. You should refer to a standard textbook such James Norris's Markov Chains. I am going to refer your state A as state 1, and state B as state 2.

Let $P$ denote your transition matrix.

$P^n_{ij}$ gives $P(X_n=j|X_0=i)$

As $n\rightarrow\infty$, $P_{ij}\rightarrow\pi_j$, which is known as the stationary distribution. I think your simulation worked out $\pi_j$. This means, if your Markov Chain runs for a long time, your current location depend less and less on your initial position.

To solve this, you need to solve the simultaneous equation $\pi P = \pi$, where $\pi = (\pi_1,\pi_2)$.

-
@stackuser Did and I take very similar views in how much we should go in term of giving help. I have given you the theory. Did has already given you both the equation and the answers. He has already told you exactly which equations to solve! Why don't you just solve them? It is just 2 simultaneous equations! What you written down completely ignored what both of us wrote... I already think Did, uncharacteristically, spoon-fed you the answer. Do you need us to chew it before you swallow too? –  Lost1 Dec 31 '13 at 22:12

I'm learning Markov Chains from the ground up, so I needed a step-by-step solution. The formulas and theory were easy to find online but that's not why I posted and I specified that in the original post. So here is how I went about solving it, and it would be great if this helps others who might have gotten a little lost or confused at the early stages of the problem (and if not then you're doing better than me).

$\pi_A = 0.7 * \pi_A + 0.5 * \pi_B$

$\pi_A - 0.7 * \pi_A = 0.5 * \pi_B$

$\pi_A = \frac{0.5}{0.3} * \pi_B$

$\frac{0.5}{0.3} \pi_B + \pi_B = 1 \hspace{10 mm} since \hspace{5 mm} \pi_A + \pi_B = 1$

$\pi_B = 0.375 = \frac{3}{8}$

$\pi_A = 0.625 = 0.7*0.625 + 0.5*0.375$

$\pi_B = 0.375 = 0.3*0.625 + 0.5*0.375$

These helped a lot in solving this problem:

http://www.haverford.edu/econ/econ365/Note%20on%20Markov%20Chains.pdf

http://www.mast.queensu.ca/~stat455/lecturenotes/set3.pdf

-