1
$\begingroup$

If P is a $3\times 3$ transition matrix. Every state has a chance of going to every other state including itself. Therefore this is not an absorbing markov chain. What I want to be able to calculate is the probability of me being in a particular state. I know how to caluclate it after n steps, but surely this can keep changing for higher values of n?

Thanks

$\endgroup$
3
  • $\begingroup$ It does keep changing, but there is the concept of stationary distribution, which represents the long-term "proportion" of time that your process spends in each state. $\endgroup$
    – user42268
    Feb 29, 2016 at 20:01
  • $\begingroup$ Thank you, but how can I calculate that from matrix P? $\endgroup$ Feb 29, 2016 at 20:29
  • $\begingroup$ Are you sure that you want to calculate the probably at step $n$, rather than the asymptotic probability (the probability at step $\infty$) $\endgroup$
    – DanielV
    Mar 2, 2016 at 4:59

1 Answer 1

1
$\begingroup$

Def:

Let $\mathbf{P}$ be the transition probability matrix of a homogeneous Markov chain. If there exists a probability vector $\mathbf{\hat{p}}$ such that

$$\mathbf{\hat{p}}\mathbf{P} = \mathbf{\hat{p}}$$

then $\mathbf{\hat{p}}$ is called the stationary distribution for the Markov chain.

A stationary distribution $\mathbf{\hat{p}}$ is a (left) eigenvector of $\mathbf{P}$ with eigenvalue $1$. Note that any nonzero multiple of $\mathbf{\hat{p}}$ is also an eigenvector of $\mathbf{P}$ but the stationary distribution $\mathbf{\hat{p}}$ is fixed by being a probability vector; that is, its components sum to one.


When you want to find the stationary distribution $\mathbf{\hat{p}}$ of your Markov chain with the $3 \times 3$ probability matrix you can solve the system of equations

$$\mathbf{\hat{p}}\mathbf{P} = \mathbf{\hat{p}}$$

with $\mathbf{P}$ = $ \begin{bmatrix} a & b & c \\ d & e & f \\ g & h & i \\ \end{bmatrix} $

you get

$ap_1 + dp_2 + gp_3 = p_1$

$bp_1 + ep_2 + hp_3 = p_2$

$cp_1 + fp_2 + ip_3 = p_3$

And as $\mathbf{\hat{p}}$ is a probability vector it must also fulfill $p_1 + p_2 +p_3 = 1$.

Solving this you obtain the stationary distribution:

$\mathbf{\hat{p}}= [p_1 \:\: p_2 \:\: p_3]$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.