Title may be a bit confusing, allow me to explain:

Say our state space $S$ = $\{1, 2, 3, 4\}$

Now let us say that we observe a Markov Chain for 6 periods, and find that its path is $\{1, 3, 4, 1, 3, 2\}$ (i.e it is in state 1 at period 1, state 3 at period 2, state 4 at period 3, and so on).

My question is how you would construct a transition probabilities matrix for this Markov chain. Plugging this data into a chunk of code I have written yields the following transition probabilities matrix:

\begin{bmatrix} 0 & 0 & 1 & 0\\ NaN & NaN & NaN & NaN\\ 0 & 0.5 & 0 & 0.5\\ 1 & 0 & 0 & 0 \end{bmatrix}

At first glance, given the data, the NaNs in this transition matrix would make sense—since the Markov chain never entered state 2, we have no data that would tell us the probabilities with which it would move to any other state from state 2. However, does this not violate the axiom which states that the rows of a transition matrix must sum to 1? Even though we do not have data that tells us the probabilities with which the Markov chain exits state 2, can we not assume that, whatever those probabilities are, they must sum to 1? As a result, can't we just replace each of the NaNs with 0.25, i.e suggesting that since we have no data that would tell us the probabilities with which the Markov chain will move to another state from state 2, we can just assume that it will enter each state in the state space from state 2 with equal probability. Does this check out mathematically? Or is there a more complicated way to address this? Or should I simply leave the matrix as is?


1 Answer 1


There are two different transition matrices - the actual one $M$, and your estimate of it based on your observations $\hat{M}$. Since you have no data for the second row, it is valid to have nothing in your estimate for it. But, if you want to actually do something with it that doesn't lead to the NaNs propagating everywhere, you can impute values (this is common practice in things like population surveys, where missing data may get imputed to allow analysis to take place over the entire dataset).

You can technically impute any value you like, but choosing to split the probabilities evenly is possibly the safest - it puts a very weak set of assumptions on the true behaviour of the process, which means that over all possible true transition matrices this is the starting point that will probably have the smallest average error in some sense.


You must log in to answer this question.