Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I understand that a Markov chain involves a system which can be in one of a finite number of discrete states, with a probability of going from each state to another, and for emitting a signal.

Thus,an $N \times N$ transition matrix and an $N \times N$ emission matrix of real numbers adequately describe a Markov chain with $N$ states and $M$ emissions.

Is it possible to have a Markov chain with an infinite number of states? For example, if $N=2$ is a LED that can glow blue or red, $N=\infty$ would be a LED which can glow a color that is any mixture of blue or red.

Can't an infinitely-large matrix be represented by a function of two variables (the two indices)?

share|cite|improve this question
Is the set of states countably infinite, or uncountable? – Rod Carvalho Sep 12 '12 at 18:35

If the state space $S$ is countably infinite, finite sums are replaced by sums of convergent series and square matrices of finite size by arrays indexed by $S\times S$, but many formulas are unchanged, at least formally.

Here is a famous example of a Markov chain on $[0,1]$. Let $(U_n)_{n\geqslant1}$ and $(V_n)_{n\geqslant1}$ denote two independent i.i.d. sequences with $U_n$ uniform on $[0,1]$ and $V_n$ uniform on $\{0,1\}$. Let $X_0$ denote any random variable with values in $[0,1]$, independent of $(U_n)_{n\geqslant1}$ and $(V_n)_{n\geqslant1}$.

Define $(X_n)_{n\geqslant0}$ recursively by $X_{n+1}=U_nX_n+(1-U_n)V_n$, for every $n\geqslant0$. Then $(X_n)_{n\geqslant0}$ is a Markov chain on $[0,1]$, whose stationary distribution can be computed explicitly. In words, $V_n$ describes the decision to choose the interval $[0,X_n]$ if $V_n=0$ or $[X_n,1]$ if $V_n=1$, then $X_{n+1}$ is uniformly distributed in this interval.

share|cite|improve this answer

Yes, Markov processes with infinitely many states are indeed considered. Random walks are a common example. The term "Markov chain" is often reserved for the case of a discrete state space. If the state space is finite, it's a "finite Markov chain". See e.g.

share|cite|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.