# Infinitesimal definition of a continuous-time Markov chain

Citing from this Wikipedia article, a continuous-time Markov chain can be described in the following manner

Let $$X_t$$ be the random variable describing the state of the process at time $$t$$, and assume the process is in a state $$i$$ at time $$t$$. Then, knowing $$X_t=i$$, $$X_{t+h}=j$$ is independent of previous values $$\left( X_s : s, and as $$h$$ → 0 for all $$j$$ and for all $$t$$,

$$\Pr(X(t+h) = j \mid X(t) = i) = \delta_{ij} + q_{ij}h + o(h)$$,

where $$\delta_{ij}$$ is the Kronecker delta, using the little-o notation. The $$q_{ij}$$ can be seen as measuring how quickly the transition from $$i$$ to $$j$$ happens.

Things I think I understood

• $$o(h)$$ encloses the probability of the events whose probability is a lower order than that of a single jump (e.g. thinking birth-death processes, two simultaneous births/death)
• the Kronecker is one only when calculating the probability that the process stays in the same state
• $$q_{ij}$$ is the transition rate of the process from $$i$$ to $$j$$.

What I don't understand is how this formula is derived. More specifically, what confuses me is the fact that, correct me if I'm wrong, $$q_{ij}h$$ represents the transition probability from state $$i$$ to $$j$$.

The continuous time Markov chain is characterized by the transition rates, the derivatives with respect to time of the transition probabilities between states $$i$$ and $$j$$.

Does that imply $$q_{ij}(t)=\frac{d}{dt} p_{ij}(t)$$? If yes, how does it correlate to the formula in the infinitesimal definition of the Markov chain?

Let $$p_{i,j}(h) = \Pr(X(t+h)=j \mid X(t)=i).$$ Since we have assumed that the chain is time-homogeneous, this quantity does not depend on $$t$$. This is the transition probability of going from state $$i$$ to state $$j$$ after time $$h$$. Note that trivially $$p_{i,j}(0) = \delta_{i,j}$$.
The transition rate $$q_{i,j}$$ can be defined as the derivative at zero of this function: $$q_{i,j} := \frac{d}{dh} p_{i,j}(h)|_{h=0}$$. Note that one of the equivalent ways of defining the derivative at zero is using the $$o(h)$$ notation: $$q_{i,j}$$ is the only constant you can multiply $$h$$ by which makes the following true $$p_{i,j}(h) = p_{i,j}(0) + q_{i,j}h + o(h).$$
One has to be careful now in that $$q_{i,j}$$ is expressing the transition rate. In general the function $$p_{i,j}(h)$$ will not be linear, so it is not true that $$q_{i,j} h$$ is the transition probability after time $$h$$, which as we said is given by $$p_{i,j}(h)$$. The reason is that during that interval of time, if we had transitioned from $$i$$ to $$j$$ we could as well have transitioned away to some other state, or we could have transitioned first to some third state and then to $$j$$, so $$p_{i,j}(h)$$ could be smaller or larger than $$q_{i,j}h$$.
What it is true is that, if we know all the transition rates $$q_{i,j}$$ for all $$i$$ and $$j$$, we can recover $$p_{i,j}(h)$$. If we now denote by $$Q=(q_{i,j})_{i,j}$$ the matrix of transition rates, and by $$P(h) = (p_{i,j}(h))_{i,j}$$ the time-dependent matrix of transition probabilities, then this is the solution of the first-order differential equation $$P'(h) = QP(h)$$ with initial condition $$P(0) = (\delta_{i,j})_{i,j}$$ is the identity matrix.
• Shouldn't it be $P(h) = (p_{i,j}(h))_{i,j}$? Jan 15, 2020 at 17:55