# In a continuous-time Markov process, is the waiting time between jumps a function of the current state?

Two books construct Markov processes from Q-matrices using waiting times and jump chains but differ in whether the waiting times depend on the current state. Can the two be reconciled?

Klenke claims that to every Q-matrix $q$ corresponds a unique Markov process that is differentiable at time $t=0$ and such that $q$ is its derivative there (Theorem 17.25). According to him, this unique process can be constructed by combining a (discrete time) Markov chain representing "jumps" between states, and a Poisson process representing the time lapse between jumps (a.k.a. the waiting time). According to this model, the waiting time $\sim \exp(\lambda)$, where $\lambda$ is a function of $Q$, and therefore unaffected by the current state.

Norris too constructs a Markov process from a Q-matrix (section 2.6). His construction is similar to Klenke's except that $\lambda$ is a function not only of $Q$, but of the state active at the beginning of the waiting time.

Can the two constructions be reconciled? What am i missing here? Is one of the authors wrong or idiosyncratic in his definition of a Markov process?

Klenke

Theorem 17.25 Let $q$ be an $E\times E$ matrix ($E$ being a countable set) such that $q(x,y)\geq0$ for all $x,y\in E$ with $x\neq y$. Assume that the following hold

i) $q(x,y)\geq0$ for all $x,y\in E$,

ii) $q(x,x)=-\sum_{y\neq x}q(x,y)$

iii) $\sup_{x\in E}|q(x,x)|<\infty$

Then $Q$ is the $Q$-matrix of a unique Markov process $X$.

Proof [Abridged] Let $I$ be the unit matrix on $E$. Define $$p(x,y)=\frac{1}{\lambda}q(x,y)+I(x,y)\space\space\mathrm{for\, }x,y\in E$$

Then $p$ is a stochastic matrix and $q=\lambda(p-I)$. Let $\left((Y_n)_{n\in\mathbb{N}_0}, (\mathrm{P}_x^Y)_{x\in E}\right)$ be a discrete Markov chain with transition matrix $p$ and let $\left((T_t)_{t\geq0}, (\mathrm{P}_n^T)_{n\in \mathbb{N}_0}\right)$ be a Poisson process with rate $\lambda$. Let $X_t:=Y_{T_t}$ and $\mathrm{P}_x=\mathrm{P}_x^Y\otimes\mathrm{P}_0^T$. The $\mathfrak{X}:=\left((X_t)_{t\geq0}, (\mathrm{P}_x)_{x\in E}\right)$ is [...] the required Markov process. [...] $\square$

Norris

A minimal right-continuous process $(X_t)_{t\geq0}$ on $I$ is a Markov chain with initial distribution $\lambda$ and generator matrix $Q$ if its jump chain $(Y_n)_{n\geq0}$ is discrete-time Markov($\lambda$, $\Pi$) and if for each $n\geq1$, conditional on $Y_0, \dots, Y_{n-1}$, its holding times $S_1, \dots, S_n$ are independent exponential random variables of parameters $q(Y_0), \dots, q(Y_{n-1})$ respectively. We say $(X_t)_{t\geq0}$ is Markov($\lambda$, $Q$) for short. We can construct such a process as follows: let $(Y_n)_{n\geq0}$ be discrete-time Markov($\lambda$, $\Pi$) and let $T_1,T_2,\dots$ be independent exponential random variables of parameter $1$, independent of $(Y_n)_{n\geq0}$. Set $S_n=T_n/q(Y_{n-1})$, $J_n=S_1+\cdots+S_n$ and $$X_t=\begin{cases}Y_n &\mathrm{if\, }J_n\leq t<J_{n+1}\mathrm{\, for\, some\, }n\\ \infty &\mathrm{otherwise}\end{cases}$$

Then $(X_t)_{t\geq0}$ has the required properties.

-

The two constructions are equivalent and their equivalence is based on the so-called thinning of Poisson processes.

Klenke starts from a homogenous Poisson process with a large rate $\lambda$. Amongst the times of this process, when at $x$, a relative proportion $1+q(x,x)/\lambda$ is used to jump from $x$ to $x$ and, for every $y\ne x$, a relative proportion $q(x,y)/\lambda$ is used to jump from $x$ to $y$. The jumps $x\to x$ have no effect, hence one is left with a proportion $q(x,y)/\lambda$ of jumps $x\to y$ amongst a global population of potential jump times with density $\lambda$, that is, the correct rate $q(x,y)$. The only condition for this construction to work is $1+q(x,x)/\lambda\geqslant0$ for every $x$, that is, $\lambda\geqslant\sup\limits_x[-q(x,x)]$, hence one can choose, as many authors do, $\lambda=\sup\limits_x[-q(x,x)]$ but any larger value of $\lambda$ will do as well.

Norris's construction might be more usual hence I will not comment on it here, except to note that $\lambda$, the initial distribution in Norris, is related in no way whatsoever to $\lambda$, the positive real number in Klenke. (My impression is that Klenke's version, more elegant, is slowly replacing the other one in the probabilists' minds.)

Edit The piece of Norris's construction missing from your account is that $\Pi$ is related to $Q$ through $\Pi(x,x)=0$ for every $x$, and, for every $y\ne x$, $$\Pi(x,y)=\frac{q(x,y)}{q(x)}\quad\text{with}\quad q(x)=-q(x,x)=\sum_{z\ne x}q(x,z).$$

-
Let me suggest that you stop waving in the air hypotheses about one of the authors being wrong or idiosyncratic or whatever, when, repeatedly, what is at stake is your understanding of otherwise perfectly correct texts. In the case at hand, the quite different nature of the object "lambda" in these expositions might have been a hint. –  Did Sep 16 '12 at 9:03
Thank you very much. I'm sorry my style has offended you. It was by no means my intension to slight anyone, neither you nor the esteemed textbook authors. I'll try to phrase my questions more carefully in the future. Could you please give me a hint as to how i can go about proving formally that the two constructions are equivalent? Alternatively, if you could recommend a book/article that carries out this project. Norris, as an example, proves the equivalence of his construction to two others, neither of which resembles Klenke's. –  Evan Aad Sep 16 '12 at 11:41
See Edit. (Note that I am not the slightest offended, simply I suggest a way for you to behave more in agreement with some common rules of courtesy. An additional advantage, according to me, would be to stop looking a priori for (often nonexistent) flaws in the texts you read, and instead, to start to try to understand them.) –  Did Sep 16 '12 at 11:58
Thank you. Is Norris's $\Pi$ the same as Klenke's $p$? Is Norris's jump chain $(Y_n)_{n\geq0}$ the same as Klenke's $(Y_n)_{n\in\mathbb{N}_0}$? –  Evan Aad Sep 16 '12 at 12:12
Both answers are "Obviously, no". Just read. –  Did Sep 16 '12 at 12:17