Consider a continuous time Markov process $\{X(t)\}_{t≥0}$ on the state space $\{0, 1, 2, . . .\}$ with stationary probabilities $\{π_0, π_1, π_2, . . .\}$. Suppose that, when currently in state $i$, the process will jump to state $j$ after an exponential amount of time with rate $q_{ij}$ and that all exponential times are independent. Assume that $X(0) = 0.$

(a) Let ν be the rate of departure from state $0$. Write ν in terms of the $q_{ij}$ .

(b) Let $Y$ be the time of the first exit from state zero. Find the distribution of $Y$.

(c) Starting from $0$, let R be the time of the first return to state zero. What is E[R]?

(d) Let T be the first time that the process has been in state $0$ for at least τ units (continuous) of time. Show that $$ E[T] = \dfrac{1}{\pi_0 \nu}(e^{\nu \tau}-1) $$ (Hint: Condition on the time of the first exit from state $0$.)

I can gather easily the answers for $a-c$ :

(a) $\nu = \Sigma_{i=1}^\infty q_{0i}$

(b) $Y $ ~ $ \text{Exp}(\nu)$

(c) $E[R]=\dfrac{1}{\nu \pi_0}$

However, I'm having trouble with part (d), as I believe every time we are at state 0, then we should be there for longer than $\tau$ units with probability $e^{-\nu \tau}$ since $P(Y>\tau)=e^{-\nu \tau}$, so it seems like T ~ Geometric($e^{-\nu \tau}$), in which case $E[T]=e^{\nu \tau}$, which is clearly not the answer given. I am also having trouble understanding how to understand the hint. What is the flaw in my thinking? Thank you


Indeed, one should condition on $Y$. If $Y\geqslant\tau$, then $T=\tau$. Otherwise, $T=Y+S+T'$ where $S$ denotes the time needed to come back to $0$ after one leaves $0$ and $T'$ is distributed as $T$. Thus, $$E(T)=\tau P(Y\geqslant\tau)+E(Y;Y\lt\tau)+E(S;Y\lt\tau)+E(T';Y\lt\tau).$$ By independence, $E(S;Y\lt\tau)=E(S)P(Y\lt\tau)$ and $E(T';Y\lt\tau)=E(T')P(Y\lt\tau)$. By definition of $T'$, $E(T')=E(T)$. By definition of $S$, $R=Y+S$ hence $E(S)=E(R)-E(Y)$. Thus, $$E(T)=\tau P(Y\geqslant\tau)+E(Y;Y\lt\tau)+E(R)P(Y\lt\tau)-E(Y)P(Y\lt\tau)+E(T)P(Y\lt\tau).$$ For every exponentially distributed $Y$, one has $$\tau P(Y\geqslant\tau)+E(Y;Y\lt\tau)=E(Y)P(Y\lt\tau),$$ hence one gets $$E(T)=\frac{E(R)P(Y\lt\tau)}{P(Y\geqslant\tau)}.$$ Using once again the distribution of $Y$, and the identity $$E(R)=\frac1{\nu\pi_0},$$ one gets the formula given as the solution to (d). (This solution uses the answers to (b) and (c). Note that the answer to (c) in the post lacks a factor $\nu$, which have been added since this answer was posted.)

  • $\begingroup$ Thank you for the thorough answer, and correcting my typo! I see that T was continuous time, not trials until T exceeded $\tau$. $\endgroup$ – NaiveHalmos Dec 29 '14 at 20:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.