I'm reading Chapter 11 of Puterman's book on Markov Decision Processes (in particular, about continuous-time Markov processes). There's a lot of notation involved, but I've tried to distill the question. Puterman defines a function $Q(t,j|s,a)$, which, as a simple example, might equal $$ Q(t,j|s,a)=\frac{1}{4}(1-e^{-\mu{t}}) $$ for some $\mu>0$. The function $Q$ is a joint probability distribution in $t\geq0$ and $j\in{S}$ for finite $S$ (in the example above, the product of the CDF of an exponential random variable with a constant). He then writes down the integral $$ \int_0^\infty e^{-\alpha{t}}Q(dt,j|s,a), $$ and asserts that the value of this integral is $<1$. Puterman states "[w]e use $Q(dt,j|s,a)$ to represent a time-differential", but I don't know what this means in the context of integration.

Question 1 What kind of integral is this? Seems like Riemann-Stieltjes or Lebesgue, but I can't tell. I thought it might be strange notation for $$ \int_0^\infty e^{-\alpha{t}}Q(t,j|s,a)dt, $$ but it seems that's not the case (as then the integral can easily be $\geq1$).

Question 2 How do you evaluate such an integral? Is there e.g. a closed-form for the $Q$ defined above?

  • $\begingroup$ The integral would be $$ \int_0^\infty e^{-\alpha t}\frac14 \mu e^{-\mu t}\ \mathsf dt = \frac\mu{4(\alpha+\mu)}. $$ $\endgroup$ – Math1000 Jan 14 at 23:07
  • $\begingroup$ @Math1000 Weird notation. Post as an answer (maybe with a small explanation?) and I’ll accept! $\endgroup$ – David M. Jan 14 at 23:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.