Let $(X_n)$ be a Markov chain on the probability space $(\Omega,\mathcal{F}, \mathbb{P})$ with state space $S$ and let $T$ be a stopping time.

I need to calculate $\mathbb{E}_x (T)$, for $x\in S$, but actually I'm not able to find a clear definition of it.

What I've understood so far is that it should be something like $\mathbb{E}[T|\mathcal{F}_0]$ (evalueted in $x$, in some sense), where $\{\mathcal{F}_n\}$ is the standard filtration relative to $(X_n)$, but I am pretty confused about domains of definition and measurability of the functions appearing here.

Can anyone give me a clear definition of this object?


It's often better to think of a Markov chain without any particular initial distribution, in which case your chain will simply be defined by its kernel $\{P(x,\cdot)\}_{x\in S}$ (or $\{P_{n,n+1}(x,\cdot)\}_{x\in S,n\in\mathbb N}$ in the inhomogeneous case). If $\mathbb P^x$ is the probability measure corresponding to starting the chain at $x\in S$, then

$$\mathbb P^\mu(\cdot):=\int\mathbb P^x(\cdot)\mu(dx)$$

is the measure corresponding to the chain with initial distribution $\mu$. If your space $S$ is sufficiently nice, e.g. if it is a Polish space, then it is possible to go the other way given some particular initial distribution. However, in that case you need some more advanced machinery such as regular conditional probabilities or disintegration of measures. This way is easier.

Now, the definition of $\mathbb E^x[T]$ should be obvious - it is simply the expectation of $T$ under the probability measure $\mathbb P^x$.

  • $\begingroup$ Thank you for your answer, but there's still something I do not understand. Namely, T is a stopping time, so it is not supposed to measurable wrt $\mathbb{P}^x$, right? $\endgroup$ – Symòn Jul 6 '17 at 13:51
  • $\begingroup$ Remember $\mathbb P^x$ is a measure, not a $\sigma$-field. We have a filtration $\{\mathcal F_n\}$ induced by the Markov chain, and necessarily our underlying $\sigma$-field $\mathcal F$ must contain each $\mathcal F_n$. Since $\{T\le n\}\in\mathcal F_n$, it is not hard to show that $\{T\in A\}$ is measurable for any $A\subset\mathbb N$. $\endgroup$ – Jason Jul 6 '17 at 13:55
  • 1
    $\begingroup$ In simpler language, $E_x[T] = E[T|X_0=x]$. This is just a definition. For example, $T$ might be the time to first get to some state, but we have to know where we are starting (we start in the initial state $x$). @Symòn $\endgroup$ – Michael Jul 6 '17 at 14:16
  • $\begingroup$ another, possibly very stupid, question: the domain of $T$ is $\Omega$, but we are integrating over $S$. do I have misunderstood anything? $\endgroup$ – Symòn Jul 6 '17 at 14:27
  • $\begingroup$ $T$ is a stopping time, so, it takes a value of time, either $T \in \{0, 1, 2, ...\}$ if you are in discrete time, or $T \in [0, \infty)$ in continuous time. $T$ is not a state so it is not in $\Omega$. If you have a basic discrete time Markov chain over a finite or countably infinite state space, then you do not need integration (summation works). Similarly, I have never seen filtrations used for such Markov chains (seems overcomplicated), but filtrations might be useful for more general state spaces. $\endgroup$ – Michael Jul 6 '17 at 14:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.