1
$\begingroup$

I have a question regarding to the following definition of a markov chain:

A family of random variables $(X_n)_{n \in \mathbb{N}}$ with values in a set $M$ is a Markov chain if $$ E[f(X_{n+1}) \mid X_0, \ldots, X_n] \overset{P\text{-a.s.}}{=} E[f(X_{n+1}) \mid X_n] $$ for all $n \geq 0$ and all bounded functions $f \colon M \rightarrow \mathbb{R}$.

I understand the definition of markov chains with conditional probabilities, but I do not know how to read the condition above with the expected values. Do I need the conditional expectation for this? Moreover, the role of the function $f$ is not really clear for me.

$\endgroup$

1 Answer 1

1
$\begingroup$

Update (attempting to answer your questions): You should indeed consider the objects as conditional expectation (e.g. the link you mention). The identity (a.s.): $${\Bbb E}[ f(X_{n+1}) | X_n,...,X_0]= {\Bbb E}[ f(X_{n+1}) | X_n]$$ is an identity between random variables (conditional expectations) and needs some interpretation.

The basic idea is that the 'law' of $X_{n+1}$ given values of $X_0,...,X_n$ should only depend on the value of $X_n$. This corresponds to a particular case of the above when we take $A\subset {\Bbb R}$ measurable and then (a.s.) $$ P(X_{n+1}\in A|X_n,...,X_0)={\Bbb E}[ 1_A\circ X_{n+1} | X_n,...,X_0]= {\Bbb E}[ 1_A\circ X_{n+1} | X_n] = P(X_{n+1}\in A|X_n).$$

It does look as a special case but giving this conditional law is equivalent to the general case of bounded $f$ by using standard approximation arguments by simple functions ($f$ is supposed bounded so as to avoid problems of $f(X_n)$ not being integrable).

To get further, and returning to general bounded $f$, let ${\cal F}_k= X_k^{-1}({\cal B}_{\Bbb R})$ be the $\sigma$-algebra generated by each $X_k$ and let ${\cal C}_n={\cal F}_n\vee ...\vee {\cal F}_0$ be the joint $\sigma$-algebra generated by $X_n,...,X_0$. a priori ${\Bbb E}[ f(X_{n+1}) | X_n,...,X_0]$ is just a ${\cal C}_n$ measurable function so that for any given $C\in{\cal C}_n$ we have:

$$ \int_{C} f(X_{n+1}) dP =\int_{C} {\Bbb E}[ f(X_{n+1}) | X_n,...,X_0] dP $$

Now being a Markov chain the statement is that in fact the random variable $Y= {\Bbb E}[ f(X_{n+1}) | X_n,...,X_0]= {\Bbb E}[ f(X_{n+1}) | X_n]$ is really ${\cal F}_n$ measurable, i.e. for $A\subset {\Bbb R}$ measurable the set $Y^{-1}(A) \in {\cal F}_n$ (modulo a null set).

One way to interpret this is that there is a measurable function $\widetilde{f}_n : {\Bbb R}\rightarrow {\Bbb R}$ such that (a.s.) $$ {\Bbb E}[ f(X_{n+1}) | X_n,...,X_0]= {\Bbb E}[ f(X_{n+1}) | X_n]=\widetilde{f}_n \circ X_n$$

We may associate Markov kernels to such a chain. Let $A\subset {\Bbb R}$ be measurable. Then by the previous there is $k_A: {\Bbb R}\rightarrow {\Bbb R}$ such that (a.s.) $ P(X_{n+1}\in A|X_n) = k_A \circ X_n$. For fixed $X_n$ this turns out to be a probability measure w.r.t. $A$, more precisely (but non-trivial):

If $dP_n$ is the law of $X_n$ then for $dP_n$ a.e. $x\in {\Bbb R}$ there is a probability law $A\subset {\Bbb R} \rightarrow k_n(A,x)$ for which (a.s.) $$ P(X_{n+1}\in A|X_n)(\omega) = k_n(A, X_n(\omega)).$$ The function $k_n$ is called the Markov kernel (at time $n$). In practice one often uses the kernels to do computations. Hope this gets closer to what you wanted.

Later added: Suppose we are in the discrete case. This simplifies some issues. We may consider a a countable collection of positive probability elementary events $C_i\in {\cal C}_n$ (with $i\in {\Bbb N}$) of the form $(X_n,...,X_0)=(x_n,...,x_0)$. Denote $V_n(\omega)=(X_n(\omega),...,X_0(\omega))$. We then have $$ P(X_{n+1}\in A| V_n) (\omega) = \sum_i P(X_{n+1}\in A|C_i) 1_{C_i}(\omega)= P(X_{n+1}\in A |C_{i(\omega)}) $$ so given $\omega$ we simply find the partition element $C_{i(\omega})$ to which $V_n(\omega)$ belongs and then calculate the usual conditional probability with respect to that.

In the same spirit suppose $C$ is a union of these partition elements. Then (omitting details in the 2nd equality) $$ \int_C P(X_{n+1}\in A| V_n) dP = \sum_{i: C_i\subset C} \int_{C_i} P(X_{n+1}\in A| V_n) dP= \sum_{i: C_i\subset C} \int_{C_i} P((X_{n+1}\in A)\cap C_i)=P((X_{n+1}\in A)\cap C)$$ So integrating over $C$ removes in a sense the 'conditioning' part of the probability. This last part is just about how to interpret conditional probabilities as r.v. and is not particularly related to Markov chains

$\endgroup$
7
  • $\begingroup$ Thank you for your answer. I will need some time to think about it. First, I have a question to the equality $ P(X_{n+1}\in A|X_n,...,X_0)={\Bbb E}[ 1_A\circ X_{n+1} | X_n,...,X_0]$. Why is $ P(X_{n+1}\in A|X_n,...,X_0)$ a $\mathcal{C}_n$-measurable function with $$ \int_{C} (1_A \circ X_{n+1}) dP = \int_{C} P(X_{n+1} \in A \mid X_n, \ldots, X_0) dP $$ for $C \in \mathcal{C}_n$? $\endgroup$
    – user148364
    Aug 27, 2016 at 20:47
  • $\begingroup$ It is more or less the definition. ${\cal C}_n$ is the $\sigma$-algebra generated by the $X_n,...,X_0$ so integrating over an element in that algebra 'removes' the 'condition' and the integral (expectation) of $1_A\circ X_{n+1}$ is the same as the probability. The LHS can also be written: $P((X_{n+1}\in A) \cap C)$. Perhaps that clarifies? $\endgroup$
    – H. H. Rugh
    Aug 27, 2016 at 21:56
  • $\begingroup$ So the condition which is removed means that the set of which we calculate the probability is contained in $\mathcal{C}_n$? This would make it clear to me. $\endgroup$
    – user148364
    Aug 27, 2016 at 23:01
  • $\begingroup$ Not quite sure what you mean? The set $(X_{n+1}\in A)$ need not be in ${\cal C}_n$. But we calculate the probability of its intersection with $C$ (which is in ${\cal C}_n$). $\endgroup$
    – H. H. Rugh
    Aug 28, 2016 at 9:20
  • $\begingroup$ Maybe I should ask what $P(\cdot \mid X_n, \ldots, X_0)$ means? Is it defined by $P(\omega \in A \mid X_n, \ldots, X_0) =P(\omega \in A \cap \mathcal{C}_n)/P(\omega \in \mathcal{C}_n)$? $\endgroup$
    – user148364
    Aug 28, 2016 at 16:31

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .