# Expected value of function of Markov chain

Let $$X$$ be a Markov chain with state space $$E = {1, 2, 3, 4}$$ and transition matrix

$$P = \begin{bmatrix} 0 & 0 & 1 & 0 \\ 0 & 0.4 & 0.6 & 0\\ 0.8 & 0 & 0.2 & 0\\ 0.2 & 0.3 & 0 & 0.5\\ \end{bmatrix}$$

Compute

$$E[f(X_5)f(X_6)|X_4 = 4]$$

for the function $$f$$ with values $$2, 4, 7,$$ and $$3$$ at states $$1, 2, 3,$$ and $$4$$ respectively.

My solution:

I assume you can find $$E[f(X_5)]E[f(X_6)]$$ due to independence.

For $$E[f(X_5)]$$, since step begins in state $$4$$ then proceeds one step, I looked at the fourth row and considered all possibilities. State $$4$$ to state $$1$$ has probability $$0.2$$ with value $$2$$. Likewise, state $$4$$ to state $$2$$: $$0.3$$ with $$4$$, state $$4$$ to state $$4$$: $$0.5$$ with $$3$$. Then taking the product of each pair then summing all you get $$3.1$$.

Then, for $$E[f(X_6)]$$, you begin in state $$4$$ but need to take two steps. You can square the transition matrix. You need only consider the fourth row since you know you begin in state $$4$$. Thus, take row $$4$$ and multiply by each column, obtaining:

$$P = \begin{bmatrix} 0.1 & 0.27 & 0.38 & 0.25\\ \end{bmatrix}$$

This part seems correct, as the elements sum to $$1$$.

Then as before take each probability with its corresponding function value in each state, find the products and sum, and I obtained $$4.69$$.

Finally take the product $$(3.1)(4.69) = 14.539$$.

The answer should be $$14.41$$. Where are my errors? And how to obtain the correct solution?

• It is not true that $X_n$'s are independent . – Kavi Rama Murthy Oct 2 '19 at 23:22

Hints for a correct approach: $$P(X_6=i,X_5=j|X_4=4)=p_{4j}p_{ji}$$ from Markov property. Compute this for all possible values of $$i$$ and $$j$$ and the use the equation
$$E(f(X_6)f(X_5)|X_4=4)=\sum_{i,j} f(i)f(j) P(X_6=i,X_5=j|X_4=4)$$.