# Problem seeing how $P(X_1=x_1|X_2=x_2) = E[P(X_1=x_1|X_3) | X_2 = x_2]$

I am reading a text where they claim that $$P(X_1=x_1|X_2=x_2) = \\ ​E[P(X_1=x_1|X_3) | X_2 = x_2] =\\ \sum_{x_3} P(X_1 = x_1|X_2 = x_2, X_3 = x_3)P(X_3=x_3|X_2=x_2) \quad (1.),$$ where $$X_1, X_2, X_3$$ are discrete random variables and $$P(A, B)$$ is notation for $$P(A \land B)$$.

I can see why $$P(X_1=x_1|X_2=x_2) = \sum_{x_3} P(X_1 = x_1|X_2 = x_2, X_3 = x_3)P(X_3=x_3|X_2=x_2)$$ since this is a consequence of that the events $$X_3 = x_3$$ for all $$x_3$$ in the range of $$X_3$$ partitions the sample space, combined with the law of total probability and the definition of conditional probability. But I don't see how the first or second equality holds in $$(1.)$$. It seems as I have the wrong interpretation of

$$​E[P(X_1=x_1|X_3) | X_2 = x_2] \quad (2.)$$

I am looking to understand where my understanding of this expression $$(2.)$$ is wrong. This is how I interpret it:

The random variables are defined over some sample space $$S$$ so that for example $$X_1 = x_1$$ is just another way of writing $$X_1(s) = x_1$$ for some $$s \in S$$. Define the random variable $$X_4: S \to [0, 1], \quad X_4(s) = P(X_1=x_1|X_3 = X_3(s)) \quad (3.)$$ Then we have that $$(2.)$$ can be written as $$E[P(X_1=x_1|X_3) | X_2 = x_2] = E[X4|X_2 = x_2] = \sum_{x_4} x_4 P(X_4=x_4 | X_2 = x_2)$$ Here I am stuck. One reason is that I don't understand how to go from summing over $$x_4$$ to summing over $$x_3$$ as is done in $$(1.)$$. Have I started correctly and how can I proceed?

• I've been thinking about this for a while and I honestly I don't think it's correct to say that $$P(X_1=x_1|X_2=x_2)=E\Big(P(X_1=x_1|X_3)|X_2=x_2\Big)$$ I do believe we can say $$P(X_1=x_1|X_2=x_2)=E\Big(P(X_1=x_1|X_2,X_3)|X_2=x_2\Big)$$ I may post my feelings about this problem in a bit, but I'm not convinced the author is correct on this one. Your interpretation of the expression $P(X_1=x_1|X_3)$ as a function of the random variable $X_3$ is sound. Sep 14 at 23:36
• Indeed, they clearly intended the second line to expand to the third, so: \begin{align}&\mathsf P(X_1=x_1\mid X_2=x_2) \\ = & ​\mathsf E\left[~\mathsf P(X_1=x_1\mid X_2, X_3)~\middle\vert~ X_2 = x_2~\raise{2.5ex}{}\right] \\ =& \sum_{x_3} \mathsf P(X_1 = x_1\mid X_2 = x_2, X_3 = x_3)~\mathsf P(X_3=x_3\mid X_2=x_2) \tag{ 1.}\end{align} Sep 15 at 3:02

I don't think we can equate $$P(X_1=x_1|X_2=x_2)$$ and $$E\Big(P(X_1=x_1|X_3)|X_2=x_2\Big)$$. To see an example of this, suppose that $$(X_1,X_2,X_3)\sim p$$ where $$p$$ is the pmf defined below: $$p(1,1,1)=0.2 \\ p(1,1,2)=0.1 \\ p(1,2,1)=0.01 \\ p(1,2,2)=0.13 \\ p(2,1,1)=0.06 \\ p(2,1,2)=0.11 \\ p(2,2,1)=0.09 \\ p(2,2,2)=0.3$$ Assume $$p(x,y,z)=0$$ for all $$(x,y,z)\notin \{1,2\}^3$$. It's not difficult to verify $$P(X_1=1|X_2=2)=\frac{14}{53}$$ On the other hand, we get with LOTUS that $$\begin{eqnarray*}E\Big(P(X_1=1|X_3)|X_2=2\Big) &=& \sum_{a,b\in \{1,2\}}P(X_1=1|X_3=b)P(X_1=a,X_3=b|X_2=2) \\ &=& \sum_{b\in \{1,2\}}P(X_1=1|X_3=b) \sum_{a\in \{1,2\}}P(X_1=a,X_3=b|X_2=2) \\ &=& \sum_{b\in \{1,2\}}P(X_1=1|X_3=b)P(X_3=b|X_2=2) \\ &=& \frac{7}{12}\cdot \frac{10}{53}+ \frac{23}{64} \cdot \frac{43}{53} \\ &\neq & \frac{14}{53} \end{eqnarray*}$$ If you wish to carry out this computation without the aid of LOTUS (as you started to do) we would need first to establish the conditional pmf of $$P(X_1=1|X_3)$$ given $$X_2=2$$. A brief calculator exercise reveals the random variable $$P(X_1=1|X_3)$$ is supported on the set $$\Big\{\frac{7}{12},\frac{23}{64}\Big\}$$ and satisfies $$P\Big(P(X_1=1|X_3)=\frac{7}{12}\Big|X_2=2\Big)=P(X_3=1|X_2=2)=\frac{10}{53}$$ $$P\Big(P(X_1=1|X_3)=\frac{23}{64}\Big|X_2=2\Big)=P(X_3=2|X_2=2)=\frac{43}{53}$$ Finally, $$\begin{eqnarray*}E\Big(P(X_1=1|X_3)|X_2=2\Big)&=&\sum_{t\in\big\{\frac{7}{12},\frac{23}{64}\big\}}t P\Big(P(X_1=1|X_3)=t|X_2=2\Big) \\ &=& \frac{7}{12}\cdot \frac{10}{53}+ \frac{23}{64} \cdot \frac{43}{53} \\ &\neq& \frac{14}{53} \end{eqnarray*}$$ As I mentioned in the comments, we may certainly conclude that $$P(X_1=1|X_2=2)=E\Big(P(X_1=1|X_2,X_3)|X_2=2\Big)$$

• Thank you for the detailed answer. Interesting to see it calculated without LOTUS. I had forgotten about LOTUS, and think I don't quite understand your use of it: 1.) When you use LOTUS to calculate $E\Big(P(X_1=1|X_3)|X_2=2\Big)$, it seems like you are considering a multivariate function $h(X_1, X_3)$? Wouldn't it be easier to consider single variable function like in grand_chat's answer $h(X_3) = P(X_1=1|X_3)$? Sep 15 at 6:44
• 2.) Possibly related: in the last line when you write $P(X_1=1|X_2=2)=E\Big(P(X_1=1|X_2,X_3)|X_2=2\Big)$, shouldn't it be $P(X_1=1|X_2=2)=E\Big(P(X_1=1|X_2=2,X_3)|X_2=2\Big)$, since it seems like the expectation is taken with respect to a single variable $x_3$? As far as I can see, in the former case, to use LOTUS, we would define $h(X_2, X_3) = P(X_1=1|X_2,X_3)$ so we would need to use multivariate expectation. In the latter case we would define $h(X_3) = (P(X_1=1|X_2=2,X_3)$ and can use single-variable expectation with respect to $x_3$. Sep 15 at 6:49
• @DancingIceCream I used the formula $$E\Big(h(X_1,X_2,X_3)|X_2=2\Big)=\sum_{a,b}h(a,2,b)P(X_1=a,X_3=b|X_2=2)$$ This is a general rule that works all the time, regardless of what your function $h$ is. I wanted to be as clear as possible in my calculations since you mentioned this formula was published in a textbook; claiming a textbook is incorrect is taking a bit of a risk on my part. Now if $h$ is a function of just $X_3$ (like it is here) then we may conclude $$E\Big(h(X_3)|X_2=2\Big)=\sum_{b}h(b)P(X_3=b|X_2=2)$$ Sep 15 at 13:21
• To use the formula above you must first find an expression for $P(X_3=b|X_2=2)$ which turns out to equal $$P(X_3=b|X_2=2)=\sum_{a}P(X_1=a,X_3=b|X_2=2)$$ Also, $E\Big(P(X_1=1|X_2,X_3)|X_2=2\Big)$ does in fact equal $E\Big(P(X_1=1|X_2=2,X_3)|X_2=2\Big)$ since we're "fixing" $X_2$ to be $2$ in both expectations. In my opinion $E\Big(P(X_1=1|X_2=2,X_3)|X_2=2\Big)$ is a bit redundant. Sep 15 at 13:21
• We would use multivariate expectation to compute an expression like $$E\Big(h(X_2,X_3)\Big)=\sum_{a,b}h(a,b)P(X_2=a,X_3=b)$$ where $$P(X_2=a,X_3=b)=\sum_{x}P(X_1=x,X_2=a,X_3=b)$$ We would not use multivariate expectation to compute $E\Big(h(X_2,X_3)|X_2=2\Big)$ since we're conditioning on the event that $X_2=2$: $$E\Big(h(X_2,X_3)|X_2=2\Big)=E\Big(h(2,X_3)|X_2=2\Big)=\sum_{b}h(2,b)P(X_3=b|X_2=2)$$ Sep 15 at 13:22

I agree with @MatthewPilling's comment. The text is wrong to assert that the middle entity $$E[P(X_1=x_1|X_3) | X_2 = x_2]$$ is equal to the outer two entities (which, as you've shown, are indeed equal to each other).

In fact for events $$A$$, $$B$$ and a discrete random variable $$Y$$ taking values $$y_1,\ldots,y_n$$, $$E( P (A\mid Y) \mid B) = \sum_{i=1}^n P(A\mid Y=y_i) P(Y=y_i\mid B),\tag1$$ which doesn't equal $$P(A\mid B)$$. To prove (1), recall that $$P(A\mid Y)$$ is a random variable $$h(Y)$$ such that $$h(y):=P(A\mid Y=y)$$. So $$E(P(A\mid Y)\mid B)= E(h(Y)\mid B)=\sum_i h(y_i) P(Y=y_i\mid B)$$ and the result follows.

• I am self teaching myself probability and need some clarification doesn't $P(A|Y=y_i)P(Y=y_1|B)=P(A,Y=y_i|B)$ upon summing across the space of $Y$ don't we get the desired $P(A|B)$. Sep 15 at 1:15
• @NoeVidales To obtain $P(A\mid B)$ you would need to sum $P(A\mid B, Y=y_i)P(Y=y_i\mid B)$ over the values $y_i$. Sep 15 at 1:20
• @NoeVidales Your equation only holds when $A$ and $B$ are conditionally independent when given $Y=y_i$. Grand_chat's statement holds in general cases. Sep 15 at 2:42
• Yes I see that. Thank you for the clarification Sep 15 at 2:56

Consider the random variable $$X_3$$ comes from a joint probability $$p(x_1, x_2, x_3)$$.

If you think everything that happens(including $$P(X_1=x_1|X_3)$$) is always constrained by the condition $$X_2 = x_2$$, then the equalities hold.

In 3.) you are assuming the function is not constrained by the conditoning.

For example, if the original sample space is $$\{(x_1,x_2,x_3)~|~x_k=0 ~\textrm{or}~ 1 \textrm{ for } ~k=1,2,3\},$$then after the conditioning by $$x_2=1$$, what you have is $$\{(x_1,1,0),(x_1,1,1)\}, ~x_1=0\textrm{ or }1.$$ The expectation in the second line works on this smaller sample space with preassigned value $$x_2=1.$$

One other way of viewing it is through the law of total expectation $$\mathcal{F}_1\subset \mathcal{F}_2$$ then $$E[X|\mathcal{F}_2|\mathcal{F}_1]=E[X|\mathcal{F}_1]$$ . Since $$\sigma(X_2)\subset \sigma(X_2,X_3)$$ then we know

$$E[X|\sigma(X_2,X_3)|\sigma(X_2)]=E[X|\sigma(X_2)]$$

If we let $$X=\mathbb{I}_{x_1}(X_1)$$

Then we get the desired equality, as noted with the other posts when conditioning on both $$X_2, X_3$$ instead of just $$X_2$$.

My first post I mistakenly miscalculated the conditional expectations.