2
$\begingroup$

I am reading a text where they claim that $$P(X_1=x_1|X_2=x_2) = \\ ​E[P(X_1=x_1|X_3) | X_2 = x_2] =\\ \sum_{x_3} P(X_1 = x_1|X_2 = x_2, X_3 = x_3)P(X_3=x_3|X_2=x_2) \quad (1.),$$ where $X_1, X_2, X_3$ are discrete random variables and $P(A, B)$ is notation for $P(A \land B)$.

I can see why $$P(X_1=x_1|X_2=x_2) = \sum_{x_3} P(X_1 = x_1|X_2 = x_2, X_3 = x_3)P(X_3=x_3|X_2=x_2)$$ since this is a consequence of that the events $X_3 = x_3$ for all $x_3$ in the range of $X_3$ partitions the sample space, combined with the law of total probability and the definition of conditional probability. But I don't see how the first or second equality holds in $(1.)$. It seems as I have the wrong interpretation of

$$​E[P(X_1=x_1|X_3) | X_2 = x_2] \quad (2.) $$

I am looking to understand where my understanding of this expression $(2.)$ is wrong. This is how I interpret it:

The random variables are defined over some sample space $S$ so that for example $X_1 = x_1$ is just another way of writing $X_1(s) = x_1$ for some $s \in S$. Define the random variable $$X_4: S \to [0, 1], \quad X_4(s) = P(X_1=x_1|X_3 = X_3(s)) \quad (3.)$$ Then we have that $(2.)$ can be written as $$ E[P(X_1=x_1|X_3) | X_2 = x_2] = E[X4|X_2 = x_2] = \sum_{x_4} x_4 P(X_4=x_4 | X_2 = x_2) $$ Here I am stuck. One reason is that I don't understand how to go from summing over $x_4$ to summing over $x_3$ as is done in $(1.)$. Have I started correctly and how can I proceed?

$\endgroup$
2
  • 1
    $\begingroup$ I've been thinking about this for a while and I honestly I don't think it's correct to say that $$P(X_1=x_1|X_2=x_2)=E\Big(P(X_1=x_1|X_3)|X_2=x_2\Big)$$ I do believe we can say $$P(X_1=x_1|X_2=x_2)=E\Big(P(X_1=x_1|X_2,X_3)|X_2=x_2\Big)$$ I may post my feelings about this problem in a bit, but I'm not convinced the author is correct on this one. Your interpretation of the expression $P(X_1=x_1|X_3)$ as a function of the random variable $X_3$ is sound. $\endgroup$ Sep 14 at 23:36
  • $\begingroup$ Indeed, they clearly intended the second line to expand to the third, so: $$\begin{align}&\mathsf P(X_1=x_1\mid X_2=x_2) \\ = & ​\mathsf E\left[~\mathsf P(X_1=x_1\mid X_2, X_3)~\middle\vert~ X_2 = x_2~\raise{2.5ex}{}\right] \\ =& \sum_{x_3} \mathsf P(X_1 = x_1\mid X_2 = x_2, X_3 = x_3)~\mathsf P(X_3=x_3\mid X_2=x_2) \tag{ 1.}\end{align}$$ $\endgroup$ Sep 15 at 3:02
3
$\begingroup$

I don't think we can equate $P(X_1=x_1|X_2=x_2)$ and $E\Big(P(X_1=x_1|X_3)|X_2=x_2\Big)$. To see an example of this, suppose that $(X_1,X_2,X_3)\sim p$ where $p$ is the pmf defined below: $$p(1,1,1)=0.2 \\ p(1,1,2)=0.1 \\ p(1,2,1)=0.01 \\ p(1,2,2)=0.13 \\ p(2,1,1)=0.06 \\ p(2,1,2)=0.11 \\ p(2,2,1)=0.09 \\ p(2,2,2)=0.3 $$ Assume $p(x,y,z)=0$ for all $(x,y,z)\notin \{1,2\}^3$. It's not difficult to verify $$P(X_1=1|X_2=2)=\frac{14}{53}$$ On the other hand, we get with LOTUS that $$\begin{eqnarray*}E\Big(P(X_1=1|X_3)|X_2=2\Big) &=& \sum_{a,b\in \{1,2\}}P(X_1=1|X_3=b)P(X_1=a,X_3=b|X_2=2) \\ &=& \sum_{b\in \{1,2\}}P(X_1=1|X_3=b) \sum_{a\in \{1,2\}}P(X_1=a,X_3=b|X_2=2) \\ &=& \sum_{b\in \{1,2\}}P(X_1=1|X_3=b)P(X_3=b|X_2=2) \\ &=& \frac{7}{12}\cdot \frac{10}{53}+ \frac{23}{64} \cdot \frac{43}{53} \\ &\neq & \frac{14}{53} \end{eqnarray*}$$ If you wish to carry out this computation without the aid of LOTUS (as you started to do) we would need first to establish the conditional pmf of $P(X_1=1|X_3)$ given $X_2=2$. A brief calculator exercise reveals the random variable $P(X_1=1|X_3)$ is supported on the set $\Big\{\frac{7}{12},\frac{23}{64}\Big\}$ and satisfies $$P\Big(P(X_1=1|X_3)=\frac{7}{12}\Big|X_2=2\Big)=P(X_3=1|X_2=2)=\frac{10}{53}$$ $$P\Big(P(X_1=1|X_3)=\frac{23}{64}\Big|X_2=2\Big)=P(X_3=2|X_2=2)=\frac{43}{53}$$ Finally, $$\begin{eqnarray*}E\Big(P(X_1=1|X_3)|X_2=2\Big)&=&\sum_{t\in\big\{\frac{7}{12},\frac{23}{64}\big\}}t P\Big(P(X_1=1|X_3)=t|X_2=2\Big) \\ &=& \frac{7}{12}\cdot \frac{10}{53}+ \frac{23}{64} \cdot \frac{43}{53} \\ &\neq& \frac{14}{53} \end{eqnarray*}$$ As I mentioned in the comments, we may certainly conclude that $$P(X_1=1|X_2=2)=E\Big(P(X_1=1|X_2,X_3)|X_2=2\Big)$$

$\endgroup$
6
  • $\begingroup$ Thank you for the detailed answer. Interesting to see it calculated without LOTUS. I had forgotten about LOTUS, and think I don't quite understand your use of it: 1.) When you use LOTUS to calculate $E\Big(P(X_1=1|X_3)|X_2=2\Big)$, it seems like you are considering a multivariate function $h(X_1, X_3)$? Wouldn't it be easier to consider single variable function like in grand_chat's answer $h(X_3) = P(X_1=1|X_3)$? $\endgroup$ Sep 15 at 6:44
  • $\begingroup$ 2.) Possibly related: in the last line when you write $P(X_1=1|X_2=2)=E\Big(P(X_1=1|X_2,X_3)|X_2=2\Big)$, shouldn't it be $P(X_1=1|X_2=2)=E\Big(P(X_1=1|X_2=2,X_3)|X_2=2\Big)$, since it seems like the expectation is taken with respect to a single variable $x_3$? As far as I can see, in the former case, to use LOTUS, we would define $h(X_2, X_3) = P(X_1=1|X_2,X_3)$ so we would need to use multivariate expectation. In the latter case we would define $h(X_3) = (P(X_1=1|X_2=2,X_3)$ and can use single-variable expectation with respect to $x_3$. $\endgroup$ Sep 15 at 6:49
  • $\begingroup$ @DancingIceCream I used the formula $$E\Big(h(X_1,X_2,X_3)|X_2=2\Big)=\sum_{a,b}h(a,2,b)P(X_1=a,X_3=b|X_2=2)$$ This is a general rule that works all the time, regardless of what your function $h$ is. I wanted to be as clear as possible in my calculations since you mentioned this formula was published in a textbook; claiming a textbook is incorrect is taking a bit of a risk on my part. Now if $h$ is a function of just $X_3$ (like it is here) then we may conclude $$E\Big(h(X_3)|X_2=2\Big)=\sum_{b}h(b)P(X_3=b|X_2=2)$$ $\endgroup$ Sep 15 at 13:21
  • $\begingroup$ To use the formula above you must first find an expression for $P(X_3=b|X_2=2)$ which turns out to equal $$P(X_3=b|X_2=2)=\sum_{a}P(X_1=a,X_3=b|X_2=2)$$ Also, $E\Big(P(X_1=1|X_2,X_3)|X_2=2\Big)$ does in fact equal $E\Big(P(X_1=1|X_2=2,X_3)|X_2=2\Big)$ since we're "fixing" $X_2$ to be $2$ in both expectations. In my opinion $E\Big(P(X_1=1|X_2=2,X_3)|X_2=2\Big)$ is a bit redundant. $\endgroup$ Sep 15 at 13:21
  • $\begingroup$ We would use multivariate expectation to compute an expression like $$E\Big(h(X_2,X_3)\Big)=\sum_{a,b}h(a,b)P(X_2=a,X_3=b)$$ where $$P(X_2=a,X_3=b)=\sum_{x}P(X_1=x,X_2=a,X_3=b)$$ We would not use multivariate expectation to compute $E\Big(h(X_2,X_3)|X_2=2\Big)$ since we're conditioning on the event that $X_2=2$: $$E\Big(h(X_2,X_3)|X_2=2\Big)=E\Big(h(2,X_3)|X_2=2\Big)=\sum_{b}h(2,b)P(X_3=b|X_2=2)$$ $\endgroup$ Sep 15 at 13:22
3
$\begingroup$

I agree with @MatthewPilling's comment. The text is wrong to assert that the middle entity $$E[P(X_1=x_1|X_3) | X_2 = x_2] $$ is equal to the outer two entities (which, as you've shown, are indeed equal to each other).

In fact for events $A$, $B$ and a discrete random variable $Y$ taking values $y_1,\ldots,y_n$, $$E( P (A\mid Y) \mid B) = \sum_{i=1}^n P(A\mid Y=y_i) P(Y=y_i\mid B),\tag1$$ which doesn't equal $P(A\mid B)$. To prove (1), recall that $P(A\mid Y)$ is a random variable $h(Y)$ such that $h(y):=P(A\mid Y=y)$. So $$ E(P(A\mid Y)\mid B)= E(h(Y)\mid B)=\sum_i h(y_i) P(Y=y_i\mid B) $$ and the result follows.

$\endgroup$
4
  • $\begingroup$ I am self teaching myself probability and need some clarification doesn't $P(A|Y=y_i)P(Y=y_1|B)=P(A,Y=y_i|B)$ upon summing across the space of $Y$ don't we get the desired $P(A|B)$. $\endgroup$ Sep 15 at 1:15
  • 1
    $\begingroup$ @NoeVidales To obtain $P(A\mid B)$ you would need to sum $P(A\mid B, Y=y_i)P(Y=y_i\mid B)$ over the values $y_i$. $\endgroup$
    – grand_chat
    Sep 15 at 1:20
  • $\begingroup$ @NoeVidales Your equation only holds when $A$ and $B$ are conditionally independent when given $Y=y_i$. Grand_chat's statement holds in general cases. $\endgroup$ Sep 15 at 2:42
  • $\begingroup$ Yes I see that. Thank you for the clarification $\endgroup$ Sep 15 at 2:56
0
$\begingroup$

Consider the random variable $X_3$ comes from a joint probability $p(x_1, x_2, x_3)$.

If you think everything that happens(including $P(X_1=x_1|X_3)$) is always constrained by the condition $X_2 = x_2$, then the equalities hold.

In 3.) you are assuming the function is not constrained by the conditoning.

For example, if the original sample space is $$\{(x_1,x_2,x_3)~|~x_k=0 ~\textrm{or}~ 1 \textrm{ for } ~k=1,2,3\}, $$then after the conditioning by $x_2=1$, what you have is $$\{(x_1,1,0),(x_1,1,1)\}, ~x_1=0\textrm{ or }1.$$ The expectation in the second line works on this smaller sample space with preassigned value $x_2=1.$

$\endgroup$
0
$\begingroup$

One other way of viewing it is through the law of total expectation $\mathcal{F}_1\subset \mathcal{F}_2$ then $E[X|\mathcal{F}_2|\mathcal{F}_1]=E[X|\mathcal{F}_1]$ . Since $\sigma(X_2)\subset \sigma(X_2,X_3)$ then we know

$$E[X|\sigma(X_2,X_3)|\sigma(X_2)]=E[X|\sigma(X_2)]$$

If we let $X=\mathbb{I}_{x_1}(X_1)$

Then we get the desired equality, as noted with the other posts when conditioning on both $X_2, X_3$ instead of just $X_2$.

My first post I mistakenly miscalculated the conditional expectations.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.