28
$\begingroup$

Intuitively, what's the difference between 2 following terms on the right hand side of the law of total variance?

$$\text{Var}(Y) = \Bbb E\left[\text{Var}\left(Y|X\right)\right] + \text{Var}\left(\Bbb E[Y|X]\right)$$

$\endgroup$
1

4 Answers 4

28
$\begingroup$

This law is assuming that you are "breaking up" the sample space for $Y$ based on the values of some other random variable $X$.

In this context, both $Var(Y|X)$ and $E[Y|X]$ are random variables. Each realization assumes that we first draw $X$ from its unconditional distribution, then sample $Y$ from its conditional distribution given $X=x$.

The first term says that we want the expected variance of $Y$ as we average over all values of $X$. HOWEVER, remember that the $Var[Y|X=x]$ is taken with respect to the conditional mean $E[Y|X=x]$. Therefore, this does not take into account the movement of the mean itself, just the variation about each, possibly varying, mean.

This is where the second term comes in: It does not care about the variability about $E[Y|X=x]$, just the variability of $E[Y|X]$ itself.

If we treat each $X=x$ as a separate "treatment", then the first term is measuring the average within sample variance, while the second is measuring the between sample variance.

$\endgroup$
21
$\begingroup$

From my experience, people learning about that theorem for the first time often have trouble understanding why the second term, i.e. $\mathrm{Var}[\mathrm{E}(Y|X)]$, is necessary. Since the question asks for the intuition, I think a visual explanation that can also act as a mnemonic device could be a useful addition to the already existing answers.

Let's assume $P(x,y)$ is given by a 2D Gaussian that is aligned with the axes:

enter image description here

Now, for each fixed value $X=X_i$, we get a distribution $P(Y|X=X_i)$, as in the figure below:

enter image description here

Since all of those 1D distributions have the same expectation $E(Y)$, it intuitively makes sense that $\mathrm{Var}[Y]$ should be the average of all their individual variances, i.e., we have $\mathrm{Var}(Y)=\mathrm{E}[\mathrm{Var}[Y|X]]$, which is the first term in the theorem as written in the question.

Now, let's see what happens if we rotate the 2D Gaussian so that it is no longer aligned with the axes:

enter image description here

We see that in this case, $\mathrm{Var}[Y]$ doesn't only depend on the individual variances of the $P(Y|X=X_i)$ distributions, but that it also depends on how spread out the distributions themselves are along the $Y$ axis. For example, the further the mean of $P(Y|X=X_n)$ is from the mean of $P(Y|X=X_1)$, the larger the overall interval spanned by all the values of $Y$ will be. As a result, it is no longer sufficient to only consider $\mathrm{Var}(Y)=\mathrm{E}[\mathrm{Var}[Y|X]]$, and we need to account for the variability of the means of the $P(Y|X=X_i)$ distributions. This is the intuition behind the second term, i.e. $\mathrm{Var}[\mathrm{E}(Y|X)]$.

$\endgroup$
1
  • 1
    $\begingroup$ Loved that visual representation. Really added something that was hard to appreciate without it. $\endgroup$
    – NathanLite
    Jun 26, 2020 at 0:45
11
$\begingroup$

The square of an expectation is distinct from the expectation of a square; that's what variance is all about.   $\mathsf {Var}(Z) = \mathsf E(Z^2)-\mathsf E(Z)^2$

And so the mean of the X-measured variation is distinct from the variation of the X-measured mean.   Though they sum to the total variation by no coincidence.

$$\begin{align} \mathsf {E}\big(\mathsf {Var} (Y\mid X)\big) ~=~& \mathsf E\big(\mathsf E(Y^2\mid X)\big)-\mathsf E\big(\mathsf E(Y\mid X)^2\big) \\[1ex] ~=~& \mathsf E(Y^2)-\mathsf E\big(\mathsf E(Y\mid X)^2\big) \\[2ex] \mathsf {Var}\big(\mathsf {E} (Y\mid X)\big) ~=~& \mathsf E\big(\mathsf E(Y\mid X)^2\big)-\mathsf E\big(\mathsf E(Y\mid X)\big)^2 \\[1ex] ~=~& \mathsf E\big(\mathsf E(Y\mid X)^2\big)-\mathsf E(Y)^2 \\[2ex] \hline \therefore ~ \mathsf {E}\big(\mathsf {Var} (Y\mid X)\big)+ \mathsf {Var}\big(\mathsf {E} (Y\mid X)\big) ~=~& \mathsf E(Y^2)-\mathsf E(Y)^2 \\[1ex] ~=~& \mathsf {Var}(Y) \end{align}$$

$\endgroup$
0
$\begingroup$

If you're familiar with one-way analysis of variance, the law of total variance is a generalization of the sum-of-squares identity $$\operatorname{SS}_T=\operatorname{SS}_W + \operatorname{SS}_B\tag1$$ which decomposes the total variation into variation within treatments and variation between treatments.


Here's the calculation in detail. Let there be $I$ treatments, with responses $y_{i,1},\ldots,y_{i,n_i}$ to treatment $i$. Write $$y_{i\cdot}:=\frac1{n_i}\sum_{j=1}^{n_i}y_{i,j}\tag2$$ for the mean response to treatment $i$, and write $$y_{\cdot\cdot}:=\frac{\sum_i\sum_j y_{i,j}}{\sum_i n_i}=\frac{\sum_i n_iy_{i\cdot}}{N}\tag3$$ for the grand mean. Define the random variable $Y$ to be the result of selecting a value uniformly at random from the $N$ values $y_{1,1},\ldots,y_{1,n_1},\ldots,y_{I,1},\ldots,y_{I,n_I}$, and let $X$ be the associated treatment, i.e., the first subscript on the $y_{i,j}$ that was selected, so that $P(X=i)=n_i/N$. Then:

Claim: $\operatorname{Var}(Y)=\frac1N\operatorname{SS}_T$.

Proof: $E(Y)$ is the average of all possible values for $Y$, so it equals the grand mean $y_{\cdot\cdot}$, and $$\operatorname{Var}(Y)=\frac1N\sum_i\sum_j (y_{i,j}-y_{\cdot\cdot})^2=:\frac1N\operatorname{SS}_T.\tag4$$

Claim: $E(\operatorname{Var}(Y\mid X))=\frac1N\operatorname{SS}_W$ and $\operatorname{Var}(E( Y\mid X))=\frac1N\operatorname{SS}_B$.

Proof: Given $X=i$, $Y$ is uniformly distributed over the $n_i$ values $y_{i,1},\ldots,y_{i,n_i}$ so its conditional mean is

$$E(Y\mid X=i)=\frac1{n_i}\sum_j y_{i,j}=:y_{i\cdot}\tag5$$ and conditional variance is $$\operatorname{Var}(Y\mid X=i)=\frac1{n_i}\sum_j(y_{i,j}-y_{i\cdot})^2.\tag6$$ Hence $E(\operatorname{Var}(Y\mid X))$ is the weighted average of (6): $$E(\operatorname{Var}(Y\mid X))=\sum_i \operatorname{Var}(Y\mid X=i) P(X=i) =\sum_i\frac1{n_i}\sum_j(y_{i,j}-y_{i\cdot})^2\frac{n_i}N=:\frac1N\operatorname{SS}_W.\tag7 $$ Similarly the expectation of $E(Y\mid X)$ is the weighted average of (5): $$E(E(Y\mid X))=\sum_i E(Y\mid X=i)P(X=i)=\sum_i y_{i\cdot}\frac{n_i}N=:y_{\cdot\cdot}\tag8$$ which of course equals the unconditional mean $E(Y)$. Compute the variance of $E(Y\mid X)$ using the formula $$ \operatorname{Var}(h(X))=\sum_i [h(i)-Eh(X)]^2P(X=i)$$ to obtain $$\operatorname{Var}(E(Y\mid X))=\sum_i[E(Y\mid X=i)-E(Y)]^2P(X=i) =\sum_i (y_{i\cdot}-y_{\cdot\cdot})^2\frac {n_i} N=:\frac1N\operatorname{SS}_B.\tag9$$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.