6
$\begingroup$

I have the formulas for the variance of linear combintion of variables given as $$ \text{Var}\Bigl(\,\sum_{i=1}^n X_i\,\Bigr)= \sum_{i=1}^n\text{Var}( X_i)+ 2\sum_{i< j} \text{Cov}(X_i,X_j). $$ I know this is derived from variance of 2 variables and extrapolation. However, what is the intuition behind this formula? Can someone explain further on how I could use this formula to understand covariance better?

$\endgroup$
14
$\begingroup$

Other answers have addressed derivation, but you asked for intuition:

Variance is a measure of uncertainty. The higher the variance the greater the uncertainty. Uncertainty is additive. If $X_1$ and $X_2$ are uncorrelated, the uncertainty in their sum is the sum of their uncertainties: $Var(X_1 + X_2) = Var(X_1) + Var(X_2)$. But what if they are correlated? Two things can happen. If they are positively correlated this will increase the uncertainty since this makes it more likely that observations of the two variables will pull away from the means in the same direction, pulling the sum even farther away from its mean. On the other hand, if they are negatively correlated, they will tend to pull in opposite directions with larger than average observations of one variable tending to be offset by smaller than average observations of the other. This will lead to a cancellation of errors whereby the sum will be even closer to its average than either of the variables are close to theirs. The covariance term $Cov(X_1,X_2)$ shows how the correlation between the variables effects the overall uncertainty. Similar intuitions hold with more than 2 variables. Every pair of variables $X_i,X_j$ can either increase or decrease the overall uncertainty of the sum. By adding all such covariances together you get the net change of uncertainty from all such pairs.

Intuitions are not theorems. The above paragraph doesn't apply to all cases. For example, one way that you could have $Cov(X_1,X_2) > 0$ is by having a large number of negative values be offset by rarer but larger positive values. $X_1$ and $X_2$ might actually tend to pull in opposite directions in the sense that $P((X_1 - \mu_1)(X_2 - \mu_2) < 0) > 0.5$ but nevertheless $Cov(X_1,X_2) > 0$ since when they pull in the same direction they really pull in the same direction. The formula itself takes all such considerations into account.

$\endgroup$
  • $\begingroup$ This should be the accepted answer $\endgroup$ – SK19 Mar 19 '18 at 11:55
  • $\begingroup$ @SK19: (a) The OP probably knows what he finds more intuitive than we do, (b) I actually found the accepted answer more intuitive to be honest... it explains where the $2$ comes from. $\endgroup$ – user541686 Mar 19 '18 at 20:33
9
$\begingroup$

This formula is actually not at all deep and works if you replace $\mathrm{Cov}$ with any symmetric bilinear form. Imagine we have an inner product $(-,-)$ and we want to find $(a+b,a+b)$. Then $$(a+b,a+b)=(a,a)+(b,b)+(a,b)+(b,a)=(a,a)+(b,b)+2(a,b)$$ You can find a similar formula for $(a+b+c,a+b+c)$, and the general form is exactly the same. It just depends on covariance being symmetric and bilinear and the variance of a variable being the covariance with itself.

$\endgroup$
8
$\begingroup$

It is very simple. It is just like opening up brackets in a $(a+b)^2$. In that case you will have $a^2+2ab+b^2$.

Hopefully that can guide you to figure out how it's derived. Let me know if you need in depth explanation though.

$\endgroup$
5
$\begingroup$

If you use the formulas $$\text{Var}(X) = E[X^2]-E[X]^2$$ and $$\text{Cov}(X,Y) = E[XY] - E[X]E[Y]$$

using the properties of expected value you can derive the formula easily.

It might also help to see variance as a special case of covariance:

$$\text{Var}(X) = \text{Cov}(X,X)$$ so we see $\text{Var}(X_i)$ is a simplified form of $\text{Cov}(X_i,X_i)$, matching the covariance sum.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.