16
$\begingroup$

Suppose that I perform a stochastic task $n$ times (like tossing a coin) and that $p$ is the probability that one of the possible outcomes occurs. If $K$ is the stochastic variable that measures how many times this outcome occurred during the whole experiment, and if all the events are mutually independent, then the probability that $K$ is equal to a specific $k$, with $0\leq k\leq n$, is $$\mathrm{Pr}\{K=k\} = \binom n k p^k (1-p)^{n-k} =: b_n(k), $$ which makes intuitive sense to me.

Now suppose that I want to know what the average, or expected, value of $K$ is going to be: the formulae tell me that $$\begin{split} \langle K\rangle &= \sum_{k=0}^n k b_n(k) = \sum_{k=0}^n k \binom n k p^k (1-p)^{n-k} \\ &= \sum_{k=0}^n k \frac{n(n-1)!}{k(k-1)!(n-k)!} p p^{k-1} (1-p)^{n-k} \\ &= np \sum_{\kappa=1}^\nu \frac{\nu!}{\kappa!(\nu-\kappa)!} p^\kappa (1-p)^{\nu-\kappa} = np(p + 1- p)^{\nu} \\ &= np, \end{split}$$ where I’ve made the substitutions $\kappa = k-1$ and $\nu = n-1$. Notwithstanding the mathematical certainty of this derivation, it also makes perfect intuitive sense to me that $np$ should be the expected value of $K$, since it is the product of the probability of the outcome times the number of trials performed: if there’s a $1/6$ chance that I roll a 5 on a fair dice, and I throw it $600$ times, then I expect to see a 5 about $100$ of those times.

If instead I want to know how I should expect the outcomes to vary around the expected value, I may compute the variance of $K$: with the same substitutions as before, $$\begin{split} \mathrm{Var}[K] &= \langle K^2\rangle - \langle K \rangle^2 = \left(\sum_{k=0}^n k^2 b_n(k) \right) -n^2p^2 \\ &= \left( \sum_{k=1}^n k^2 \binom n k p^k (1-p)^{n-k}\right) -n^2p^2 \\ &= \left( np \sum_{\kappa = 0}^\nu (\kappa +1) \binom \nu \kappa p^\kappa (1-p)^{\nu-\kappa} \right)-n^2p^2 \\ &= \left(np \Big(\sum_{\kappa=0}^\nu \kappa b_\nu(\kappa) +(p + 1-p)^\nu \Big) \right) -n^2p^2 \\ &= np(\nu p + 1) - n^2 p^2 = np( np - p + 1 - np) \\ &= np(1-p). \end{split}$$ Again, the derivation is mathematically crystalline; but why should I expect that this be the formula for the variance of $K$? Why does multiplying the expected value of $K$ times the probability that my outcome doesn’t occur give me a measure of the dispersion of $K$? In other words, how can I justify this formula for variance intuitively in a similar way as I can with the formula for the mean?

EDIT. Up until now, I’ve received answers that are just perfectly good explanations of how to derive the formula for the variance of $K$ in ways that differ from the one presented above. That’s not what I’m asking for. The ideal answer should contain as few formulae as possible, and use simple enough words to explain not why the formula is mathematically true, but why it’s reasonable and couldn’t possibly be otherwise – something like the intuitive explanation for $\langle K\rangle = np$ that I gave above.

$\endgroup$
6
  • 6
    $\begingroup$ Nice question, +1. If someone comes up with a good answer, that could have implications for the teaching of probability and statistics. $\endgroup$ Jun 21, 2018 at 13:08
  • 3
    $\begingroup$ That’s also part of the reason why I’m asking it. It is so much better to introduce these concepts to the uninitiated first through intuitive reasoning, then through formalism; however this one formula in particular seems to survive any attempt to give it a sensible explanation in words. $\endgroup$
    – giobrach
    Jun 21, 2018 at 13:17
  • 3
    $\begingroup$ I think it's intrinsically harder to justify a variance formula in an intuitive sense, as opposed to a mean value. At first glance you can justify some of the implications of the formula, such as the variance should be zero when $p=0,1$ and you should have maximum variance when $p=0.5$. I'm not saying it's impossible though, I'm genuinely interested in seeing such a justification. $\endgroup$
    – Paul
    Jun 21, 2018 at 13:43
  • 1
    $\begingroup$ Shouldn't $\binom{n}{k}p^k(1-p)^k$ be $\binom{n}{k}p^k(1-p)^{n-k}$? $\endgroup$
    – robjohn
    Aug 11, 2019 at 3:30
  • 1
    $\begingroup$ @giobrach: I fixed one more occurrence. $\endgroup$
    – robjohn
    Mar 24, 2021 at 22:18

6 Answers 6

4
$\begingroup$

A binomial random variable is a sum of independent Bernoulli random variables. So if you accept that a Bernoulli random variable has variance $p(1-p)$, then the formula for the variance of a Binomial random variable follows from the "variance of sum" rule. Moreover, the variance of a Bernoulli random variable can be seen at a glance using the formula $\text{Var}(X) = E(X^2) - E(X)^2$.

$\endgroup$
3
  • 3
    $\begingroup$ I literally included a complete derivation of the formula in the question itself. As per the title, I’m interested in the intuition behind it, not the mathematical steps necessary to its justifcation. So, why should I accept that formula for Bernoulli variance? Why is it intuitively obvious? $\endgroup$
    – giobrach
    Jun 21, 2018 at 13:19
  • 1
    $\begingroup$ I would also like to see a more direct intuitive explanation, which hopefully somebody else will be able to provide, but with this approach it's at least possible to see the formula at a glance. $\endgroup$
    – littleO
    Jun 21, 2018 at 13:23
  • 1
    $\begingroup$ @littleO For the individual Bernouilli random variable $B$ it should be no surprise that the variance is $0$ when $p=0$ or $p=1$ (you are almost certain what the value is going to be) or that it is maximised when $p=\frac12$. As for calculating it, $B^2=B$ so $E[B^2]=E[B]=p$ so the variance is $E[B^2]-(E[B])^2=p-p^2=p(1-p)$ and this looks highly plausible. Add together $n$ i.i.d. of then and the variance intuitively increases with $n$ (no offsets) so $np(1-p)$. You may not find this immediately intuitive, but I find the three components of $n$, $p$ and $(1-p)$ completely reasonable $\endgroup$
    – Henry
    Jun 21, 2018 at 14:00
3
$\begingroup$

I understand the logic to be the same in the case of Bernoulli random variables as in the case of Binomial r.v.s because of independence. In the case of Bernoulli r.v.s, just as in the case of the binomial, the concept of variance is implicit in the notion of expected value itself. That is, the more the expected value per trial deviates from 1 or 0, the more we have variability in our results. Variability is maximised the less we can say, either, that we have all ones or all zeros, and the closer we get to equal ones and equal zeros. This occurs as the mean value, p, approaches 1/2. When our expected value is neither one nor zero (that is, when there is randomness in our outcomes) we start to record distances of x from the mean value. Our distance from the expected value is the same as our expected value when x=0, when our outcome does not occur, which occurs with a probability (1-p). At the same time, our distance can be the complement of this with the complementary probability. Calculating the expected difference of x from its mean means taking account of both distances and their probabilities which, because of symmetry, yields a duplication of the product of p (the expected value of x), and the probability of the non occurrence of the outcome. This symmetry recalls the fact that distances from x = 0 or x = 1 are equally relevant to the variability, because they equally prescribe the deviation from certainty we have in relation to the event, however the contribution they provide to this sum has in both instances to be tempered by the level of certainty we can have in relation to them (their own probability). The squaring of complement terms necessary to calculate the variance reduces this duplicate to one. Thus the variance is simply the expected value of x and the probability of the non occurrence of the outcome.

$\endgroup$
3
  • 2
    $\begingroup$ Please edit and use MathJax to properly format math expressions. $\endgroup$ Aug 11, 2019 at 2:53
  • 1
    $\begingroup$ ...around so. I made an example. $\endgroup$
    – peterh
    Aug 11, 2019 at 3:09
  • 1
    $\begingroup$ Thanks. I'm still learning all of this, will give it a go. $\endgroup$ Aug 11, 2019 at 7:58
1
$\begingroup$

You have $Var(X+Y) = Var(X)+Var(Y)$ for i.i.d vars and we have $$ K = \sum I_i, $$ With i.i.d $I_i = 1$ because the nubber of succcesses is has the same as distribution as the number of times $I_i$ is 1 and they sum the the number of successes.

Now with probability $p$ $I_i=1$ and zero with probability $1-p$. and also $$ Var(I_i) = p(1-p) $$ Once can motivate this formula by noting the variance is quadratic in $p$ and if almost all mass is at 1 then we have almost zero variance and if we have almost all mass at 0 then symmetrically we have almost zero variance from there you get the formula. The n is from the principle that variance of a sum of independent variables is the same as the sum of the individual variance.

$\endgroup$
1
  • 2
    $\begingroup$ See my comment to littleO’s answer. $\endgroup$
    – giobrach
    Jun 21, 2018 at 13:22
0
$\begingroup$

For any distribution,

$$E((X-E(X))^2)=E(X^2)-E(X)^2.$$

The variance is the second order moment from which you subtract the squared average.

In a Bernouilli scheme ($B=0/1$ drawings), $B^2=B$ so that

$$\text{Var}(B)=E((B-E(B))^2)=E(B)-\overline E(B)^2=p-p^2=pq$$

and

$$\text{Var}\left(\sum B\right)=npq.$$

$\endgroup$
3
  • 5
    $\begingroup$ This doesn't answer the question. $\endgroup$
    – Paul
    Jun 21, 2018 at 14:01
  • 1
    $\begingroup$ @Paul: please substantiate this comment. $\endgroup$
    – user65203
    Jun 21, 2018 at 14:08
  • 1
    $\begingroup$ Take a look at OP's edit. $\endgroup$
    – Paul
    Jun 21, 2018 at 14:32
0
$\begingroup$

I am not a statistician and never got into calculus stuff. I have always used intuition to make sense of statistics in social science and I will try to explain how I conceptualize the formula for variance. Also a short thank you @Mathem-antics whose answer helped me write this out better.

If P is the probability of an event occurring (i.e. my response variable being, Y=1) and we already established that the mean of the distribution is E(Y)= P, what is the variance?

To intuitively answer that let's think of two examples:

When is variance at its maximum?

Randomness from the toss of a coin, when the probability of getting head/tail is equal= 0.5 and 0.5

When is it at its lowest?

When we always get same result. Such as probability of a Cambridge grad passing an elementary school exam, here probability of passing is 100 and failing is 0. Y=1 (which is passing) will always happen. Therefore the variance ought to be 0 here.

Now let's understand the spectrum in which this occurs

When we move away from toss of a coin scenario (where both P and 1- P is 0) and go in either direction, there is a pattern and variance starts to decrease. For all combinations of P and 1- P, (from a spectrum of 1-0) the maximum possible variance occurs during the 50/50 scenario and lowest in 100/0 scenario. So the formula of variance must be designed such that it captures this information. And the best way to do that is through multiplication, if you look at the following table:

probability table

Therefore, maximum variance is captured at p=0.5, which is expected, and starts decreasing and drops to 0 when p=1 or 0. Therefore, intuitively it makes sense to multiply them, because the moment you see 0.25 you know it is completely random or when you 0.18 you know that there is definitely a pattern in the distribution, one of the events is more likely to happen and the distribution is dense around that. This is how I see it anyways.

Since I don't have deep understanding of stats there may be some technical error in the phrasing of my sentences, I apologize for that in advance.

$\endgroup$
0
$\begingroup$

I recently asked myself identically this question. The conclusion I came to is that the variance is a function of the mean because the mean is on a bounded interval. That is, obviously/by definition, half of the outcomes have to be above the mean. When the mean approaches 1 (or likewise, as it approaches 0), it becomes harder to have a large variance, because the outcomes that would have been required to "average out" the lower valued outcomes simply cannot exist because of the interval on which the values are defined. Graphically, these are the distributions you get at different means (here the "means" are the 'r_obs' values defined in the legend). For the blue one (u = 0.05) to have a larger variance, since it has to stretch out symmetrically because of the particular model in question, you can see it would have to stretch past 0 to the left, which is not allowed.

enter image description here

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .