91
$\begingroup$

Equivalently, about variance?

I realize it measures the spread of a distribution, but many other metrics could do the same (e.g., the average absolute deviation). What is its deeper significance? Does it have

  • a particular geometric interpretation (in the sense, e.g., that the mean is the balancing point of a distribution)?
  • any other intuitive interpretation that differentiates it from other possible measures of spread?

What's so special about it that makes it act as a normalizing factor in all sorts of situations (for example, convert covariance to correlation)?

$\endgroup$
10
  • 13
    $\begingroup$ Have you heard the term "moment?" The variance is the second moment about the mean. See HERE $\endgroup$
    – Mark Viola
    Jan 12, 2019 at 20:44
  • 3
    $\begingroup$ Possible duplicate of Intuition behind Variance forumla $\endgroup$ Jan 13, 2019 at 16:27
  • 2
    $\begingroup$ @MarkViola And? Variance can be generalized, therefore it's meaningful? $\endgroup$
    – Jack M
    Jan 14, 2019 at 9:16
  • 4
    $\begingroup$ The absolute value deviation is a perfectly valid measure of deviation. However absolute values are very hard to work with analytically, squares are much easier. That's one answer: calculabillity. $\endgroup$
    – Winther
    Jan 14, 2019 at 10:23
  • 2
    $\begingroup$ Your question could have been "is standard deviation a natural concept ?" : the best proof that it wasn't evident is that it has been "discovered" around 1900, which is very late for a "mathematical being", moreover with a simple definition. $\endgroup$
    – Jean Marie
    Jan 14, 2019 at 13:08

10 Answers 10

94
$\begingroup$

There's a very nice geometric interpretation.

Random variables of finite mean form a vector space. Covariance is a useful inner product on that space. Oh, wait, that's not quite right: constant variables are orthogonal to themselves in this product, so it's only positive semi-definite. So, let me be more precise - on the quotient space formed by the equivalence relation "is a linear transformation of", covariance is a true inner product. (If quotient spaces are an unfamiliar concept, just focus on the vector space of zero-mean, finite-variance variables; it gets you the same outcome in this context.)

Right, let's carry on. In the norm this inner product induces, standard deviation is a variable's length, while the correlation coefficient between two variables (their covariance divided by the product of their standard deviations) is the cosine of the "angle" between them. That the correlation coefficient is in $[-1,\,1]$ is then a restatement of the vector space's Cauchy-Schwarz inequality.

$\endgroup$
14
  • 10
    $\begingroup$ Interesting approach. Is it a personal interpretation or a standard one? If it's standard, are there any resources you can provide? I haven't seen it in any book... $\endgroup$
    – blue_note
    Jan 12, 2019 at 21:03
  • 6
    $\begingroup$ @blue_note You're most likely to encounter it in a discussion of regression, since regressing $Y$ against $X$ writes $Y$ as a multiple of $X$, plus a variable orthogonal to $X$ in this sense. In fact, the coefficients involved in such an expression square to the proportion of variance explained. This has a well-understood connection to probability in quantum mechanics. But really, any source that explains why there's a $^2$ in $R^2$ will at least hint at these ideas. $\endgroup$
    – J.G.
    Jan 12, 2019 at 21:06
  • 4
    $\begingroup$ Can someone provide a concrete example or other similar dumbing down of this answer? $\endgroup$ Jan 13, 2019 at 4:43
  • 3
    $\begingroup$ A paragraph on wikipedia about it @blue_note $\endgroup$ Jan 13, 2019 at 8:59
  • 20
    $\begingroup$ I need a 3Blue1Brown video on this. $\endgroup$
    – RcnSc
    Jan 14, 2019 at 13:31
16
$\begingroup$

I take it as unproblematic that the standard deviation is important in the normal distribution since the standard deviation (or variance) is one of its parameters (though it could doubtless be reparameterized in various ways). By the Central Limit Theorem, the normal distribution is in turn relevant for understanding just about any distribution: If $X$ is a normal variable with mean $\mu$ and standard deviation $\sigma$, then for large $n$

$$\frac{\overline{X} - \mu}{\frac{\sigma}{\sqrt{n}}}$$

is approximately standard normal. No other measure of dispersion can so relate $X$ with the normal distribution. Said simply, the Central Limit Theorem in and of itself guarantees that the standard deviation plays a prominent role in statistics.

$\endgroup$
3
  • $\begingroup$ Related question to this: The role of variance in Central Limit Theorem $\endgroup$
    – Winther
    Jan 14, 2019 at 10:45
  • 5
    $\begingroup$ No other measure of dispersion can so relate $X$ with the standard normal distribution, but that's only because the standard normal distribution is defined to have unit variance. If we defined it to have unit interquartile range instead, then for large $n$ we would say that $$\frac{\overline{X}-\mu}{IQR/\sqrt n}$$ is approximately standard normal. $\endgroup$ Jan 14, 2019 at 18:28
  • $\begingroup$ @MishaLavrov Good point (of the sort that I was alluding to in my parenthetical about reparameterization) but if you regard $\sigma$ in the normal distribution to be a good measure of dispersion then the Central Limit Theorem gives you a reason to use it as a measure of dispersion in other distributions. I don't think that appeal to CLT is decisive, but it should be part of the discussion about the importance of the standard deviation. $\endgroup$ Jan 14, 2019 at 19:13
3
$\begingroup$

An interesting feature of the standard deviation is its connection to the (root) mean square error. This measures how well a predictor does in predicting the values. The root mean square error of using the mean as a predictor is the standard deviation, and this is the least root mean square error that you can get with a constant predictor.

(This, of course, shifts the question to why the root mean squared error is interesting. I find it a bit more intuitive than the standard deviation, though: you can see it as the $L_2$ norm of the error vector, corrected for the number of points.)

$\endgroup$
1
  • 1
    $\begingroup$ Good point. However, it indeed shifts the question. Although I can see that in a vector space, in a standard 2D plot of (X, Y) pairs I can see what the variance is on the eg. horizontal axis $\endgroup$
    – blue_note
    Jan 13, 2019 at 11:34
2
$\begingroup$

When defining "standard deviation", we want some way to take a bunch of deviations from a mean and quantify how big they typically are using a single number in the same units as the deviations themselves. But any definition of "standard deviation" induces a corresponding definition of "mean" because we want our choice of "mean" to always minimize the value of our "standard deviation" (intuitively, we want to define "mean" to be the "middlemost" point as measured by "standard deviation"). Only by defining "standard deviation" in the usual way do we recover the arithmetic mean while still having a measure in the right units. (Without getting into details, the key point is that the quadratic becomes linear when we take the derivative to find its critical point.)

If we want to use some other mean, we can of course find a different "standard deviation" that will match that mean (the progress is somewhat analogous to integration), but in practice it's just easier to transform the data so that the arithmetic mean is appropriate.

$\endgroup$
1
  • $\begingroup$ If all you want is to minimization at the mean and the right units, why not sum/integrate the magnitude of the deviations? $\endgroup$ Jan 13, 2019 at 2:36
2
$\begingroup$

The normal distribution has maximum entropy among real distributions supported on $(-\infty, \infty)$ with specified standard deviation (equivalently, variance). (Reference.) Consequently, if the only thing you know about a real distribution supported on $\mathbb{R}$ is its mean and variance, the distribution that presumes the least prior information is the normal distribution.

I don't tend to think of the statement above as the important fact. It's more: normal distributions appear frequently and knowing the location parameter (mean) is reasonable. So what else do I have to know to make the least presumptive model be the normal distribution? The dispersion (variance).

$\endgroup$
2
$\begingroup$

The following is from An Introduction to Probability Theory and Its Applications, Vol. 1 by W. Feller.

From Section IX.4: Variance

  • Some readers may be helped by the following interpretation in mechanics. Suppose that a unit mass is distributed on the $x$-axis so that the mass $f(x_j)$ is concentrated in $x_j$. Then the mean $\mu$ is the abscissa of the center of gravity, and the variance is the moment of inertia.

  • Clearly different mass distributions may have the same center of gravity and the same moment of inertia, but it is well known that some important mechanical properties can be described in terms of these two quantities.

$\endgroup$
2
$\begingroup$

If you draw a random sample from a normal distribution with mean $\mu$ and variance $\sigma^2$ then the mean and variance of the sample are sufficient statistics. This means that these two statistics contain all the information in the sample. The distribution of any other statistic (function of the observed values in the sample) given the sample mean and variance is independent of the true population mean and variance.

For the normal distribution the sample variance is the optimal estimator of the population variance. For example the population variance could be estimated by a function of the mean deviation or by some function of the order statistics (interquartile range or the range) but the distribution of that estimator would have a greater spread than the sample variance.

These facts are important as, following the central limit theorem, the distribution of many observed phenomena is approximately normal.

$\endgroup$
2
$\begingroup$

Probably the most useful property of the variance is that it it additive: the variance of the sum of two independent random variables is the sum of the variances.

This does not occur with other estimators of the spread.

$\endgroup$
1
$\begingroup$

Consider Casella/Berger, Statistical Inference, Section 10.3.2:

Theorem 10.3.2: Consider a point estimation problem for a real-valued parameter $\theta$. In each of the following two situations, if $\delta^\pi \in D$ then $\delta^\pi$ is the Bayes rule (also called the Bayes estimator).

a. For squared error loss, $\delta^\pi (x) = E(\theta | x)$.

b. For absolute error loss, $\delta^\pi (x) = \text{median of } \pi(\theta | x)$.

My interpretation of this is that using standard deviation leads one in the direction of an estimator for the mean; whereas using average absolute deviation leads one in the direction of an estimator for the median.

$\endgroup$
0
$\begingroup$

Here are a few comments:

First off, I think it should be acknowledged that the OP's question is well-warranted, and it is the subject of an ongoing debate in a variety of circles (abstract-mathematical or otherwise). See e.g. https://stats.stackexchange.com/q/118/297795 or https://mathoverflow.net/q/1048/66883.

Next, the question seems to me to be somewhat multifaceted, in that it is not at all obvious to me that "having deeper significance" and "having intuitive interpretation" are positively correlated. Granted, both of these are vague terms to begin with. On that note I'll try to address both of them separately.

I believe it is reasonable (from the point of view of abstract mathematics) to interpret "having deeper significance" to be related to a question of the type "What property or list of properties of the (nonlinear) functional called the variance determines it among some larger class of functionals, and up to which relation?" (think the characterization of the Laplacian among all constant coefficient linear differential operators). It is straightforward that this question too is too vague and highly area-dependent (not only in how it can be answered but even in its formulation); below I'll list some examples.

As for an intuitive interpretation, here is one way to think of it. Let us take for granted (as it's done traditionally) that the variance of a random variable ought to be its covariance with itself:

$$\operatorname{var}(X) = \operatorname{cov}(X,X).$$

The traditional formula for covariance is:

$$\operatorname{cov}(X,Y) = \mathbb{E}((X-\mathbb{E}(X))(Y-\mathbb{E}(Y))) = \mathbb{E}(XY) - \mathbb{E}(X)\mathbb{E}(Y).$$

Reverse engineering the first expression leads to a Hilbert space interpretation as in the answer of J.G.. The caveat is the artifact of the introduced quotient space (What makes "$X\sim Y \iff Y=aX+b$ for some $a,b\in\mathbb{R}, a\neq0$" special, from a Hilbert space theory point of view?).

I'd like to propose reverse engineering the second expression, which leads to a quasi-homomorphism interpretation (in the sense of http://perso.ens-lyon.fr/ghys/articles/groupscircle.pdf, p. 349). Indeed, let $(\Omega,\mathcal{F},\mathbb{P})$ be a probability space and consider the space $L^0((\Omega,\mathcal{F},\mathbb{P});\mathbb{R})$ of real valued random variables defined on it. This space is a (topological) vector space; and pointwise multiplication makes it a (topological) commutative algebra. Further we have the (partially defined) expectation operator $\mathbb{E}$ determined by $\mathbb{P}$:

$$\mathbb{E}: L^0((\Omega,\mathcal{F},\mathbb{P});\mathbb{R})\rightsquigarrow \mathbb{R}, X\mapsto \int_{\Omega} X(\omega)\, d\mathbb{P}(\omega).$$

("$\rightsquigarrow$" means "partially defined": Notation for "function from a subset of $X$ into $Y$"?)

When $X$ and $Y$ have finite expectation, we have that $\mathbb{E}(X+Y) - [\mathbb{E}(X)+ \mathbb{E}(Y)] = 0$ (which is a convoluted way of saying that expectation is linear), but even when all constituents are finite it is not the case that $\mathbb{E}(XY) - [\mathbb{E}(X) \mathbb{E}(Y)] = 0$ always; the extent to which this fails is exactly covariance. Specializing to variance we get that $\operatorname{var}(X)$ tells to which extent $\mathbb{E}$ fails to be multiplicative when restricted to the algebra generated by $X$.

I should note that I find this to be intuitive from the point of view of probability theory, in the sense that multiplicative properties of $\mathbb{E}$ (or of $\mathbb{P}$) (i.e. some form of independence) are at the core of probability theory, and it's what distinguishes probability theory from abstract measure theory. (I believe I read a similar statement in one of Terry Tao's blogs, which influenced this interpretation, but I don't remember which one.)

Of course it would be disingenuous of me to pretend that there are no caveats with this reverse engineering too (beyond the fact that it is reverse engineering). Here are some that comes to mind:

  • I don't think it is straightforward that variance and covariance (or correlation) ought to be related (or ought to be related the way they are), as most answers seem to be assuming (Of course one could also question the relation between standard deviation and variance, but this questioning seems to me to be significantly more pedantic.). In functional analytical language, one could prefer to consider Banach space theory with no reference to Hilbert space theory. Case in point: the covariance of $X$ and $Y$ "intuitively" (from the statistical point of view) says how the change in $X$ and the change in $Y$ are related; how is the variance of $X$ not a priori $1$, "intuitively"? (This reasoning is not completely off; indeed it leads to the correlation coefficient). Observe that of course the traditional definition of covariance determines the traditional definition of variance and vice versa (by polarization); what I am pointing out here is different.

  • In regards to the (first instance of) "intuitively" of the previous bullet I'd like to point out the paper "Co-Relations and Their Measurement, Chiefly from Anthropometric Data" by Galton which the contemporary statistics community seems to take as the starting point of the concept of correlation (see e.g. Stigler's "Francis Galton's Account of the Invention of Correlation"). I found the Galton paper valuable beyond historical value, since he is trying to justify why the mathematical gadget he introduces quantifies the mathematically vague concept he is interested in. What is not emphasized is that what Galton calls correlation is actually median absolute deviance from median. I haven't looked deeper into the statistics literature, but there are obvious inquiries required by this. Still, dismissively one could say there is nothing special about any of this (assuming the import of a concept from biology/genetics into mathematics approximates some intuitional geodesic). For more on this line of thought the book Modeling, Measuring and Managing Risk by Pflug and Romisch seems to be valuable.

  • There is plenty of stuff that I brush under the rug by taking expectation to be a partially defined operator. In particular, the behavior of $\mathbb{E}$ under $k$-tuple products are not clear (to me at least) from its behavior under double products. Though this interpretation could lead to moments (like many others commented) and more sophisticated tools.

  • On that note, what ought to be the covariance of three random variables $X,Y,Z$? A Hilbert space interpretation would lead one to think of a $3\times 3$ covariance matrix with entries binary covariance (similar to "matrix coefficients"), and this seems to be common in statistics/probability. From a dynamical point of view (which happens to be my point of view) it seems $\mathbb{E}(XYZ) - [\mathbb{E}(X)\mathbb{E}(Y)\mathbb{E}(Z)]$ (similar to "multiple mixing") is more interesting. $\mathbb{E}((X-\mathbb{E}(X))(Y-\mathbb{E}(Y))(Z-\mathbb{E}(Z)))$ is also valid to consider of course.

  • Independence and vanishing covariance are not the same as is well known; I hope I am not being misleading when I say "some form of independence" above. Related to this is the paper by Renyi I mention below.


  • Now let us discuss the deeper significance of variance when interpreted in the way I did above. More explicitly, the question is of the following form: let $(\Omega,\mathcal{F},\mathbb{P})$ be a probability space, and $\mathcal{C}\subseteq L^0((\Omega,\mathcal{F},\mathbb{P});\mathbb{R})$ be a collection random variables whose variance is defined. $\mathcal{C}$ is further assumed to be closed under whatever operations one might need to be syntactic. Let us take an arbitrary (possibly nonlinear) functional $\mathfrak{F}:\mathcal{C}\to \mathbb{R}$. What properties imposed on $\mathfrak{F}$ would guarantee the existence of a function $\mathfrak{f}:\mathbb{R}\to\mathbb{R}$ such that $\mathfrak{F} = \mathfrak{f}\circ \operatorname{var}$? Further, which properties of $\mathfrak{F}$ would guarantee which properties of $\mathfrak{f}$? The way I interpret this is that $\mathfrak{f}$ is some deterministic dependency of the anonymous functional $\mathfrak{F}$ on $\operatorname{var}$. As an example, for $\mathfrak{F}: X\mapsto \operatorname{var}(aX+b)$, $\mathfrak{f}: x\mapsto a^2x$ does the job uniquely (i.e. $\operatorname{var}(aX+b) = a^2\operatorname{var}(X)$).

In this framework the first paper I'd like to mention is "Why the variance?" by Kagan and Shepp. They consider functionals of the form

$$\mathfrak{D}_{\phi,a}: X\mapsto \mathbb{E} (\phi(X-a)),$$

where $\phi\in C^0(\mathbb{R},\mathbb{R})$ and $a\in\mathbb{R}$. Observe that $\mathfrak{D}_{|\cdot|^2,\mathbb{E}(X)}(X)=\operatorname{var}(X)$. For arbitrary $\phi$ (and $X$), $a\mapsto \mathfrak{D}_{\phi,a}(X)$ need not have a minimum; even when it does have a minimum said minimum need not be $a=\mathbb{E}(X)$. Using a terminology from a follow-up paper by Fainleib titled "On a characterization of measures of dispersion", let us define the $\phi$-base $\mathbb{B}_\phi(\mathbb{P})$ of $\mathbb{P}$ to be the following collection of random variables:

$$\mathbb{B}_\phi(\mathbb{P})=\left\{X\in L^0(\Omega;\mathbb{R})\left\vert \min_{a\in\mathbb{R}}\mathfrak{D}_{\phi,a}(X) = \mathfrak{D}_{\phi,0}(X)\right.\right\}.$$

(It is instructive to consider the cases $\phi = \operatorname{id}_\mathbb{R}$, $\phi = $ constant, $\phi = |\cdot|$, $\phi= |\cdot|^2$. The $\phi$-bases are: nothing, everything, random variables with vanishing median w/r/t $\mathbb{P}$ and random variables with vanishing mean w/r/t $\mathbb{P}$, respectively.)

Also observe that we could have been explicit with the dependency of $\mathfrak{D}_{\phi,a}$ on $\mathbb{P}$ from the get-go, if we work with honest to heavens measurable functions (and not their equivalence classes modulo negligible sets).

Theorem (Kagan-Shepp): If $\phi\in C^0(\mathbb{R},\mathbb{R})$ is such that all bounded random variables with vanishing mean w/r/t $\mathbb{P}$ are in $\mathbb{B}_\phi(\mathbb{P})$, then $\phi(x)= Ax^2+B$ for some $A\in\mathbb{R}_{\geq0}$ and $B\in \mathbb{R}$.

I should note that they in fact prove the result for vector-valued bounded random variables (with the obvious modification to the conclusion).

It is a straightforward algebraic manipulation to see that for $\phi:x\mapsto Ax^2+B$ with $A>0$,

$$\mathfrak{D}_{|\cdot|^2,a}(X) = A\,(a- \mathbb{E}(X))^2 + \left[\mathfrak{D}_{|\cdot|^2,0}(X)- A\,(\mathbb{E}(X))^2 \right].$$

In particular the minimum of $a\mapsto \mathfrak{D}_{\phi,a}(X)$ is $\mathfrak{D}_{|\cdot|^2,0}(X)- A\,(\mathbb{E}(X))^2$, said minimum is attained exactly when $a=\mathbb{E}(X)$, and finally the $\phi$-base $\mathbb{B}_\phi(\mathbb{P})$ of $\mathbb{P}$ is exactly the collection of random variables with zero mean w/r/t $\mathbb{P}$ (i.e. the kernel of $\mathbb{E}$).

This shows that, as the authors declare, "the second moment is essentially the only characteristic defined for all bounded random variables $X$ that is minimized when taken around $a=\mathbb{E}(X)$", and the minimum value is exactly $\operatorname{var}(X)$.

About twenty years after the Kagan-Shepp paper a similar result, but with an emphasis on the additivity property of variance for independent random variables, is obtained by Poschadel in the paper "On a characterization of variance and covariance".

  • An alternative framework is used by Mattner in the paper "What are cumulants?". His emphasis is on the semigroup structure of the space $\operatorname{Prob}(\mathbb{R},\mathcal{B})$ of Borel probability measures on the real line given by convolution. Using the $k$-th derivative of (some branch of) logarithm applied to the characteristic function ( = Fourier transform) of a probability measure $\mathbb{P}$ he defines the $k$-th cumulant of $\mathbb{P}$, and the $2$nd cumulant of $\mathbb{P}$ turns out to be exactly the variance of $\mathbb{P}$. Here it is proved, among other things, that (modulo some details I don't want to get into here) the only continuous additive maps on the convolution semigroup of probabilities (actually the subsemigroup of probabilities with "finite moments of all orders") are constant multiples of variance. Observe that this fits into the more general framework I mentioned above, if we impose the condition on $\mathfrak{F}$ that it be "version-independent", i.e. if $X$ and $Y$ have the same law ( i.e. if $X_\ast(\mathbb{P})=Y_\ast(\mathbb{P})$, where the lowerscript $\ast$ signifies the pushforward), then $\mathfrak{F}(X)=\mathfrak{F}(Y)$ (See the book of Pflug and Romisch I mentioned above for more on version-independent "deviation-type" functionals, from the statistical point of view).

  • Finally let me mention the paper "On Measures of Dependence" by Renyi. He focuses on alternatives to the correlation coefficient, and he surveys a variety of gadgets according to whether or not each one of these gadgets satisfies the postulates he puts forth for a reasonable measure of dependence ought to satisfy. The traditional correlation coefficient (or its absolute value) fails to satisfy all said postulates, but the so-called maximal correlation, introduced by Gebelein, satisfies all.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.