Intuition for Standard Deviation

Question

I've been studying descriptive statistics and am having a hard time understanding the actual intuition behind standard deviation. I'm trying to get a practical feeling for it and so I'm trying to draw conclusions from it using a distribution of 20 numbers, from 1 to 20. I know the mean is 10.5 and the absolute average deviation is 5, which is pretty intuitive.

Now when taking the standard deviation I get the value 5.77 which still makes some sense if I think about it as the average euclidean deviation from the mean. So I imagine adding orthogonal distances and then averaging them $\frac{\sum(x_i-\bar x)^2}{n}$ and taking the square root of that at the end to get the actual average distance. The formula makes sense from an euclidean perspective. So all that being said, my questions:

1) Why would an euclidean average distance be more accurate than an absolute deviation from the mean? I actually think absolute average deviation is more accurate since it doesn't infer any direction of the values. When taking the euclidean distance, I'm pretty much saying every value is placed at a 90° angle from each other. That does not sound right. So why the Euclidean distance? (I'm aware of this article but If someone could actually explain what efficiency is that would be very helpful: https://www.leeds.ac.uk/educol/documents/00003759.htm)

2) If The advantage of using SD is because of all the math we have developed around normal distribution shapes (68%, 95%, 99,7%...) wouldn't it be better to just rewrite that model with the new average deviation?

3) Ill probably post another question in the future about this, but when calculating standard error, this standard deviation seems to get even worse, since we need corrections for finite populations. Does this make any sense?

Answer 1

Given a sample $X_1, X_2, \dots, X_n,$ suppose we make a 'stripchart' (also called 'dotplot') of the data. Shown below are stripcharts of three samples of size $n=5:$ $$X = (0, 2, 2, 2, 4),\, Y = (0, 1, 2, 3, 4),\, Z = (0, 0, 2, 4, 4).$$

The sample mean $\bar X = \frac{1}{n}\sum_{i=1}^n X_i$ can be considered as the 'balance point' or 'center of gravity' of the stripchart of a sample $X_i$ (where all dots have the same weight). The sample mean $\bar X$ is also a good estimate of the mean $\mu_X$ of the population from which the sample is randomly chosen. In particular, $E(\bar X) = \mu_X.$ In the figure, $\bar X = \bar Y = \bar Z = 2.$

One way to measure the dispersion or variability of a sample is its range $R = \max(X_i) - \min(X_i).$ For our three small datasets $R_X = R_Y = R_Z = 4,$ so the sample range is not an effective measure of dispersion for distinguishing among our three datasets.

Yet, it seems intuitively clear that the $Z_i$s are the most disperse and the $X_i$s are the least disperse. As a physical model think of a vertical shaft or axis at 2, around which a stripchart is to be spun. The moments-of-inertia increase as we move from the $X_i$ to the $Y_i$ to the $Z_i.$ Roughly speaking, that is to say, the $Z_i$ make the best flywheel.

A measure of dispersion that closely parallels the definition of moment-of-inertia is the sample variance. It is roughly the average of the squared deviations about the mean: $S_X^2 = \frac{1}{n-1}\sum_{i=1}^n (X_i - \bar X)^2.$ For our three examples, $S_X^2 = 2.0,\, S_Y^2 = 2.5,\,$ and $S_Z^2 = 4.$ So the variances do increase appropriately as we move from the $X_i$ to the $Y_i$ to the $Z_i.$ [The positive square root $S_X$ of the sample variance $S_X^2$ is called the 'sample standard deviation', which helps to explain the notation.]

One reason for using $n-1$ instead of $n$ in the denominator of the sample variance is to make $S_X^2$ a good estimator of the population variance $\sigma_X^2.$ In particular, $E(S_X^2) = \sigma_X^2.$ [Roughly speaking, another reason is that $X$ can be viewed as a vector in 5-dimensional space; one dimension is "used" to estimate $\mu$ by $\bar X,$ leaving $n-1 = 4$ dimensions to estimate $\sigma^2.$ To make this idea precise requires a side trip into linear algebra that I won't take here.]

Note: A variety of other potential measures of sample dispersion, including the one your mention, have been advocated by respected practitioners. Several of them use absolute deviations from the center (sometimes called 'discrepancies'). Examples are $\frac{1}{n}\sum_{i=1}^n |X_i - \bar X|,\; \frac{1}{n}\sum_{i=1}^n |X_i - H_X|,$ and the sample median of $|X_i - H_X|,$ where $H_X$ is the sample median. At various times and places both of the first two have been called 'MAD' (for Mean Absolute Deviation).

While each of these may have advantages in specific applications, none of them is in widespread regular use. Objections include the difficulty of doing proofs (absolute values can lead to the need to consider cases) and the lack of easily accessible distribution theory (except perhaps via simulation). [By contrast, for normal data, $\frac{(n-1)S_X^2}{\sigma_X^2} \sim \mathsf{Chisq(\nu = n-1)},$ a widely tabled and programmed family of distributions.]

Also, population variances have the advantage that $Var(X + Y) = Var(X) + Var(Y),$ provided that $X$ and $Y$ are uncorrelated.