1
$\begingroup$

Say we have n terms in SST, $x_1,x_2,\ldots,x_n$.

The reasoning that I have seen everywhere is that degree of freedom is $n-1,$ so we divide by $n-1.$

$$\mathrm{MST} = \frac{x_1+x_2+\cdots+x_n}{n-1}$$

My reasoning is that, if degree of freedom is $n-1$, consider $n-1$ "free" terms at a time, then find the expected value

\begin{align} m_1 & =\frac{x_2+\cdots+x_n}{n-1} \\ m_2 & =\frac{x_1+x_3+\cdots+x_n}{n-1} \\ & \,\,\,\vdots \\ m_n & =\frac{x_1+x_2+\cdots+x_{n-1}}{n-1} \\[10pt] E(m) & = \frac{\sum_{i=1}^n m_i}{n} \\[6pt] & = \frac{\frac{(n-1)(x_1+x_2+\cdots+x_n)}{n-1}}{n} \\[6pt] & = \frac{x_1+x_2+\cdots+x_{n}}{n} \end{align}

But we don't divide the summation by $n$ in actual formula. Why?

Please explain it in terms of degree of freedom only.


Addendum: Additional questions from linked Comment:

  1. Thanks. Variance can be computed without referring to mean. Why do we assume that the variance will be computed using the mean?

  2. Also when you refer to variance being chi-squared distribution of (n-1) degree of freedom, are we not self referencing, using degree of freedom to explain degree of freedom? Or are the degree of freedom of chi-squared dist. diff. from that of variance?

  3. About the subspace thing. I will have to read it up. I was half asleep. Woke up on notification. Tried looking it up, couldn't grasp. I will take a look at them tomorrow. If you have any links for reading on the matter, please share.

  4. My original question was not "questioning the degree of freedom", I just assumed it to be true. But why do we divide it by the said degree of freedom when calculating the MST? The reasoning I have read online is that the "non free term" is not really contributing, which is why I proposed why don't we drop the non contributing term, and calculate the MST using the method I proposed in the question?

Thanks.

$\endgroup$
  • $\begingroup$ Below I show simple examples of 'degrees of freedom' for the sample variance. In an ANOVA the fundamental idea is the same, but the formulas for finding degrees of freedom are a little messier. $\endgroup$ – BruceET Jun 13 '18 at 18:25
  • $\begingroup$ I edited your Question to include issues raised in your linked comment, so that everyone can see them. Of course, feel free to edit or remove them as you like. $\endgroup$ – BruceET Jun 13 '18 at 18:47
  • 1
    $\begingroup$ (1) Usual definition is $S^2 = \frac{1}{n-1}\sum_{i=1}^n (x_i - \bar x)^2.$ (2) If $Q = Z_1 + \cdots + Z_n,$ where $Z_i$ are indep std normal, then $Q \sim \mathsf{Chisq}(df=n),$ so not quite self-referential. (4) Needed for F-test: If $Q_1 \sim \mathsf{Chisq}(\nu_1)$ and (independently) $Q_2 \sim \mathsf{Chisq}(\nu_2),$ then $\frac{Q_1/\nu_1}{Q_2/\nu_2} \sim \mathsf{F}(\nu_1, \nu_2).$ See note in my Answer. $\endgroup$ – BruceET Jun 13 '18 at 21:21
  • $\begingroup$ @BruceET 1. According to en.wikipedia.org/wiki/Degrees_of_freedom_(statistics),most of the time the sample variance has N − 1 degrees of freedom, since it is computed from N random scores minus the only 1 parameter estimated as intermediate step, which is the sample mean), so does that imply, if we don't estimate sample mean as intermediate step, we will have N degree of freedom? $\endgroup$ – q126y Jun 14 '18 at 4:48
  • $\begingroup$ I don't think there is a way to compute the sample variance of $n$ observations without doing the equivalent of finding the mean so as to introduce constraint(s) resulting in $n-1$ degrees of freedom. It's a matter of the definition of the sample variance. // The note in my answer about ANOVA and @MichaelHardy's mention of regression both give examples of situations where the variance of something is estimated by a statistic that has degrees of freedom other than $n-1.$ $\endgroup$ – BruceET Jun 14 '18 at 7:50
1
$\begingroup$

Consider the data vector $(x_1, x_2, \dots, x_n)$ to be a vector in $n$-space. Specifying 'degrees of freedom' is shorthand for specifying the dimensionality of a sub-space. This is not always a concept that assists intuition. Here is about the simplest example I can think of. The table below shows the computation of the sample variance of $n = 4$ observations: $S^2 = \frac{1}{n-1}\sum_{i-1}^4 (x_i - \bar x)^2.$

     i   x   devn from mean   squared devn
     -------------------------------------
     1   3        0              0
     2   1       -2              4
     3   6        3              9
     4   2       -1              1
     -------------------------------------
  Total 12        0             14
  Sample mean: 12/4 = 3
  Sample var:  14/3

As soon at $\bar x = 3$ is computed, one says that a 'degree of freedom' has been 'lost'. Imagine that one of the four data rows of the table has not printed clearly. It can be reconstructed knowing the sample mean and the remaining $n-1 = 3$ observations.

Specifically, suppose the third row is too smudgy to read: then "3 + 1 + (smudge) + 2 = 12" so the smudge must be 6.

Technically speaking, the deviations $(x_i - \bar x)$ from the mean lie in a 3-dimensional subspace of the original 4-dimensional data space.

On a more advanced level, for normal data, the sample variance $S^2$ is an estimate of the population variance $\sigma^2.$ Then $Q = \frac{(n-1)S^2}{\sigma^2} \sim \mathsf{Chisq}(n-1),$ the chi-squared distribution with $n-1$ degrees of freedom. One use of this is that if values $L$ and $U$ cut 2.5% of the probability from the left and right tails of this distribution, respectively, then a 95% confidence interval (CI) for $\sigma^2$ is of the form $\left(\frac{(n-1)S^2}{U},\,\frac{(n-1)S^2}{L}\right).$

Example: Here are five observations from $\mathsf{Norm}(\mu=100, \sigma=15).$ A 95% CI for $\sigma^2$ is $(92.7, 2135),$ which contains $\sigma^2 = 225.$ A 95% CI for $\sigma$ is $(9.63, 46.20),$ which contains $\sigma = 15.$ For practical purposes, these are probably not useful confidence intervals, but that's the accuracy you can expect from only $n = 5$ observations. (Computations use R statistical software.)

x = round(rnorm(5, 100, 15));  x    # get sample of size 5 and print data
## 120 126  92  95  95
var(x)                              # find sample variance
258.3                               
qchisq(c(.025, .975), 4)            # find L and U for CHISQ(df=4)
##  0.4844186 11.1432868
CI = c(4*var(x)/11.143, 4*var(x)/0.484); CI
##   92.72189 2134.71074            # CI for population variance
sqrt(CI)
##  9.629221 46.202930              # CI for population standard deviation

Note: In a balanced one-factor ANOVA with $g$ groups and $r$ replications in each group: $\text{SS(G) + SS(E) = SS(Tot).}$ This is the Pythagorean Theorem in $ng-1$ dimensional space, where SS's are squared lengths of sides of a right triangle. Here $\text{DF(G)} = g-1,$ $\text{DF(E)} = g(r-1)$ and $\text{DF(Tot)} = gr -1.$ For each: SS/DF = MS, so $F = \text{MS(G)/MS(E)} \sim \mathsf{F}(g-1, g(r-1)).$ Also, $\frac{g(r-1)\text{MS(E)}}{\sigma^2}=\text{SS(E)}/\sigma^2 \sim \mathsf{Chisq}(g(r-1)),$ which can be used to obtain a confidence interval for the error variance $\sigma^2.$

$\endgroup$
1
$\begingroup$

To say that “degrees of freedom” means $\text{“}n-1\text{''}$ is to miss things.

For example, in a simple linear regression model that says $y_i = a+bx_i + \text{error}_i,$ where $a,b$ are estimated by ordinary least squares, the number of degrees of freedom for error, which appears in the denominator in an F-test, or whose square root appears in the denominator in a t-test, is $n-2,$ where $n$ is the number of data points, i.e. $i=1,\ldots,n.$

For the rest, this question has often been answered. I suggest you start with Wikipedia's article titled "Bessel's correction" and then post any specific questions about it here.

$\endgroup$
  • $\begingroup$ I suggest you start with Wikipedia's article titled "Bessel's correction" and then post any specific questions about it here Thanks. I will have to read related concepts mentioned in the page too, to get better understanding. It will take some time. I will get back to you after that. $\endgroup$ – q126y Jun 14 '18 at 12:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.