What is degree of freedom in statistics?

Question

In statistics, degree of freedom is widely used in regression analysis, ANOVA and so on. But, what is degree of freedom ?

Wikipedia said that

1. The number of degrees of freedom is the number of values in the final calculation of a statistic that are free to vary.

2. Mathematically, degrees of freedom is the number of dimension of the domain of a random vector, or essentially the number of 'free' components: how many components need to be known before the vector is fully determined.

However, it is still hard for me to understand the concept intuitively.

Question:

Could anyone provide a intuitive explain to the concept and anything can help me understand?

Thanks!

---

Update:

I'm NOT asking how to calculate degree of freedom.

Let me give an example:

For Chi-squared distribution, different degree of freedom produces different probability density function. Could you explain it intuitively?

Answer 1

Intuitively degrees of freedom denotes how many independent things are there. As we introduce constraints, we take away the degree of freedom.

First I'll try to answer your question about Chi-square.

Chi-square distribution with $n$ degree of freedom is the sum of squares $n$ independent standard normal distributions $N(0,1)$ hence we've got $n$ things that vary independently.

I'll start with mechanical example, as degree of freedom is similar in every field.

Consider an airplane flying. It has three degrees of freedom in the usual universe of space, and can be located only if three coordinates are known. These might be latitude, longitude, and altitude; or might be altitude, horizontal distance from some origin, and an angle; or might be direct distance from some origin, and two direction angles. If we consider a given instant of time as a section through the space-time universe, the airplane moves in a four‑dimensional path and can be located by four coordinates, the three previously named and a time coordinate.Hence it now has $4$ d.f.

Note that we assumed that plane is not rotating.

Now considering statistical degrees of freedom..

Similar meaning.

Degree of freedom of a statistic is number of values in calculation of statistic that are independent to vary. As we add restriction to observations, we reduce the degree of freedom. Imposing a relationship upon the observations is equivalent to estimating a parameter from them. The number of degrees of freedom is equal to the number of independent observations, which is the number of original observations minus the number of parmeters estimated from them.

Consider the calculation of mean $\frac {\sum_{i=1}^n X_n }{n}$, we are interested in estimation of error which are estimated by residues. Sum of residuals is $0$. Knowledge of any $n-1$ residues gives the remaining residue. So, only $n-1$ can vary independently. Hence they have $n-1$ d.f.

However d.f is mainly used in regression analysis, and ANOVA. You may note that all the distributions with so called d.f correspond to particular cases in linear statistics. Hence d.f is at the best artificial as they are not constraints on the random variable, but are actually degree of freedom of some quantities in some application from where these distributions originated.

Also, For people who are interested, < http://courses.ncssm.edu/math/Stat_Inst/Worddocs/DFWalker.doc > seems to be quite good read.

Answer 2

Two people are sitting at a bar, you and your friend. There are two sorts of juice before you, one sweet, one sour. After you have chosen your drink, say the sweet one, your friend has no more choice - so degree of freedom is "1": only one of you can choose.

Generalize it to a group of friends to understand higher degrees of freedom...

Answer 3

When you estimate parameters in statistics, suppose you use a parameter (which you have estimated) to estimate another parameter, then you lose $1$ degree of freedom and you end up with $n-1$ degrees of freedom where $n$ is the number of 'individuals' in the population with which you are working.

One example is when working with normal distributions

$$\sigma=\sqrt{\frac{\sum(x-\mu)}{n}^2}$$

let $s$ be the estimate of the population standard deviation ($\sigma$). We use the mean value $\bar x$ instead of $\mu$.

$$s=\sqrt{\frac{\sum(x-\bar x)}{n-1}^2}$$

Here $\bar x$ is an estimate of $\mu$. We estimate $\sigma$ by using an estimate of $\mu$. Hence we lose $1$ of degree of freedom, resulting in $n-1$ degrees of freedom.

Answer 4

To understand the concept intuitively, ask the question of why it is needed. And start from a simpler question, why is the variance computed as $V = \frac 1 {N-1} \sum (x_i-\bar x)^2$ with a $N-1$ as denominator.

The fact is that you are interested in the variance of the population, not of the one of sample. Now obviously the sample is less dispersed than the population (because it is very likely that your sample missed a few of the extreme values), so the variance computed on the sample is lower than the variance of the population.

You have to correct the bias.

Intuitively, the observed average $\bar x = \frac 1 N \sum x_i$ is not exact but only an approximation of the population mean. The variance of this approximation should be added to the observed variance on the sample in order to to get the best approximate of the population sample. Now this variance can be computed: $\sigma^2(\bar X) = \frac 1 N \sigma^2(X)$ (using the iid if the $X_i$). So the sample variance is $1 -\frac 1 N$ the variance of the population.

Formally (in case you need to read twice the previous reasoning), compute the expected value of $\sum (X_i-\bar X)^2$. You will find $(N-1) \sigma^2$ rather than $N \sigma^2$, hence the population variance is $\frac N {N-1}$ the sample variance (as claimed).

When you follow the computations, you start by replacing $\bar X$ by its definition $\bar X = \frac 1 N \sum X_i$, develop the squares, expand the sum, and then one of the term disappears. Namely $N\bar X=\sum X_i$ appears twice with opposite sign, negative in the double product $2 \bar X X$ and positive in the square $\bar X^2$. So $\sum (X_i-\bar X)^2$ is the sum of $N-1$ terms equal in expectation.

This is because the $X_i$ are not independent but linked by one relation $\sum X_i=N \bar X$. In general, if you know the $X_i$ are linked by $p$ independent linear relations $f_j$, then you can cancel out $p$ terms out of the sum $\sum (X_i-f_j(X_i))^2$. Hence the unbiased estimator, $\sum (X_i-f_j(X_i))^2 \approx \frac N {N-p} \sum (x_i-f_j(x_i))^2$.

In regression, ANOVA, etc, the independents relations are not so independent because it is often supposed that the sum of the independents variables (causes) have the same average than the dependent variable (effect). $\sum a_i \bar X_i = \bar Y$. Hence the degree of freedom $N-1$, $p-1$ and $N-p$ and unbiasing factors $\frac N {N-1}$, $\frac p {p-1}$ and $\frac N {N-p}$ for $SS_{Total}$, $SS_{Model}$ and $SS_{Error}$ respectively.

In two words, the degree of freedom is the number of independent relationships linking a set of variables, taking into account the variable introduced for intermediary estimators.

Answer 5

I've always viewed "degrees of freedom" as a measure of the number of choices that are available to you in a situation. This of course is not meant to be rigorous, but works intuitively in examples.

Consider the problem of choosing a polynomial of degree $2$ (with coefficients in some fixed field $k$). Then we have three choices to make, corresponding to the selection of $a_0,a_1,a_2$ in $$f(x):=a_2x^2+a_1x+a_0.$$ We should then expect the number of degrees of freedom in this example to be three. And indeed, the space of such polynomials is a vector space, of degree $3$. If we suppose now that we only care for polynomials admitting $x=1$ as a root, we have -- on an intuitive level -- used up one of our degrees of freedom, and we should expect to be able to make two more independent choices. This is indeed the case: we are left with a vector space of dimension $2$.

Note that these notions only really make sense for vector spaces and "choices" that correspond to the intersection of hyperplanes. This all can be generalized within the framework of algebraic geometry (and intersection theory), but matters become complicated when hypersurfaces need not intersect as expected.