3
$\begingroup$

Chi squared, $\chi ^{2}$, is calculated using the formula:

$$ \chi ^{2} = \sum \frac{{(O_i - E_i)}^{2}}{E_i}$$

$\chi ^{2}$ is used to determine how well a particular model fits some observed data. The way I justify this formula is that we want a model that resembles the data very closely. Hence, we will need to check how different the model is to the observed data.

  1. We can check how different individual data points are from the expected values by $(O_i - E_i)$. Thus, $(O_i - E_i)$ is justified.

  2. To determine how well the model fits the data as a whole we can sum $(O_i - E_i)$ for all data points. $(O_i - E_i)$ will need to be squared to remove negative terms in the summation. Negative terms will lower $\chi ^{2}$ and give a flawed goodness of fit. Thus, $\sum {(O_i - E_i)}^{2}$ is justified.

My question is why do we normalize ${(O_i - E_i)}^{2}$ by $E_i$? This seems unnecessary to me.

$\endgroup$
  • 1
    $\begingroup$ This has the effect of scaling the squared differences so that the value of $\chi^2$ does not depend on the square of the order of magnitude of the data $\endgroup$ – David Quinn Dec 27 '16 at 20:44
  • 1
    $\begingroup$ But if you want $\chi ^{2}$ to not depend on the order of magnitude of the data should you not normalize by $E_i ^{2}$ as you are trying to normalize the squared difference? $\endgroup$ – PiccolMan Dec 27 '16 at 21:08
4
$\begingroup$

The general idea is that we want the test statistic $\chi^{2}$ to be independent of the scale of the data (or more precisely, the spread) as the actual probability distribution is independent of the scale. That is, given a distribution with variance $\sigma^{2}$, we can always divide out by $\sigma^{2}$ to get a unit-variance distribution, and it is both valid and simpler to always just work with a unit-variance distribution (we just have one fewer parameter to think about).

More specifically to $\chi^{2}$, the reason that we divide by $E_{i}$ has to do with the underlying distribution. First, it is important to realize that what $\chi^{2}$ random variables are. Generally, a $\chi^{2}$ variable with $n$ degrees of freedom is the sum of squares of $n$ unit-variance normal random variables. So when we define $$ \chi^{2} = \sum \frac{(O_{i} - E_{i})^{2}}{E_{i}} $$ we are using the fact that each $\frac{(O_{i} - E_{i})}{\sqrt{E_{i}}}$ is normally distributed with unit-variance. More precisely, we can treat $O_{i}$ as being normally distributed with mean $E_{i}$ and variance $E_{i}$. The value $\frac{(O_{i} - E_{i})}{\sqrt{E_{i}}}$ is the standardized residual.

There is a twist that explains why $E_{i}$ is the variance: the value $O_{i}$ is actually poisson distributed. This makes sense since we are working with counts and not something continuous. In the poisson distribution, the variance is precisely the same as the mean $E_{i}$. The question then is why we can treat $O_{i}$ as normally distributed. The reason is that when the expected counts $E_{i}$ is `large enough', the poisson distribution is reasonably similar to the normal distribution (very, very roughly).

On a final note, the variance of a $\chi^{2}$ variable is actually not one but is dependent on the degrees of freedom. Regardless, the distribution only has the one parameter (the degrees of freedom) and no variance parameter, so it's the same idea as using a unit-variance distribution.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Cant we divide this by $O_i$ instead of $E_i?$ How is it exactly different? $\endgroup$ – Siddarth May 31 '19 at 7:02
  • $\begingroup$ @Siddarth Since we are modeling $O_{i}$ as poisson distributed with mean $E_{i}$ (and thus variance $E_{i}$), the normalized residual is by definition $(O_{i} - E_{i})/\sqrt{E_{i}}$. There is no corresponding reason to divide by $O_{i}$. $\endgroup$ – Jacob Maibach Jun 2 '19 at 18:31
0
$\begingroup$

A small analogy:

If you have a quadrillion dollars, would you mind misplacing a $\$100$ dollar bill in the trash?

Other case:

If you are completely broke and starving, would you mind misplacing a $\$100$ dollar bill in the trash?

In one case, the $\$100$ matters a lot, but in the other, $\$100$ may as well be a penny on the street.

In the case of statistics, each value evaluated in the sum of the $\chi^2$ must be normalized, or else the significance of the test will change drastically when using large observed and expected values.

For instance (one data point used as example):

$\chi^2=\sum \frac{(3-2)^2}{2}$ should be just as statistically significant as $\chi^2=\sum \frac{(300-288)^2}{288}$, but without the "normalizer", the $\chi^2$'s would be drastically different.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.