Simple Q - How to Interpret Significance Levels in a Chi-Square Test I've just started learning an introductory statistics course, and I wanted to clarify my understanding of significance levels in the Chi-Squares test (both for determining independence and goodness of fit). I've seen people use 5% or 1% as standard SL's (though I'm assuming they're arbitrarily chosen), but I'm trying to understand the meaning of significance levels and how you interpret them. Do I have the correct interpretation?
Example, with a Sig Lvl of 5%: 'If your chi-square statistic (given your degrees of freedom) is greater than the critical value for a 5% Sig Lvl, then there is less than a 5% chance of achieving your observed values given your null hypothesis is true. Thus, with your observed values being as they are, it's likely the null hypothesis is not true, so we reject it.'
If that is correct, is that the same as saying 'There is a greater than 95% chance that your null hypothesis is false, given what your observed values are'? 
I may have misunderstood this completely, but I really appreciate any insight. 
Many thanks
 A: Chi-squared g00dness-of-fit (GOF) tests are widely used and often
misinterpreted. Here are two examples that involve testing to
judge whether a die is fair.
Example 1: Suppose we roll a die 60 times, and get the following
summary table of results.
face:   1  2  3  4  5  6 
freq:  12  8 11 15  6  8 

If the die is fair, then we say we would 'expect' each face to
occur 10 times. Of course, that would be an 'average' result.
In view of random variation, it would be a very rare outcome 
to see a frequency of exactly 10 for each of the six faces.
The question is how much different from the 'expected' results
$E_i = 10$ can the actual results $X_i$ be before we reject the
null hypothesis that each face has probability $p_i = 1/6?$
The usual way to measure departure from the idealized outcome
is to compute the GOF statistic
$$Q = \sum_{i=1}^6 \frac{(X_i - E_i)^2}{E_i}.$$
For the data shown above, we have $Q = 5.4.$ Notice that if
all six observed frequencies were 10's, we would have $Q = 0,$
so large values of $Q$ correspond to poor fit to the null
hypothesis that the die is fair. 
If the null hypothesis is true, $Q \stackrel{aprx}{\sim} \mathsf{Chisq}(\nu = 5),$
the chi-squared distribution with $\nu = 6 - 1 = 5$ degrees of
freedom. This is an approximation, but with all expected values $E_i > 5,$
some theory and some simulation studies show that the approximation is
good enough to use in testing the null hypothesis.
If we are testing the null hypothesis at the 5% level of significance,
the 'critical value' above which we reject the null hypothesis is 
$c = 11.0705.$  Because $Q < c$ we do not reject the null hypothesis.
We say that the data are consistent with behavior of a fair die.
The value $c$ cuts 5% of the area from the upper tail of 
$\mathsf{Chisq}(5).$
In R statistical software, the test procedure looks like this, where
face is the vector of the 60 outcomes tabled above. [Unless a vector
of probabilities other than $p = (1/6, 1/6, 1/6, 1/6, 1/6, 1/6)$ is
specified, the program assumes the 'given probabilities' are equally likely.]
chisq.test(table(face))

        Chi-squared test for given probabilities

data:  table(face)
X-squared = 5.4, df = 5, p-value = 0.369

The P-value is the probability a fair die would give a $Q$-value greater
than our result $Q = 5.4.$ [Another way to test at
the 5% level is to reject the null hypothesis if the P-value is smaller
than 5%.] 
The figure below shows the density curve of $\mathsf{Chisq}(5).$ The
vertical dotted red line is at the critical value $c = 11.0705,$
the vertical solid black line is at the observed value $Q = 5.4,$
and the area beneath the curve to the right of the black line is
the P-value.

Example 2:  By placing a lead weight beneath the corner of a die
where faces 4, 5, and 6 meet it would be possible to make an unfair die
with probabilities $$p = (7/36, 7/36, 7/36, 5/36, 5/36, 5/36).$$ With $n = 60$ rolls of such
an altered die, the
expected counts would be $$E = \left(11\frac23, 11\frac23, 11\frac23,
8\frac13, 8\frac13, 8\frac13 \right).$$
Now we ask whether our data are also consistent with 60 rolls of
such an unfair die. Again the 'null distribution' of $Q$ is
$\mathsf{Chisq}(5)$ and the critical value is $c=11.0705.$
However, we must use the new expected values $E_i$ in the
formula for the GOF statistic, so that $Q = 7.2 < c$ and
the null hypothesis is (once again) not rejected.
chisq.test(table(face), p=c(7,7,7,5,5,5)/36)

        Chi-squared test for given probabilities

data:  table(face)
X-squared = 7.2, df = 5, p-value = 0.2062

So we cannot say in Example 1 that we have "proved" the die is fair.
The data are also consistent with a die that is biased as described
in the current example.  With only $n = 60$ rolls of the die,
we do not have enough information to distinguish between a fair
die and a somewhat biased one.
If the die were truly biased as described and the number of rolls had
been greater
(perhaps 600 instead of 60), then we would very likely get
data that are clearly not consistent with a fair die.
Note: The data for these examples resulted from 60 rolls of a
die that I suppose is fair. (Transparent plastic and no signs of
tampering.)
