1
$\begingroup$

So I'm super confused. Looking online for example:here or here or here or in any other website I could find ANOVA is " used to determine whether there are any statistically significant differences between the means of three or more independent (unrelated) groups" or "are useful for comparing (testing) three or more means (groups or variables) for statistical significance" or "ANOVA is used to compare differences of means among more than 2 groups.".

However I'm studying linear regression models and both in simple and in multiple linear regression, our lecturer uses ANOVA in a totally different way!

We use ANOVA to compare the linear regression model (where response depends linearly on explanatory variable) with the simpler "constant" model where there is no relationship between response and explanatory variable. (and similarly for multiple linear regression we compare whether we need $p>q$ explanatory variables or only $q$ are enough)

Hence is my lecturer teaching us something wrong? Are they two different types of ANOVA but with the same name? Or are they the same thing and I don't see the connection?

$\endgroup$
1
$\begingroup$

They're both ANOVA; they're both about partitioning the total corrected sum of squares into an explained part and an unexplained part.

Suppose \begin{align} U_{ij} = {} & (\text{average for group }i) + (\text{error for observation $j$ in group $i$}), \tag 1 \\ & (\text{The numbers of observations in different groups may differ.}) \\[10pt] \text{and } V_i = {} & \alpha + \beta x_i + (\text{error for observation $i$}). \tag 2 \end{align}

In the first case, let $\overline U_i$ be the sample average in group $i$. Then $U_{ij} - \overline U_i$ is the residual $\hat\varepsilon_{ij}$ and $\overline U_i - \overline U$ is the difference between the $i$th group mean and the grand mean. Then we have \begin{align} \sum_{ij} (U_{ij} - \overline U)^2 & = \text{total sum of squares} \\[10pt] \sum_{ij} (\overline U_i - \overline U)^2 & = \text{explained part of the total sum of squares} \\[6pt] \sum_{ij} (U_{ij} - \overline U_i) & = \text{unexplained part of the total sum of squares} \end{align} (Note that both indices $i,j$ are included in the first sum even though the index $j$ does not appear in the terms being added.)

In the second case, let $\widehat\alpha,\widehat\beta$ be the least-squares estimates of $\alpha,\beta$ and let $\overline V$ be the mean of all of $V_i.$ Then $V_i - (\widehat\alpha + \widehat\beta x_i)$ is the residual $\varepsilon_i.$ We have \begin{align} \sum_i (V_i - \overline V)^2 & = \text{total sum of squares} \\[10pt] \sum_i (\widehat\alpha+\widehat\beta x_i - \overline V)^2 & = \text{explained part of the total sum of squares} \\[10pt] \sum_i (V_i - (\widehat\alpha+ \widehat\beta x_i))^2 & = \text{unexplained part of the total sum of squares} \end{align}

In the group-means case, the null hypothesis is that there are no difference in group means; in the fitting-a-line case the null hypothesis is that the slope $\beta$ is zero, so no differences in $V$-values result from changes in the $x$-value.

In both cases, there is a test statistic: $$ F = \frac{(\text{explained part})/\text{df}}{(\text{unexplained part})/\text{df}}, $$ where $\text{“df''}$ is not the same in every case. In the group means case, the number of degrees of freedom in the numerator is $1$ less than the number of groups. In the denominator is is $1$ less than the number of members of a group, then summed over all groups. In the fitting-a-line case, the number of degrees of freedom in the numerator is $2$ and in the denominator is $2$ less than the number of observations.

If we assume i.i.d. mean-$0$ normally distributed errors, then $F$ has a standard central $F$-distribution in both cases, if the null hypothesis is true, and has a stochastically larger non-central $F$-distribution if the null hypothesis is false.

$\endgroup$
  • $\begingroup$ SO basically a simple linear regression anova with $H_0: \beta_1=0$ which compares the linear to the simpler "constant" model, is equivalent to a t-test for equality of two means? $\endgroup$ – Euler_Salter Apr 5 '17 at 13:14
  • $\begingroup$ And similar, is it true in multiple linear regression ($Y_i = \beta_0+\beta_1x_i+..+\beta_qx_q+..+\beta_px_p$) that a null hypothesis $H_0: \beta_q=..=\beta_p=0$ which compares the model with $p >q$ mean parameters to the model with only $q$ mean parameters, is equivalent to a test comparing the means of groups $q$, $q+1$,..., $p$? $\endgroup$ – Euler_Salter Apr 5 '17 at 13:16
  • 1
    $\begingroup$ @Euler_Salter : It's equivalent in the sense that the test statistic has the same distribution, and $F$-distribution, if the null hypothesis is true and a noncentral $F$-distribution if the null hypothesis is false. $\endgroup$ – Michael Hardy Apr 5 '17 at 17:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.