5
$\begingroup$

I know that in general this is true because the smaller model is nested within the larger model, so the larger model must have SSE at least as low as the smaller one, but I'm having a hard time proving this mathematically.

I know that $$R^{2} = \frac{\mathbf{{\hat{\boldsymbol\beta '} X ' y}} - n \bar{\mathbf{y}}^{2}}{\mathbf{y'y} - n\bar{\mathbf{y}}^{2}}$$

So my idea was to show that $\mathbf{{\hat{\boldsymbol\beta '} X ' y}}$ is increasing with $p$ (assuming $\mathbf{X}$ is $n$ x $p$) since the rest of the equation is constant wrt $\mathbf{X}$. I've gotten as far as $$\sum\limits_{i=1}^{n}\sum\limits_{j=1}^{p} \hat{\boldsymbol\beta}_j \ x_{ij} \ y_{i}$$That is where I'm stuck.

$\endgroup$
6
$\begingroup$

I think it's best to use matrix notation to answer this. Suppose the original estimate by OLS is $$y=X\beta+u \tag{1}$$ where $X$ is an $n\times p$ matrix of observables, $\beta$ is a $p\times 1$ vector of coefficients obtained by OLS and $u$ is a $p\times 1$ vector of residuals. Now we add a new variable $X_0$ and perform a new OLS estimate $$y=X_0\hat{\beta}_0+X\hat{\beta}+v \tag{2}.$$ Since $\beta$, $\hat{\beta}_0$ and $\hat{\beta}$ are all OLS estimates we know that $u^TX=v^TX_0=v^TX=0$.

Combine $(1)$ and $(2)$ to get $$X\beta+u=X_0\hat{\beta}_0+X\hat{\beta}+v \tag{3}.$$ If we multiple both sides of $(3)$ by $u^T$ we have $$u^Tu=u^TX_0\hat{\beta}_0+u^Tv.$$ Similarly, multiplying by $v^T$ gives $$v^Tu=v^Tv.$$ This tells us that $$u^TX_0\hat{\beta}_0=u^Tu-v^Tv \tag{4}.$$

Finally,

$ \begin{align*} v^Tv&=(y-X_0\hat{\beta}_0-X\hat{\beta})^T(y-X_0\hat{\beta}_0-X\hat{\beta}) \\ &=(X\beta+u-X_0\hat{\beta}_0-X\hat{\beta})^T(X\beta+u-X_0\hat{\beta}_0-X\hat{\beta})\qquad\text{from $(1)$} \\ &=(X(\beta-\hat{\beta})-X_0\hat{\beta}_0+u)^T(X(\beta-\hat{\beta})-X_0\hat{\beta}_0+u) \\ &=(X(\beta-\hat{\beta})-X_0\hat{\beta}_0)^T(X(\beta-\hat{\beta})-X_0\hat{\beta}_0)+2u^T(X(\beta-\hat{\beta})-X_0\hat{\beta}_0)+u^Tu \\ &=(X(\beta-\hat{\beta})-X_0\hat{\beta}_0)^T(X(\beta-\hat{\beta})-X_0\hat{\beta}_0)-2u^TX_0\hat{\beta}_0+u^Tu \\ &=(X(\beta-\hat{\beta})-X_0\hat{\beta}_0)^T(X(\beta-\hat{\beta})-X_0\hat{\beta}_0)-2(u^Tu-v^Tv)+u^Tu \qquad\text{using $(4)$}\\ &=(X(\beta-\hat{\beta})-X_0\hat{\beta}_0)^T(X(\beta-\hat{\beta})-X_0\hat{\beta}_0)-u^Tu+2v^Tv \\ u^Tu&=(X(\beta-\hat{\beta})-X_0\hat{\beta}_0)^T(X(\beta-\hat{\beta})-X_0\hat{\beta}_0)+v^Tv \\ &\geq v^Tv. \end{align*} $

This says that the new SSE is less than the original SSE. It follows that $R^2$ must increase (weakly).

$\endgroup$
2
  • $\begingroup$ You mean $R^{2}$ must increase, correct? $\endgroup$ Oct 20 '16 at 15:25
  • $\begingroup$ You're right. Fixed. $\endgroup$
    – user375366
    Oct 21 '16 at 0:10
1
$\begingroup$

$R^2$ is $\frac{SST-SSE}{SST}$. The OLS regression method tries to find the values of the intercept and the coefficients so that SSE is MINIMIZED (and $R^2$ is maximized).

Let the equation for estimating the $y$ with $k$ independent variables be

$$\hat y^k = a + b_1×x_1 + b_2×x_2 + .. +b_k×x_k\tag {1}$$ where $$y = \hat y^k + u \tag{2}$$

Here $u$ denotes the residual error.

The SSE for the above relationship for $n$ observations is

$$SSE_k = \sum_{i=0}^n(y_i - \hat y_i^k)^2$$

OLS will minimize the above expression by choosing the values of the intercept $~a~$ and $~b_1,~ b_2, ~b_3, \cdots, b_k$.

Let us now add another explanatory variable $~z~$ to the above equation. Let the coefficient of that one be $~c~$. The estimate equation (with the extra variable $~z~$) will be

$$\hat y^{k+z} = a + b_1×x_1 + b_2×x_2 + \cdots +b_k×x_k + c×z \tag{3}$$ Where, $$ y = \hat y^{k+z} + v \tag{4} $$ and $v$ denotes the residual error.

Please note that I am using $z$ in the superscripts and subscripts as an indicator for the extra variable in the model instead of $k+1$ for ease of explanation. Please read $~k+z~$ as $~k$ variables and $z$. It is also to be noted that the OLS estimates for the intercept and coefficients for equation 1 and equation 3 can be different.

The SSE for equation 4 is

$$SSE_{k+z} = \sum_{i=0}^n(y_i - \hat y^{k+z}_i)^2 = \sum_{i=0}^n \{(y_i - \hat y^k_i)-c×z_i\}^2$$ $$=SSE_k + \sum_{i=0}^n\{(c\times z_i)^2 -2(y_i-\hat y_i^k)(c\times z_i)\}$$

The OLS algorithm will now try to choose new values for $~a,~ b_1,~ b_2,\cdots, b_k$ and a value for $~c~$ so that the above term is minimized. Clearly, it has to be either same as $MIN(SSE_k)$ or lower (think about it; the algorithm always has the option of setting $~c~$ to zero and using the OLS estimates of $~a~$ and $~b_1,~ b_2, ~b_3, \cdots, b_k$ for the model with $k$ variables to get the $MIN(SSE_k)$ as the minimum value for $SSE_{k+z}$, if it cannot find other estimates of the intercept and the coefficients so that the new $SSE_{k+z}$ is lower than the $MIN(SSE_k)$).

This shows that as more variables are added to the model without changing the sample size, $R^2$ can only decrease or remain the same.

Graphically, think about the situation where you start with one variable. The equation is

$$y = a + bx.$$

Let us also assume that you have only three observations. The OLS will try to fit the best line through these $3$ points to maximize $R^2$. The value of that $R^2$ has to be between $~0~$ and $~1~$. If a new variable is added, then OLS will find a plane that goes through all the three points making a perfect fit and $R^2$ for this model will be $~1~$ which will be either same as the one variable model (if that was a perfect fit) or better.

$\endgroup$
2
  • $\begingroup$ Welcome to MSE. Please edit and use MathJax to properly format math expressions. $\endgroup$ Jul 8 '19 at 5:10
  • $\begingroup$ Thank you. Fixed $\endgroup$
    – user687405
    Jul 8 '19 at 19:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.