# Simple linear regression and sum of squared errors

Let $Y_i$ be independent $N(\beta_0 + \beta_1x_i, \sigma^2$ for $1\leq i\leq n$, where $\{x_i\}_i=1^n, \beta_0, \beta_1, \sigma^2 >0$ are constants. Let $\hat{\beta_0}, \hat{\beta_1}$ be the solution to \begin{align} \min_{\beta_0, \beta_1} \sum_{i = 1}^n (Y_i - \beta_0 - \beta_1x_i)^2. \end{align}

Note that $\hat{\beta_0}, \hat{\beta_1}$ are of the form \begin{align} \hat{\beta_0} &= \sum_{i = 1}^n a_i Y_i = \mathbf{a}^T\mathbf{Y} \\ \hat{\beta_1} &= \sum_{i = 1}^n b_i Y_i = \mathbf{b}^T\mathbf{Y} \\ \end{align}

The questions are:

(a) Find $c$ such that $\mathbf{d} = \mathbf{a} + c\mathbf{b}$ satisfies $\mathbf{b}^T\mathbf{d} = 0$, and consider the following orthogonal linear transformation

\begin{align} \left[ \begin{array}{c} Z_1 \\ \vdots \\ Z_n \end{array} \right] = \sigma^{-1} \mathbf{K} \left[ \begin{array}{c} Y_1 - \beta_0 - \beta_1 x_1 \\ \vdots\\ Y_n - \beta_0 - \beta_1 x_n \end{array} \right], \end{align}

where the first and second rows are \begin{align} \mathbf{k}_1 \triangleq \mathbf{d}^T / \lVert \mathbf{d} \rVert_2, \\ \mathbf{k}_2 \triangleq \mathbf{b}^T / \lVert \mathbf{b} \rVert_2, \end{align}

and the rows $\mathbf{k}_3, \cdots, \mathbf{k}_n$ are orthonormal vectors. In other words, $\mathbf{K}^T\mathbf{K} = \mathbf{I}$, i.e., $\mathbf{K}^{-1} = \mathbf{K}^T$, so that $\det(\mathbf{K}) = \pm 1$ and $\sum_{i = 1}^n Z_i^2 = \sum_{i = 1}^n (Y_i - \beta_0 - \beta_1 x_i)^2/\sigma^2$.

(b) Define SSE (sum of squared errors) $\triangleq \sum_{i = 1}^n (Y_i - \hat{\beta_0} - \hat{\beta_1} x_i)^2$. Show that for any orthogonal matrix as in Part (a),

\begin{align} SSE/\sigma^2 = \sum_{i = 1}^n \frac{(Y_i - \beta_0 - \beta_1 x_i)^2}{\sigma^2} - Z_1^2 - Z_2^2 = Z_3^2 + \cdots + Z_n^2. \end{align}

Find the joint distribution of $(\hat{\beta_0}, \hat{\beta_1}, SSE/\sigma^2)$.

Part (a) To find $c$. we let \begin{align} &\mathbf{b}^T(\mathbf{a} + c\mathbf{b}) = \mathbf{b}^T\mathbf{a} + c\lVert \mathbf{b} \rVert_2^2 = 0 \nonumber\\ & \Leftrightarrow c = -\frac{\mathbf{b}^T\mathbf{a}}{\lVert \mathbf{b} \rVert_2^2}. \end{align}
Now we need help for Part (b). Do we need to derive the closed form for $\hat{\beta_0}$ and $\hat{\beta_1}$ in order to proceed?