Closed form for coefficients in Multiple Regression model

Question

I want to find $\hat{\beta}$ in ordinary least squares s.t. $\hat{Y} = \hat{\beta}_0 + \hat{\beta}_1 X_1 + \cdots + \hat{\beta}_n X_n $. I know the way to do this is through the normal equation using matrix algebra, but I have never seen a nice closed form solution for each $\hat{\beta}_i$. I'm thinking as a generalization of the simple linear regression case,

$$ \hat{\beta}_i = \frac{ Cov(X_i, Y) }{Var(X_i) },$$

where $ Y = \beta_0 + \beta_1 X_1 + \cdots + \beta_n X_n + \epsilon_i $.

Is my conjecture for the form of the regression coefficients true? And what would $\hat{\beta_0}$ be?

Answer 1

Your way of using the letter $n$, rather than using that for the sample size, is irritating. I'll write consistently with that and use $m$ for the sample size.

You have a design matrix $$ X=\begin{bmatrix} 1 & x_{11} & \cdots & x_{1n} \\ 1 & x_{21} & \cdots & x_{2n} \\ \vdots & \vdots & & \vdots \\ 1 & x_{m1} & \cdots & x_{mn} \end{bmatrix} $$ Then $$ \begin{bmatrix} Y_1 \\ \vdots \\ Y_m \end{bmatrix} = \begin{bmatrix} 1 & x_{11} & \cdots & x_{1n} \\ 1 & x_{21} & \cdots & x_{2n} \\ \vdots & \vdots & & \vdots \\ 1 & x_{m1} & \cdots & x_{mn} \end{bmatrix} \begin{bmatrix} \beta_0 \\ \beta_1 \\ \vdots \\ \beta_n \end{bmatrix} + \begin{bmatrix} \varepsilon_1 \\ \vdots \\ \varepsilon_m \end{bmatrix}. $$ Write this as $$ Y=X\beta+\varepsilon. $$ Then the least-squares estimate are $$ \hat\beta = (X^T X)^{-1} X^T Y. $$ The potentially messy part --- the only nonlinear part --- is the matrix inversion.

If you want just $\hat\beta_k$, put a row vector in front of the above, with the $k$th entry equal to $1$ and the others $0$.

Answer 2

Your question is about the regression components. Let us partition $$ X=[X_1\quad X_2] $$ and $$ \beta=\begin{pmatrix} \beta_1\\\beta_2 \end{pmatrix}. $$ Then the regression model can be written as $$ y=X_1\beta_1+X_2\beta_2+e. $$

The OLS estimator of $\beta$ is obtained by $$ y=X\hat\beta=X_1\hat\beta_1+X_2\hat\beta_2+\hat e. $$ Let $$ M_1=I-X_1(X_1'X_1)^{-1}X_1'\\ M_2=I-X_2(X_2'X_2)^{-1}X_2' $$ After tedious manipulation of linear algebra, the subcoefficients have the formula $$ \hat\beta_1=(X_1'M_2X_1)^{-1}(X_1'M_2y)\\ \hat\beta_2=(X_2'M_1X_2)^{-1}(X_2'M_1y). $$ This is the general formula. To answer you question, assume that the sub design matrix $X_2=x_2$ is a column vector, i.e. the corresponding variable in the true model is a scalar. Note that $M_1,M_2$ are symmetric and idempotent. We can write $$ \hat\beta_2=((M_1x_2)'M_1x_2)^{-1}((M_1x_2)'M_1y)=(x_2^{*\prime}x_2^*)^{-1}x_2^{*\prime}y^*, $$ where $x_2^*$ and $y^*$ are the regression residuals of $x_2$ and $y$ on $X_1$. This formula says that the individual coefficient is also determined by other variables.

The empirical analog of your conclusion says that the OLS estimator $\hat\beta_2$ can be obtained by running a regression of $y$ on $x_2$ alone. This is in general false. However, if $x_2$ is orthogonal to $X_1$, your proposition is true, as claimed by the orthogonal partitioned regression theorem. In the design matrix the variable corresponding to the intercept term is taken as a column vector $(1,...,1)'$ and included in the formula. There is no need to consider $\hat\beta_0$ separately.