# Closed form for coefficients in Multiple Regression model

I want to find $\hat{\beta}$ in ordinary least squares s.t. $\hat{Y} = \hat{\beta}_0 + \hat{\beta}_1 X_1 + \cdots + \hat{\beta}_n X_n$. I know the way to do this is through the normal equation using matrix algebra, but I have never seen a nice closed form solution for each $\hat{\beta}_i$. I'm thinking as a generalization of the simple linear regression case,

$$\hat{\beta}_i = \frac{ Cov(X_i, Y) }{Var(X_i) },$$

where $Y = \beta_0 + \beta_1 X_1 + \cdots + \beta_n X_n + \epsilon_i$.

Is my conjecture for the form of the regression coefficients true? And what would $\hat{\beta_0}$ be?

-

Your way of using the letter $n$, rather than using that for the sample size, is irritating. I'll write consistently with that and use $m$ for the sample size.

You have a design matrix $$X=\begin{bmatrix} 1 & x_{11} & \cdots & x_{1n} \\ 1 & x_{21} & \cdots & x_{2n} \\ \vdots & \vdots & & \vdots \\ 1 & x_{m1} & \cdots & x_{mn} \end{bmatrix}$$ Then $$\begin{bmatrix} Y_1 \\ \vdots \\ Y_m \end{bmatrix} = \begin{bmatrix} 1 & x_{11} & \cdots & x_{1n} \\ 1 & x_{21} & \cdots & x_{2n} \\ \vdots & \vdots & & \vdots \\ 1 & x_{m1} & \cdots & x_{mn} \end{bmatrix} \begin{bmatrix} \beta_0 \\ \beta_1 \\ \vdots \\ \beta_n \end{bmatrix} + \begin{bmatrix} \varepsilon_1 \\ \vdots \\ \varepsilon_m \end{bmatrix}.$$ Write this as $$Y=X\beta+\varepsilon.$$ Then the least-squares estimate are $$\hat\beta = (X^T X)^{-1} X^T Y.$$ The potentially messy part --- the only nonlinear part --- is the matrix inversion.

If you want just $\hat\beta_k$, put a row vector in front of the above, with the $k$th entry equal to $1$ and the others $0$.

-
Note that correlations among the columns of $X$ affect the outcome. If all the columns of $X$ were orthogonal to each other, then you could treat it as $n$ separate simple regressions. –  Michael Hardy Jan 1 '13 at 0:54
I'm familiar with the normal equation and matrix solution. I was looking for a clean expression for $\beta_i$ that wouldn't require computing a matrix inverse. Is my conjecture for $\beta_i$ incorrect? –  Peter Jan 1 '13 at 1:08
Your conjecture is unclear: You're writing $y_i$ when you apparently mean the $i$th case, but then you're writing $\beta_i$ and $x_i$ when you mean something different from that. However, you would need more than just one of the predictors except in the case where all of them are uncorrelated. –  Michael Hardy Jan 1 '13 at 13:08
I've edited my post to make it somewhat clearer. If we think of the predictor variables as random variables $X_i$, I'm asking if the regression coefficients take the given forms. If we had a data matrix of observed values for the explanatory and response variables, then covariance and variance would be replaced by their estimators. –  Peter Jan 1 '13 at 23:42

Your question is about the regression components. Let us partition $$X=[X_1\quad X_2]$$ and $$\beta=\begin{pmatrix} \beta_1\\\beta_2 \end{pmatrix}.$$ Then the regression model can be written as $$y=X_1\beta_1+X_2\beta_2+e.$$

The OLS estimator of $\beta$ is obtained by $$y=X\hat\beta=X_1\hat\beta_1+X_2\hat\beta_2+\hat e.$$ Let $$M_1=I-X_1(X_1'X_1)^{-1}X_1'\\ M_2=I-X_2(X_2'X_2)^{-1}X_2'$$ After tedious manipulation of linear algebra, the subcoefficients have the formula $$\hat\beta_1=(X_1'M_2X_1)^{-1}(X_1'M_2y)\\ \hat\beta_2=(X_2'M_1X_2)^{-1}(X_2'M_1y).$$ This is the general formula. To answer you question, assume that the sub design matrix $X_2=x_2$ is a column vector, i.e. the corresponding variable in the true model is a scalar. Note that $M_1,M_2$ are symmetric and idempotent. We can write $$\hat\beta_2=((M_1x_2)'M_1x_2)^{-1}((M_1x_2)'M_1y)=(x_2^{*\prime}x_2^*)^{-1}x_2^{*\prime}y^*,$$ where $x_2^*$ and $y^*$ are the regression residuals of $x_2$ and $y$ on $X_1$. This formula says that the individual coefficient is also determined by other variables.

The empirical analog of your conclusion says that the OLS estimator $\hat\beta_2$ can be obtained by running a regression of $y$ on $x_2$ alone. This is in general false. However, if $x_2$ is orthogonal to $X_1$, your proposition is true, as claimed by the orthogonal partitioned regression theorem. In the design matrix the variable corresponding to the intercept term is taken as a column vector $(1,...,1)'$ and included in the formula. There is no need to consider $\hat\beta_0$ separately.

-