I want to find $\hat{\beta}$ in ordinary least squares s.t. $\hat{Y} = \hat{\beta}_0 + \hat{\beta}_1 X_1 + \cdots + \hat{\beta}_n X_n $. I know the way to do this is through the normal equation using matrix algebra, but I have never seen a nice closed form solution for each $\hat{\beta}_i$. I'm thinking as a generalization of the simple linear regression case,

$$ \hat{\beta}_i = \frac{ Cov(X_i, Y) }{Var(X_i) },$$

where $ Y = \beta_0 + \beta_1 X_1 + \cdots + \beta_n X_n + \epsilon_i $.

Is my conjecture for the form of the regression coefficients true? And what would $\hat{\beta_0}$ be?


Your way of using the letter $n$, rather than using that for the sample size, is irritating. I'll write consistently with that and use $m$ for the sample size.

You have a design matrix $$ X=\begin{bmatrix} 1 & x_{11} & \cdots & x_{1n} \\ 1 & x_{21} & \cdots & x_{2n} \\ \vdots & \vdots & & \vdots \\ 1 & x_{m1} & \cdots & x_{mn} \end{bmatrix} $$ Then $$ \begin{bmatrix} Y_1 \\ \vdots \\ Y_m \end{bmatrix} = \begin{bmatrix} 1 & x_{11} & \cdots & x_{1n} \\ 1 & x_{21} & \cdots & x_{2n} \\ \vdots & \vdots & & \vdots \\ 1 & x_{m1} & \cdots & x_{mn} \end{bmatrix} \begin{bmatrix} \beta_0 \\ \beta_1 \\ \vdots \\ \beta_n \end{bmatrix} + \begin{bmatrix} \varepsilon_1 \\ \vdots \\ \varepsilon_m \end{bmatrix}. $$ Write this as $$ Y=X\beta+\varepsilon. $$ Then the least-squares estimate are $$ \hat\beta = (X^T X)^{-1} X^T Y. $$ The potentially messy part --- the only nonlinear part --- is the matrix inversion.

If you want just $\hat\beta_k$, put a row vector in front of the above, with the $k$th entry equal to $1$ and the others $0$.

| cite | improve this answer | |
  • $\begingroup$ Note that correlations among the columns of $X$ affect the outcome. If all the columns of $X$ were orthogonal to each other, then you could treat it as $n$ separate simple regressions. $\endgroup$ – Michael Hardy Jan 1 '13 at 0:54
  • 1
    $\begingroup$ I'm familiar with the normal equation and matrix solution. I was looking for a clean expression for $\beta_i$ that wouldn't require computing a matrix inverse. Is my conjecture for $\beta_i$ incorrect? $\endgroup$ – Peter Jan 1 '13 at 1:08
  • $\begingroup$ Your conjecture is unclear: You're writing $y_i$ when you apparently mean the $i$th case, but then you're writing $\beta_i$ and $x_i$ when you mean something different from that. However, you would need more than just one of the predictors except in the case where all of them are uncorrelated. $\endgroup$ – Michael Hardy Jan 1 '13 at 13:08
  • $\begingroup$ I've edited my post to make it somewhat clearer. If we think of the predictor variables as random variables $X_i$, I'm asking if the regression coefficients take the given forms. If we had a data matrix of observed values for the explanatory and response variables, then covariance and variance would be replaced by their estimators. $\endgroup$ – Peter Jan 1 '13 at 23:42

Your question is about the regression components. Let us partition $$ X=[X_1\quad X_2] $$ and $$ \beta=\begin{pmatrix} \beta_1\\\beta_2 \end{pmatrix}. $$ Then the regression model can be written as $$ y=X_1\beta_1+X_2\beta_2+e. $$

The OLS estimator of $\beta$ is obtained by $$ y=X\hat\beta=X_1\hat\beta_1+X_2\hat\beta_2+\hat e. $$ Let $$ M_1=I-X_1(X_1'X_1)^{-1}X_1'\\ M_2=I-X_2(X_2'X_2)^{-1}X_2' $$ After tedious manipulation of linear algebra, the subcoefficients have the formula $$ \hat\beta_1=(X_1'M_2X_1)^{-1}(X_1'M_2y)\\ \hat\beta_2=(X_2'M_1X_2)^{-1}(X_2'M_1y). $$ This is the general formula. To answer you question, assume that the sub design matrix $X_2=x_2$ is a column vector, i.e. the corresponding variable in the true model is a scalar. Note that $M_1,M_2$ are symmetric and idempotent. We can write $$ \hat\beta_2=((M_1x_2)'M_1x_2)^{-1}((M_1x_2)'M_1y)=(x_2^{*\prime}x_2^*)^{-1}x_2^{*\prime}y^*, $$ where $x_2^*$ and $y^*$ are the regression residuals of $x_2$ and $y$ on $X_1$. This formula says that the individual coefficient is also determined by other variables.

The empirical analog of your conclusion says that the OLS estimator $\hat\beta_2$ can be obtained by running a regression of $y$ on $x_2$ alone. This is in general false. However, if $x_2$ is orthogonal to $X_1$, your proposition is true, as claimed by the orthogonal partitioned regression theorem. In the design matrix the variable corresponding to the intercept term is taken as a column vector $(1,...,1)'$ and included in the formula. There is no need to consider $\hat\beta_0$ separately.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.