PROBLEM STATEMENT: Consider the following 2-variable linear regression where the error $e_i$ 's are independently and identically distributed with mean $0$ and variance $1$;

$$y_i = α + β(x_i − \bar x) + e_i ,\ i = 1,2,...,n.$$

Let $\hat α$ and $ \hat β$ be ordinary least squares estimates of $α$ and $β$ respectively. What is the correlation coefficient between $\hat α$ and $\hat β$?

MY ATTEMPT: I use the standard optimization technique to minimize the sum of squares of the error terms. By differentiating by $\alpha$ and $\beta$, I find $$\hat \alpha = \bar y,\ \hat \beta = \frac{\sum x_iy_i-n\bar x\bar y}{\sum x_i^2 - n\bar x^2}.$$ I am stuck here. How do I use the fact that $e_i$'s are i.i.d in order to find the correlation coefficient between $\hat \alpha$ and $\hat \beta$? Firstly, I do not think I understand the problem correctly. In order to calculate the correlation coefficient, I must have a set of values of $\hat \alpha$s and $\hat \beta$s. The $e_i$'s are i.i.d random variables each having mean $0$ and variance $1$. Based on the different values that the different $e_i$s take, I solve the minimization problem every time and find that $\hat \alpha$ and $\hat \beta$ are only dependent on $x_i,y_i$ as above and hence always the same. How then do I find the correlation coefficient?

I have knowledge of only the definitions of elementary terms in the topic of regression, and I am self-studying this. I am sure the problem must have a very easy solution as it is meant to be solved in a few minutes with an extremely elementary knowledge of statistics.

  • $\begingroup$ The covariance between regression coefficients is related to either their finite-sample or asymptotic distribution. Under a number of assumptions each of them is the bivariate normal with some covariance matrix. Given that covariance matrix you can calculate the corresponding correlation... $\endgroup$ – d.k.o. Jun 19 '15 at 6:07

(Typically in linear regression we condition on $x_i$ and only regard $Y_i$ as random, so in what follows we treat each $x_i$ as constant.)

For this problem it's important to remember that $\text{Cov}(Y_i, Y_j) = 0$ whenever $i \neq j$, otherwise $\text{Cov}(Y_i, Y_i) = \text{Var}(Y_i) = 1$. Also recall that covariance is a linear operation, so for random variables $X, Y$ and $Z$ and constants $a$ and $b$ we can write $\text{Cov}(X, aY + bZ) = a \text{Cov}(X, Y) + b \text{Cov}(X, Z)$. Here is the calculation:

$$ \begin{align} \text{Cov} ( \hat{\alpha}, \hat{\beta}) &= \text{Cov} \left (\bar{Y}, \frac{ \sum_{i=1}^{n} x_i Y_i - n \bar{x} \bar{Y}}{ \sum_{i=1}^{n} x_i^2 - n \bar{x}^2} \right ) \\ &= \frac{1}{n (\sum_{i=1}^{n} x_i^2 - n \bar{x}^2)} \sum_{j=1}^{n} \text{Cov} \left (Y_j, \sum_{i=1}^{n} x_i Y_i - n \bar{x} \bar{Y} \right ) \\ &= \frac{1}{n (\sum_{i=1}^{n} x_i^2 - n \bar{x}^2)} \sum_{j=1}^{n} \left [ \sum_{i=1}^{n} x_i \text{Cov} \left ( Y_j, Y_i \right ) - n \bar{x} \text{Cov} \left ( Y_j, \bar{Y} \right ) \right ] \\ &= \frac{1}{n(\sum_{i=1}^{n} x_i^2 - n \bar{x}^2)} \sum_{j=1}^{n} \left [ x_j - \bar{x} \sum_{i=1}^{n} \text{Cov}(Y_j, Y_i) \right ] \\ &= \frac{1}{n(\sum_{i=1}^{n} x_i^2 - n \bar{x}^2)} \sum_{j=1}^{n} \left ( x_j - \bar{x} \right ) \\ &= 0 . \end{align} $$

which means the regression coefficients are uncorrelated. (This happens whenever the predictors have been centered by subtracting off their mean.)


Let's see if some cleaner notation can capture the gist of it. Define:

$$ J=\begin{bmatrix} y_1\\ y_2\\ \vdots \\ y_n\\ \end{bmatrix},\space J=\begin{bmatrix} 1\\ 1\\ \vdots \\ 1\\ \end{bmatrix},\space X=\begin{bmatrix} x_1-\bar{x}\\ x_2-\bar{x}\\ \vdots \\ x_n-\bar{x}\\ \end{bmatrix} $$

Please note the key here is $J\perp X$ - can't emphasize this enough.

Therefore, we have $\hat{\alpha}=(J^\intercal J)^{-1}J^{\intercal}Y$, $\hat{\beta}=(X^\intercal X)^{-1}X^{\intercal}Y$. So,

$$ \begin{align*} Cov(\hat{\alpha},\hat{\beta}) &=(J^\intercal J)^{-1}J^{\intercal}\space Cov(Y, Y)\space X(X^\intercal X)^{-1}\\ &=(J^\intercal J)^{-1}J^{\intercal}\space\sigma^2I\space X(X^\intercal X)^{-1}\\ &=\sigma^2(J^\intercal J)^{-1}(J^\intercal X)(X^\intercal X)^{-1}\\ &=0 \space\space\space (Again, J\perp X) \end{align*} $$

From this, we can draw a more general conclusion:

In linear regression $Y=X_1\beta_1+X_2\beta_2+\epsilon$, we have $\hat{\beta}_1\perp\hat{\beta}_2$ if $X_1\perp X_2$.

Even more general, let $\Sigma=Cov(\epsilon)$ be a general positive definite matrix. In a inner product space defined by $<u,v>=u^\intercal\Sigma v$, the above conclusion still stands, i.e. we have $\hat{\beta}_1^{\intercal}\Sigma \hat{\beta}_2=0$ if $X_1^{\intercal}\Sigma X_2=0$.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.