Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

If $\hat{Y}$ is the OLS linear regression model for $Y$, what can I say about $\operatorname{Cov}(\hat{Y},Y)$? Is this value $0$?

share|cite|improve this question
In linear minimum-mean-square error estimation where $\hat{Y}=aX+b$ with $a$ and $b$ chosen so as to minimize $E[(Y-\hat{Y})^2]$, the residual error $Y-\hat{Y}$ is orthogonal to the estimate $\hat{Y}$, that is, $\text{cov}(Y-\hat{Y},\hat{Y}) = 0$ and so $$\text{cov}(\hat{Y},Y) = \text{var}(\hat{Y})$$ – Dilip Sarwate Dec 27 '12 at 17:16
I would like to ask further if var$(\hat{Y})$ = var$(Y)$? And, on the space of random variables, why do in some cases we use expectation as the inner product and in others we use the covariance? – Peter Dec 27 '12 at 17:49
No, $\text{var}\hat{Y)} = \rho^2\text{var}(Y)$ (where $\rho$ is the Pearson correlation coefficient) is generally smaller than $\text{var}(Y)$. – Dilip Sarwate Dec 27 '12 at 19:47
up vote 4 down vote accepted

$\newcommand{\var}{\operatorname{var}}$ $\newcommand{\cov}{\operatorname{cov}}$ If you know matrix algebra, one often writes $$ \begin{array}{ccccccccccc} Y & = & X & \beta & + & \varepsilon \\ \\ (n\times1) & & (n\times p) & (p\times1) & & (n\times1) \end{array} $$ where $X$ is observable and "fixed" (i.e. not random), $\beta$ is unobservable and fixed, $\varepsilon$ is unobservable and random, and $Y$ is observable and random. The $n\times n$ matrix $H = X(X^T X)^{-1}X^T$ projects orthogonally onto the column space of $X$, and $$ \hat Y = HY. $$ Recall that if $Y$ is an $n\times 1$ random column vector, then $$ V=\var(Y) = \mathbb E\Big( (Y-\mathbb E Y)(Y - \mathbb E Y)^T \Big) $$ is an $n\times n$ matrix. And $$ \begin{array}{cccccccccccccccc} \cov\Big( & A & Y & , & B & Y & \Big) & = & A & \var(Y) & B^T \\ \\ & (j\times n) & (n\times1) & & (k\times n) & (n\times1) & & & (j\times n) & (n\times n) & (n\times k) \end{array} $$ is a $j\times k$ matrix.

So $$ \cov(\hat Y, Y) = \cov(HY, Y) = H \cov(Y,Y) = H\sigma^2 I_{n\times n} = \sigma^2 H. $$

You could also write $$ \cov(\hat Y, Y) = \cov(\hat Y, \hat Y) + \cov(\hat Y, \hat\varepsilon) $$ $$ = \cov(HY, HY) + 0 = H\cov(Y,Y) H^T = H(\sigma^2 I_{n\times n})H^T = \sigma^2 HH^T. $$

But, being the matrix of an orthogonal projection, $H$ is both its own transpose and its own square, so this reduces to the same thing we got by the other method.

share|cite|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.