# Find $\operatorname{Cov}(\hat{\beta_0}, \hat{\beta_1})$.

Let $Y_1,Y_2,\ldots,Y_n$ and $X_1,X_2,\ldots,X_m$ be random variables with $E(Y_i)=\mu_i$ and $E(X_j)=\xi_j$. Define $$U_1=\sum_{i=1}^n a_i Y_i\quad\text{and}\quad U_2=\sum_{j=1}^m b_j X_j$$ for constants $a_1,a_2,\ldots,a_n$ and $b_1,b_2,\ldots,b_m$. Then the following hold:

$\quad\textbf a\,\,$ $E(U_1)=\sum_{i=1}^n a_i\mu_i.$

$\quad\textbf b\,\,$ $V(U_1)=\sum_{i=1}^n a_i^2 V(Y_i)+2\sum\sum_{1\leqslant i \leqslant j \leqslant n} a_i a_j \operatorname{Cov}(Y_i,Y_j)$, where the double sum is over all pairs $(i,j)$ with $i\lt j$.

$\quad\textbf c\,\,$ $\operatorname{Cov}(U_1, U_2)=\sum_{i=1}^n \sum_{j=1}^m a_i b_j \operatorname{Cov}(Y_i, X_j)$.

I am also given a hint to use this which I do not know how I can apply this to this question? Suppose that $Y_1, Y_2, ..., Y_n$ are independent normal random variables with $E(Y_i) = \beta_0 + \beta_1 x_i$ and $V(Y_i) = \sigma^2$, for $i = 1, 2, ..., n$. The maximum-likelihood estimators of $\beta_0$ and $\beta_1$ are the same as the least-squares estimators of Find $\operatorname{Cov}(\hat{\beta_0}, \hat{\beta_1})$.

What I got so far is $$\operatorname{Cov}(\hat{\beta_0}, \hat{\beta_1}) = \operatorname{Cov}(\bar{Y}-\hat{\beta_1}\bar{x}, \hat{\beta_1})$$

How can I move forward with this?

• We are missing quite a few definitions to help you with this. I'm guessing $\beta_0,\beta_1$ are fixed constants, and $x_i$ depends only on $i$? Are you given $x_1,\dots,x_n$, and have to estimate $\beta_0$ and $\beta_1$ by maximum likelihood? Could you define $\hat\beta_0,\hat\beta_1$ and $\tilde x$? – Ian May 10 '14 at 3:26

This assumes that the $x_{i}$ are fixed, the model is quite different if they are not. However, this assumption is reasonable given the expected value and variance of the $Y_{i}$. Usually, for any linear relationship, we have the model $$Y_{i}=\beta_{0}+\beta_{1}x_{i}+\epsilon_{i},$$ where $\beta_{0},\beta_{1} \in \mathbb{R}$, the $x_{i}$ are fixed, and $\mathbb{E}(\epsilon_{i})=0, \text{Var}(\epsilon_{i})=\sigma^{2}$ for a constant $\sigma^{2}$. This is very similar to your context, so I will assume this is the correct context. Denote the random variables $Y_{i}$ with lower case $y_{i}$ for consistency with typical regression notation. We have $$\text{Cov}(\hat{\beta_{0}},\hat{\beta_{1}})=\text{Cov}(\bar{y}-\hat{\beta_{1}}\bar{x},\hat{\beta_{1}}) \\=\text{Cov}(\bar{y},\hat{\beta_{1}})-\text{Cov}(\hat{\beta_{1}}\bar{x},\hat{\beta_{1}})\\=\text{Cov}(\bar{y},\hat{\beta_{1}})-\bar{x}\text{Cov}(\hat{\beta_{1}},\hat{\beta_{1}})\\=\text{Cov}(\bar{y},\hat{\beta_{1}})-\bar{x}\text{Var}(\hat{\beta_{1}}),$$ where the third equality comes from the fact that the $x_{i}$ are fixed and the fourth equality comes from the definitions of variance and covariance. From here, substitute the given expression for $\hat{\beta_{1}}$ (this expression and the one given for $\hat{\beta_{0}}$ can be derived from the usual method of maximum likelihood or from the least-squares estimators, they are equivalent), and use your knowledge about covariances and variances to finish the calculation (some information that you need will come from the proposition - "the following" in your box). Important hint: you will need to write $\hat{\beta_{1}}$ as a linear combination of the $y_{i}$, this is the key to completing the computation. Note that the covariance function and the vector space of random variables form an inner product space, so $\text{Cov}(\cdot,\cdot)$ is linear in the first argument (and second argument as a result of the symmetry clause of inner products) and thus, $$\text{Cov}\left(\sum_{i=1}^{n} c_{i}Z_{i},X\right)=\sum_{i=1}^{n} c_{i}\text{Cov}(Z_{i},X)$$ (this will be very useful as well) for constants $c_{1},c_{2},...,c_{n}$ and any random variables $Z_{1},Z_{2},...,Z_{n},X$. Also, here is yet another two useful identities/derivations:$$\sum_{i} x_{i}(x_{i} - \bar{x})=\sum_{i} x_{i}(x_{i} - \bar{x}) - \bar{x} \sum_{i}(x_{i} - \bar{x})\\=\sum_{i}(x_{i} - \bar{x})(x_{i}-\bar{x})=\sum_{i} (x_{i} - \bar{x})^{2}= S_{xx},$$ and similarly, $$\sum_{i} x_{i}(y_{i} - \bar{y})=\sum_{i} x_{i}(y_{i} - \bar{y}) - \bar{x} \sum_{i}(y_{i} - \bar{y})\\=\sum_{i}(x_{i} - \bar{x})(y_{i}-\bar{y})= S_{xy},$$ since we know that $\sum_{i}(x_{i}-\bar{x})=\sum_{i}(y_{i}-\bar{y})=0$.

• I am given an answer $cov(\bar{y}, \hat{\beta_1}) = 0$ ... I am wondering why? is this what you are hint is alluring to? – afsdf dfsaf May 10 '14 at 15:33
• I was going to give you that hint as well to make it easier, but looks like you were given it. My hint definitely applies to figuring that out. There are several ways you can go about computing $\text{Var}(\hat{\beta_{1}})$, and this is all that is left for you. Please accept/upvote if this was helpful. – afedder May 11 '14 at 3:01
• The variance of $\hat{\beta_{1}}$ ends up being $$\text{Var}(\hat{\beta_{1}})=\frac{\sigma^{2}}{S_{xx}}.$$ You should verify this, however. – afedder May 11 '14 at 3:04
• So your final answer should be $$\text{Cov}(\hat{\beta_{0}},\hat{\beta_{1}})=-\bar{x} \frac{\sigma^{2}}{S_{xx}},$$ but, again, you should check this as an exercise. – afedder May 11 '14 at 3:09
• Can you please upvote/accept? – afedder May 11 '14 at 3:12

$cov(\hat{\beta_0}, \hat{\beta_1})=cov (\hat{y}-\hat{\beta_1}\bar{x}, \hat{\beta_1})$

$=E[(\bar{y}-\hat{\beta_1}\bar{x})(\hat{\beta_1})]-E[\hat{y}-\hat{\beta_1} \bar{x}].E[\hat{\beta_1}]$

$=E[\hat{\beta_1}\bar{y}-\bar{x}\hat{\beta_1^2}]-(\hat{y}-\bar{x}\beta_1) (\beta_1)$

$=\bar{y}E[\hat{\beta_1}]-\bar{x}E[\hat{\beta_1^2}]-\bar{y}\beta_1+\bar{x}E[\hat{\beta_1}].E[\hat{\beta_1}]$

$=\bar{y}\beta_1-\bar{x}E[\hat{\beta_1^2}]-\bar{y}\beta_1+\bar{x}E[\hat{\beta_1}].E[\hat{\beta_1}]$

$=-\bar{x}(E[\hat{\beta_1^2}]-E[\hat{\beta_1}].E[\hat{\beta_1}])$

$=-\bar{x}. Var[\hat{\beta_1}]$

Variance of $\beta_1$ can be easily found in textbooks or online help. So I am omitting it.

$=-\bar{x}[\frac{\sigma ^2}{S_{xx}}]$