# linear regression, expectation and mean squared error

Let us assume that data is generated according to a true model $$y_i = \beta_{true}x_i + \epsilon_i$$ for $i = 1, ..., n$

Assume that $x_i$ are fixed, and $\epsilon_i$~ N(0, $\sigma^2$) independently.

Let $$\hat\beta =\frac{\sum^{n}_{i=1}y_ix_i}{\sum^{n}_{i=1}x^2_i + \lambda}$$ be the shrinkage estimator from the ridge regression.

How to calculate expectation and variance of $\hat\beta$, and mean squared error E$[(\hat\beta - \beta_{true})^2]$ ?

I'm stuck on this part for expectation. What to do next? $$E(\hat\beta)= E(\frac{\beta_{true}\sum^{n}_{i=1}x_i^2 + \sum^{n}_{i=1}x_i\epsilon_i}{\sum^{n}_{i=1}x^2_i + \lambda}) = (E(\frac{\beta_{true}\sum^{n}_{i=1}x_i^2 }{\sum^{n}_{i=1}x^2_i + \lambda})$$

• Since $x_i$ is fixed $(E(\frac{\beta_{true}\sum^{n}_{i=1}x_i^2 }{\sum^{n}_{i=1}x^2_i + \lambda})=\frac{\beta_{true}\sum^{n}_{i=1}x_i^2 }{\sum^{n}_{i=1}x^2_i + \lambda}$. Oct 22, 2014 at 7:21
• @user103828 how would you calculate the variance? Oct 22, 2014 at 7:32
• @user103828 Is the variance then $$\frac{\sigma^2\sum^{n}_{i=1}x_i^2 }{(\sum^{n}_{i=1}x^2_i + \lambda)^2}$$ Oct 22, 2014 at 7:42
• yes that is correct. see below. Oct 22, 2014 at 7:54

\begin{align*} E(\hat\beta) &= E\left(\frac{\beta_{true}\sum^{n}_{i=1}x_i^2 + \sum^{n}_{i=1}x_i\epsilon_i}{\sum^{n}_{i=1}x^2_i + \lambda}\right) = E \left(\frac{\beta_{true}\sum^{n}_{i=1}x_i^2 }{\sum^{n}_{i=1}x^2_i + \lambda} \right)=\frac{\beta_{true}\sum^{n}_{i=1}x_i^2 }{\sum^{n}_{i=1}x^2_i + \lambda} \\ var(\hat\beta) &= var\left(\frac{\beta_{true}\sum^{n}_{i=1}x_i^2 + \sum^{n}_{i=1}x_i\epsilon_i}{\sum^{n}_{i=1}x^2_i + \lambda}\right) = var \left(\frac{\sum^{n}_{i=1}x_i\varepsilon_i }{\sum^{n}_{i=1}x^2_i + \lambda} \right)=\frac{\sigma^2 \sum^{n}_{i=1}x_i^2 }{(\sum^{n}_{i=1}x^2_i + \lambda)^2} \end{align*} where the second equality follows because everything is a constant except $\varepsilon_i$ and the second equality follows because with $var(\varepsilon_i)=\sigma^2$, $\varepsilon_i$ are independent so we can take out the sum and taking out constant results in squaring them.
• So my last question is... to minimize $\lambda$, do we set $\lambda = \sqrt{\frac{\sigma^2 \sum^{n}_{i=1}x^2_i}{\beta_{true}^2}}$?? I obtained this through the $$MSE = \frac{\sigma^2 \sum^{n}_{i=1}x_i^2 }{(\sum^{n}_{i=1}x^2_i + \lambda)^2} + \frac{\lambda^2 \beta_{true}^2 }{(\sum^{n}_{i=1}x^2_i + \lambda)^2}$$ Is this correct? Oct 22, 2014 at 8:04