0
$\begingroup$

Let us assume that data is generated according to a true model $$y_i = \beta_{true}x_i + \epsilon_i$$ for $i = 1, ..., n$

Assume that $x_i$ are fixed, and $\epsilon_i$~ N(0, $\sigma^2$) independently.

Let $$\hat\beta =\frac{\sum^{n}_{i=1}y_ix_i}{\sum^{n}_{i=1}x^2_i + \lambda}$$ be the shrinkage estimator from the ridge regression.

How to calculate expectation and variance of $\hat\beta$, and mean squared error E$[(\hat\beta - \beta_{true})^2]$ ?

I'm stuck on this part for expectation. What to do next? $$E(\hat\beta)= E(\frac{\beta_{true}\sum^{n}_{i=1}x_i^2 + \sum^{n}_{i=1}x_i\epsilon_i}{\sum^{n}_{i=1}x^2_i + \lambda}) = (E(\frac{\beta_{true}\sum^{n}_{i=1}x_i^2 }{\sum^{n}_{i=1}x^2_i + \lambda})$$

$\endgroup$
4
  • 1
    $\begingroup$ Since $x_i$ is fixed $(E(\frac{\beta_{true}\sum^{n}_{i=1}x_i^2 }{\sum^{n}_{i=1}x^2_i + \lambda})=\frac{\beta_{true}\sum^{n}_{i=1}x_i^2 }{\sum^{n}_{i=1}x^2_i + \lambda}$. $\endgroup$ – user103828 Oct 22 '14 at 7:21
  • $\begingroup$ @user103828 how would you calculate the variance? $\endgroup$ – user3325193 Oct 22 '14 at 7:32
  • $\begingroup$ @user103828 Is the variance then $$\frac{\sigma^2\sum^{n}_{i=1}x_i^2 }{(\sum^{n}_{i=1}x^2_i + \lambda)^2}$$ $\endgroup$ – user3325193 Oct 22 '14 at 7:42
  • 1
    $\begingroup$ yes that is correct. see below. $\endgroup$ – user103828 Oct 22 '14 at 7:54
0
$\begingroup$

\begin{align*} E(\hat\beta) &= E\left(\frac{\beta_{true}\sum^{n}_{i=1}x_i^2 + \sum^{n}_{i=1}x_i\epsilon_i}{\sum^{n}_{i=1}x^2_i + \lambda}\right) = E \left(\frac{\beta_{true}\sum^{n}_{i=1}x_i^2 }{\sum^{n}_{i=1}x^2_i + \lambda} \right)=\frac{\beta_{true}\sum^{n}_{i=1}x_i^2 }{\sum^{n}_{i=1}x^2_i + \lambda} \\ var(\hat\beta) &= var\left(\frac{\beta_{true}\sum^{n}_{i=1}x_i^2 + \sum^{n}_{i=1}x_i\epsilon_i}{\sum^{n}_{i=1}x^2_i + \lambda}\right) = var \left(\frac{\sum^{n}_{i=1}x_i\varepsilon_i }{\sum^{n}_{i=1}x^2_i + \lambda} \right)=\frac{\sigma^2 \sum^{n}_{i=1}x_i^2 }{(\sum^{n}_{i=1}x^2_i + \lambda)^2} \end{align*} where the second equality follows because everything is a constant except $\varepsilon_i$ and the second equality follows because with $var(\varepsilon_i)=\sigma^2$, $\varepsilon_i$ are independent so we can take out the sum and taking out constant results in squaring them.

$\endgroup$
1
  • $\begingroup$ So my last question is... to minimize $\lambda$, do we set $\lambda = \sqrt{\frac{\sigma^2 \sum^{n}_{i=1}x^2_i}{\beta_{true}^2}}$?? I obtained this through the $$MSE = \frac{\sigma^2 \sum^{n}_{i=1}x_i^2 }{(\sum^{n}_{i=1}x^2_i + \lambda)^2} + \frac{\lambda^2 \beta_{true}^2 }{(\sum^{n}_{i=1}x^2_i + \lambda)^2}$$ Is this correct? $\endgroup$ – user3325193 Oct 22 '14 at 8:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.