0
$\begingroup$

Let us assume that data is generated according to a true model $$y_i = \beta_{true}x_i + \epsilon_i$$ for $i = 1, ..., n$

Assume that $x_i$ are fixed, and $\epsilon_i$~ N(0, $\sigma^2$) independently.

Let $$\hat\beta =\frac{\sum^{n}_{i=1}y_ix_i}{\sum^{n}_{i=1}x^2_i + \lambda}$$ be the shrinkage estimator from the ridge regression.

How to calculate expectation and variance of $\hat\beta$, and mean squared error E$[(\hat\beta - \beta_{true})^2]$ ?

I'm stuck on this part for expectation. What to do next? $$E(\hat\beta)= E(\frac{\beta_{true}\sum^{n}_{i=1}x_i^2 + \sum^{n}_{i=1}x_i\epsilon_i}{\sum^{n}_{i=1}x^2_i + \lambda}) = (E(\frac{\beta_{true}\sum^{n}_{i=1}x_i^2 }{\sum^{n}_{i=1}x^2_i + \lambda})$$

$\endgroup$
4
  • 1
    $\begingroup$ Since $x_i$ is fixed $(E(\frac{\beta_{true}\sum^{n}_{i=1}x_i^2 }{\sum^{n}_{i=1}x^2_i + \lambda})=\frac{\beta_{true}\sum^{n}_{i=1}x_i^2 }{\sum^{n}_{i=1}x^2_i + \lambda}$. $\endgroup$
    – user103828
    Oct 22, 2014 at 7:21
  • $\begingroup$ @user103828 how would you calculate the variance? $\endgroup$ Oct 22, 2014 at 7:32
  • $\begingroup$ @user103828 Is the variance then $$\frac{\sigma^2\sum^{n}_{i=1}x_i^2 }{(\sum^{n}_{i=1}x^2_i + \lambda)^2}$$ $\endgroup$ Oct 22, 2014 at 7:42
  • 1
    $\begingroup$ yes that is correct. see below. $\endgroup$
    – user103828
    Oct 22, 2014 at 7:54

1 Answer 1

0
$\begingroup$

\begin{align*} E(\hat\beta) &= E\left(\frac{\beta_{true}\sum^{n}_{i=1}x_i^2 + \sum^{n}_{i=1}x_i\epsilon_i}{\sum^{n}_{i=1}x^2_i + \lambda}\right) = E \left(\frac{\beta_{true}\sum^{n}_{i=1}x_i^2 }{\sum^{n}_{i=1}x^2_i + \lambda} \right)=\frac{\beta_{true}\sum^{n}_{i=1}x_i^2 }{\sum^{n}_{i=1}x^2_i + \lambda} \\ var(\hat\beta) &= var\left(\frac{\beta_{true}\sum^{n}_{i=1}x_i^2 + \sum^{n}_{i=1}x_i\epsilon_i}{\sum^{n}_{i=1}x^2_i + \lambda}\right) = var \left(\frac{\sum^{n}_{i=1}x_i\varepsilon_i }{\sum^{n}_{i=1}x^2_i + \lambda} \right)=\frac{\sigma^2 \sum^{n}_{i=1}x_i^2 }{(\sum^{n}_{i=1}x^2_i + \lambda)^2} \end{align*} where the second equality follows because everything is a constant except $\varepsilon_i$ and the second equality follows because with $var(\varepsilon_i)=\sigma^2$, $\varepsilon_i$ are independent so we can take out the sum and taking out constant results in squaring them.

$\endgroup$
1
  • $\begingroup$ So my last question is... to minimize $\lambda$, do we set $\lambda = \sqrt{\frac{\sigma^2 \sum^{n}_{i=1}x^2_i}{\beta_{true}^2}}$?? I obtained this through the $$MSE = \frac{\sigma^2 \sum^{n}_{i=1}x_i^2 }{(\sum^{n}_{i=1}x^2_i + \lambda)^2} + \frac{\lambda^2 \beta_{true}^2 }{(\sum^{n}_{i=1}x^2_i + \lambda)^2}$$ Is this correct? $\endgroup$ Oct 22, 2014 at 8:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.