Suppose we want to estimate $\beta$ by minimizing $L(\beta)=\sum_{i=1}^n(y_i-\beta x_i)^2+\lambda|\beta|$, where $\lambda$ is a fixed positive constant. Calculate the estimate.

How would I calculate $\beta$ in this scenario?

where the linear regression model is $$y_i = \beta x_i + \epsilon _i; i = 1,..., n$$

  • 1
    $\begingroup$ Take the derivative $\dfrac{dL}{d\beta}$ and trying to get it equal to $0$ would be a start. Clearly it need not be continuous at $\beta=0$ so you should probably try positive and negative $\beta$ separately. $\endgroup$ – Henry Oct 13 '14 at 20:12

Consider first $\beta>0$. First order condition with respect to $\beta$ would imply $$\frac{\partial L(\beta)}{\partial\beta}=0 \Rightarrow -\sum^{n}_{i=1} 2(y_i-\beta x_i)x_i +\lambda=0\Rightarrow \sum^{n}_{i=1}(y_i-\beta x_i)x_i=\lambda/2\Rightarrow \beta=\frac{-\lambda/2+\sum^{n}_{i=1}y_ix_i}{\sum^{n}_{i=1}x^2_i}$$ So your optimal estimate is $$\beta^*=\frac{-\lambda/2+\sum^{n}_{i=1}y_ix_i}{\sum^{n}_{i=1}x^2_i}$$ if $\beta<0$ then $$\beta^*=\frac{\lambda/2+\sum^{n}_{i=1}y_ix_i}{\sum^{n}_{i=1}x^2_i}$$

  • $\begingroup$ You may need to extend this to cases where $\left| \sum_i y_ix_i \right| \lt \lambda/2$. You could also mention that the second derivative is $2 \sum_i x_i^2$, i.e. positive $\endgroup$ – Henry Oct 13 '14 at 20:39
  • $\begingroup$ @ Henry: You are correct. A positive second derivative would imply that it is a minimum. But I skipped the verification part. What about the other cases? $\endgroup$ – Arian Oct 13 '14 at 22:17
  • 2
    $\begingroup$ I suspect that when $\left| \sum_i y_ix_i \right| \le \lambda/2$ you should be considering $\beta^*=0$ as otherwise it will be the wrong sign. $\endgroup$ – Henry Oct 13 '14 at 22:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.