# linear regression model beta estimate

Suppose we want to estimate $\beta$ by minimizing $L(\beta)=\sum_{i=1}^n(y_i-\beta x_i)^2+\lambda|\beta|$, where $\lambda$ is a fixed positive constant. Calculate the estimate.

How would I calculate $\beta$ in this scenario?

where the linear regression model is $$y_i = \beta x_i + \epsilon _i; i = 1,..., n$$

• Take the derivative $\dfrac{dL}{d\beta}$ and trying to get it equal to $0$ would be a start. Clearly it need not be continuous at $\beta=0$ so you should probably try positive and negative $\beta$ separately. – Henry Oct 13 '14 at 20:12

Consider first $\beta>0$. First order condition with respect to $\beta$ would imply $$\frac{\partial L(\beta)}{\partial\beta}=0 \Rightarrow -\sum^{n}_{i=1} 2(y_i-\beta x_i)x_i +\lambda=0\Rightarrow \sum^{n}_{i=1}(y_i-\beta x_i)x_i=\lambda/2\Rightarrow \beta=\frac{-\lambda/2+\sum^{n}_{i=1}y_ix_i}{\sum^{n}_{i=1}x^2_i}$$ So your optimal estimate is $$\beta^*=\frac{-\lambda/2+\sum^{n}_{i=1}y_ix_i}{\sum^{n}_{i=1}x^2_i}$$ if $\beta<0$ then $$\beta^*=\frac{\lambda/2+\sum^{n}_{i=1}y_ix_i}{\sum^{n}_{i=1}x^2_i}$$
• You may need to extend this to cases where $\left| \sum_i y_ix_i \right| \lt \lambda/2$. You could also mention that the second derivative is $2 \sum_i x_i^2$, i.e. positive – Henry Oct 13 '14 at 20:39
• I suspect that when $\left| \sum_i y_ix_i \right| \le \lambda/2$ you should be considering $\beta^*=0$ as otherwise it will be the wrong sign. – Henry Oct 13 '14 at 22:25