I'm reviewing old notes, and I know I solved this way back when, but can't remember how to know:

Consider the simple linear regression model:

$$Y_i = a + bX_i + \epsilon_i$$

where $Y_i$ is the dependent variable, $X_i$ is the independent variable, and $a$ and $b$ are model parameters, and $\epsilon_i$ is the error term. Derive the estimators for parameters a and b when trying to minimize the sum of the squared error terms.

I think the first steps are to rearrange the equation into $Y_i - a - bX_i = \epsilon_i$ and then differentiate the equation with respect to $a$ and $b$. But then I have no idea where to go from there.

Much thanks.


1 Answer 1


Write down an expression for the squared error: $$e=\sum_i (y_i - a x_i - b)^2$$

Then, set $\frac{\partial e}{\partial a} = 0$, $\frac{\partial e}{\partial b} = 0$ and solve the resulting system of equations for $a$ and $b$. Since the expression for $e$ is convex in $a$ and $b$, this minimizes the squared error (or you can use the second derivative test).

A smarter way to do this would be to use the orthogonality principle from linear algebra.

  • $\begingroup$ Thanks for the help - I'll follow it through those lines. I don't follow this statement however: "Since the expression for e is convex in a and b..." What does "convex in" mean? I don't think it affects what I understand i have to do, but I'd like to understand your thinking there... $\endgroup$
    – traggatmot
    Commented Aug 4, 2014 at 2:40
  • $\begingroup$ It means the error is a convex function of $a$ and $b$ (which implies that any stationary point [where the gradient is zero] is a minimum). See Boyd and Vandenberghe's Convex Optimization, for example, for details on convex functions. You need an argument for why setting the gradient of the error (the partial derivatives) to zero minimizes the error, which could be done by using convexity or using something like the second derivative test. $\endgroup$
    – Batman
    Commented Aug 4, 2014 at 2:45
  • $\begingroup$ Gotcha. Understand it now. $\endgroup$
    – traggatmot
    Commented Aug 4, 2014 at 2:47

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .