Consider a regression problem where the data values y1, . . . , yn are observed values of response variables Y1, . . . , Yn. In the notes we assume that, for given values x1, . . . , xn of the predictor variable, the Yi satisfy the simple linear regression model Yi = a + bxi + ei, where the ei are i.i.d.~ N(0, sigma^2). The least squares estimates of the regression parameter(s) are defined to be the values which minimise the sum of squares of the differences between the observed yi and the fitted values. For this model, E(Yi | xi) = a + bxi, so the least squares estimates are the values minimising [sigma (I=1,..n)(yi-(a+bxi))^2] Now consider alternative model which takes the form Yi = gammaxi+ei, i=1,...,n,where the ei satisfy the same assumptions as before but where there is now a single unknown regression parameter gamma . This model is sometimes used when it is clear from the problem description that the value of E(Y) must be zero if the corresponding x value is zero. Derive an expression, in terms of the xi and yi values, for the least squares estimate for gamma for this new model and suggest, with reasons, an appropriate estimate for sigma^2.

I don't really have a clue where to start, I know how to work out the least squares estimates of a and b using summary statistics, but I'm not at all sure how to apply this to finding gamma or sigma^2. I'm also not sure how adding on normally distributed ei might fit in?...


1 Answer 1


In the simple regression model, the least squares estimates for $a$ and $b$ were chosen to minimize $\sum_{i=1}^n (y_i - (a+ b x_i))^2$. (This is where the name least squares comes from.) To get the actual expressions you may have had to do some calculus.

Can you think about what changed in the second model, and figure out how you would modify the above procedure, to estimate $\gamma$?

Choose $\gamma$ to minimize $\sum_{i=1}^n (y_i - \gamma x_i)^2$.

Similarly, modify the procedure for estimating $\sigma^2$ in simple regression, in order to find the appropriate estimate in the second model.

  • $\begingroup$ No, you do not have to do some calculus. You can't have thought about this very much if you think that's the only way to do it, or even that it's the best way. $\endgroup$ Apr 11, 2018 at 23:11
  • $\begingroup$ @MichaelHardy Perhaps my choice of words was poor; I only wanted to suggest that calculus was an option, not that it was the only way or the best way. But personally I do not see why using calculus to solve the least squares problem is so bad. $\endgroup$
    – angryavian
    Apr 12, 2018 at 4:09
  • $\begingroup$ Using calculus is "bad" because it pointless complicated things, although that is actually a complicated story that is not best understood in these elementary problems. $\endgroup$ Apr 12, 2018 at 15:46

You must log in to answer this question.