# Linear regression throug the origin versus mean?

Assume that I have data that can be described by:

$y_i = \beta x_i + \epsilon_i, \epsilon_i \sim (0,\sigma_{\epsilon})$,

then the least squares estimator is given by

$\hat{\beta_1} = \frac{\sum_{i=1}^N x_iy_i}{\sum_{i=1}^N x_i^2}$.

Why is it wrong to use the following estimator?

$\hat{\beta_2} = \frac{1}{N}\sum_{i=1}^N \frac{y_i}{x_i}$.

Do they not estimate the same parameter?

• Consider case where (x,y) = (0,0.003) is in the set. – greggo Aug 13 '14 at 17:02
• I see your point, but if $x_i \neq 0$ do they give you the same estimate? – trienko Aug 13 '14 at 17:04
• No, the first one gives you the estimate that minimizes the sum of error squared - which includes ignoring all (0,y) points, since the value of beta has no effect on those. Likewise (x,y) where x is small will have lower relative effect on beta than where x is large. The second one will give a different result in the general case. – greggo Aug 13 '14 at 17:08
• Isn't the $\mathbb{E}[\hat{\beta_1}] = \beta$, and the $\mathbb{E}[\hat{\beta_2}]= \beta$. If I calculate the two expectations it seems to be true. Only if $x_i \neq 0$. – trienko Aug 13 '14 at 17:12

The ordinary least squares estimator, $\widehat \beta_{OLS}=\sum \frac{x_i y_i}{x_i^2}$, as others have mentioned minimizes the sum of squares error $\widehat \beta_{OLS}=argmin_b \sum(y_i -bx_i)^2$. The major reason that it is so widely used is because it is BLUE, best linear unbiased estimator (http://en.wikipedia.org/wiki/Gauss%E2%80%93Markov_theorem) i.e. among all unbiased estimators is has lowest variance ($var(\widehat \beta_{OLS})$ is smallest among all unbiased estimators). Hence, your estimator is unbiased but $\widehat \beta_{OLS}$ is also unbiased and has lower variance.

There are other estimators that are sometimes thought to be better than $\widehat \beta_{OLS}=\sum \frac{x_i y_i}{x_i^2}$. For example, an estimator that minimizes the absolute difference $\widehat \beta_{LAD}=argmin_b \sum |y_i -bx_i|$ is less sensitive to outliers http://en.wikipedia.org/wiki/Least_absolute_deviations.

Actually, I think the estimator $\widehat \beta_1=\frac{1}{N}\sum \frac{y_i}{x_i}$ is not a very good one because it is sensitive to small values of $x_i$ while the other estimator are not... if anything I think $\widehat \beta_2=\frac{\sum y_i}{\sum x_i}$ is better.... Also it is hard to see how to extend the estimator $\widehat \beta_1$ to a multivariate regression.

Let's calculate the variances. Assume for simplicity that the $x_i$ are nonrandom. Then, $$var(\widehat \beta_1)=var(\frac{1}{N}\sum \frac{y_i}{x_i}) =\frac{1}{N^2}var( \sum \frac{(x_i\beta+\varepsilon_i)}{x_i}) =\frac{\sigma^2_{\varepsilon}}{N^2} \sum \frac{1}{x_i^2} \\ var(\widehat \beta_2)=var( \frac{\sum y_i}{\sum x_i})=var( \frac{\sum(x_i\beta+\varepsilon_i)}{\sum x_i}) = \frac{N\sigma^2_{\varepsilon}}{(\sum x_i)^2} \\ var(\widehat \beta_{OLS})=var( \frac{\sum x_iy_i}{\sum x_i^2}) =var( \frac{\sum x_i(x_i\beta+\varepsilon_i)}{\sum x_i^2}) =var( \frac{\sum x_i\varepsilon_i}{\sum x_i^2}) = \frac{\sigma^2_{\varepsilon} }{\sum x_i^2}$$ we can see that the issue with $\widehat \beta_1$ is when $x_i$ is small... I'll let you apply Cauchy-Swartz to actually prove that $var(\widehat \beta_{OLS})\leq var(\widehat \beta_{1})$.

• How can I calculate the variance of $\hat{\beta_2}$ and compare it with $\hat{\beta_1}$? The variance of $\hat{\beta_1} = \frac{\sigma_{\epsilon}^2}{\sum x_i^2}$? I am really trying to understand this? – trienko Aug 14 '14 at 7:35
• I will add the calculations of variance to answer. – user103828 Aug 14 '14 at 9:33
• I think you missed a square in your last variance calculation... – trienko Aug 14 '14 at 11:31

The estimator you define also makes sense. But like others said this is the different estimator. Actually it solves the different problem: $$y_i/x_i = \beta +\epsilon_i$$ and minimizes the different sum of squares $\sum_i(y_i/x_i-\beta)^2$. Of course $x_i$ cannot be zero in such case. So the main idea under choosing the estimator what error exactly you want to minimize.

• I think it should be $\varepsilon_i/x_i$ in the equation. – user103828 Aug 13 '14 at 18:27
• No. $\epsilon_i$ is not observable. This problem is not equivalent to the original. If you divide $\epsilon_i$ by $x_i$ ($x_i\ne 0$ ) you don't change the original problem. – Alexander Vigodner Aug 13 '14 at 18:38
• sorry, misread.... i thought you were saying that they were equivalent. – user103828 Aug 14 '14 at 10:41

It's not 'wrong' to use that estimator, which is the mean of the ratios given by each point.

But they are not the same; the first one is the unique value of $\beta$ which minimizes

$$E(\beta) = \sum_i \left( \beta x_i - y_i \right)^2$$ and will give a different result in the general case.

In the case where all the $x_i, y_i$ can be exactly fitted with a specific $\beta$ value, both estimates will yield that value.

Qualitatively, $\beta_2$ will tend to give larger fitting errors for points with larger values of $x$, and a better fit for smaller values, compared to $\beta_1$

Also, in the case where all your $x$ fall in a narrow relative range, e.g. $100 \le x \le 104$; so the ratio $\max (|x_i|) / \min(|x_i|)$ is not much larger than 1: there will be very little difference between the two estimates, even when the points don't fall close to a straight line.

• How do I know that $\hat{\beta_1}$ is a better estimator than $\hat{\beta_2}$? It must be more stable, I am very confused? – trienko Aug 13 '14 at 17:20
• The 'quality' of an estimator is defined by what you want to do with the estimate, and how well it works there. 'Least-squares error' is simply one very commonly used approach. – greggo Aug 13 '14 at 17:23