1
$\begingroup$

Assume that I have data that can be described by:

$y_i = \beta x_i + \epsilon_i, \epsilon_i \sim (0,\sigma_{\epsilon})$,

then the least squares estimator is given by

$\hat{\beta_1} = \frac{\sum_{i=1}^N x_iy_i}{\sum_{i=1}^N x_i^2}$.

Why is it wrong to use the following estimator?

$\hat{\beta_2} = \frac{1}{N}\sum_{i=1}^N \frac{y_i}{x_i}$.

Do they not estimate the same parameter?

$\endgroup$
  • 1
    $\begingroup$ Consider case where (x,y) = (0,0.003) is in the set. $\endgroup$ – greggo Aug 13 '14 at 17:02
  • $\begingroup$ I see your point, but if $x_i \neq 0$ do they give you the same estimate? $\endgroup$ – trienko Aug 13 '14 at 17:04
  • 1
    $\begingroup$ No, the first one gives you the estimate that minimizes the sum of error squared - which includes ignoring all (0,y) points, since the value of beta has no effect on those. Likewise (x,y) where x is small will have lower relative effect on beta than where x is large. The second one will give a different result in the general case. $\endgroup$ – greggo Aug 13 '14 at 17:08
  • $\begingroup$ Isn't the $\mathbb{E}[\hat{\beta_1}] = \beta$, and the $\mathbb{E}[\hat{\beta_2}]= \beta$. If I calculate the two expectations it seems to be true. Only if $x_i \neq 0$. $\endgroup$ – trienko Aug 13 '14 at 17:12
1
$\begingroup$

The ordinary least squares estimator, $\widehat \beta_{OLS}=\sum \frac{x_i y_i}{x_i^2}$, as others have mentioned minimizes the sum of squares error $\widehat \beta_{OLS}=argmin_b \sum(y_i -bx_i)^2$. The major reason that it is so widely used is because it is BLUE, best linear unbiased estimator (http://en.wikipedia.org/wiki/Gauss%E2%80%93Markov_theorem) i.e. among all unbiased estimators is has lowest variance ($var(\widehat \beta_{OLS})$ is smallest among all unbiased estimators). Hence, your estimator is unbiased but $\widehat \beta_{OLS}$ is also unbiased and has lower variance.

There are other estimators that are sometimes thought to be better than $\widehat \beta_{OLS}=\sum \frac{x_i y_i}{x_i^2}$. For example, an estimator that minimizes the absolute difference $\widehat \beta_{LAD}=argmin_b \sum |y_i -bx_i|$ is less sensitive to outliers http://en.wikipedia.org/wiki/Least_absolute_deviations.

Actually, I think the estimator $\widehat \beta_1=\frac{1}{N}\sum \frac{y_i}{x_i}$ is not a very good one because it is sensitive to small values of $x_i$ while the other estimator are not... if anything I think $\widehat \beta_2=\frac{\sum y_i}{\sum x_i}$ is better.... Also it is hard to see how to extend the estimator $\widehat \beta_1$ to a multivariate regression.

Let's calculate the variances. Assume for simplicity that the $x_i$ are nonrandom. Then, $$var(\widehat \beta_1)=var(\frac{1}{N}\sum \frac{y_i}{x_i}) =\frac{1}{N^2}var( \sum \frac{(x_i\beta+\varepsilon_i)}{x_i}) =\frac{\sigma^2_{\varepsilon}}{N^2} \sum \frac{1}{x_i^2} \\ var(\widehat \beta_2)=var( \frac{\sum y_i}{\sum x_i})=var( \frac{\sum(x_i\beta+\varepsilon_i)}{\sum x_i}) = \frac{N\sigma^2_{\varepsilon}}{(\sum x_i)^2} \\ var(\widehat \beta_{OLS})=var( \frac{\sum x_iy_i}{\sum x_i^2}) =var( \frac{\sum x_i(x_i\beta+\varepsilon_i)}{\sum x_i^2}) =var( \frac{\sum x_i\varepsilon_i}{\sum x_i^2}) = \frac{\sigma^2_{\varepsilon} }{\sum x_i^2}$$ we can see that the issue with $\widehat \beta_1$ is when $x_i$ is small... I'll let you apply Cauchy-Swartz to actually prove that $var(\widehat \beta_{OLS})\leq var(\widehat \beta_{1})$.

$\endgroup$
  • $\begingroup$ How can I calculate the variance of $\hat{\beta_2}$ and compare it with $\hat{\beta_1}$? The variance of $\hat{\beta_1} = \frac{\sigma_{\epsilon}^2}{\sum x_i^2}$? I am really trying to understand this? $\endgroup$ – trienko Aug 14 '14 at 7:35
  • 1
    $\begingroup$ I will add the calculations of variance to answer. $\endgroup$ – user103828 Aug 14 '14 at 9:33
  • $\begingroup$ I think you missed a square in your last variance calculation... $\endgroup$ – trienko Aug 14 '14 at 11:31
1
$\begingroup$

The estimator you define also makes sense. But like others said this is the different estimator. Actually it solves the different problem: $$ y_i/x_i = \beta +\epsilon_i $$ and minimizes the different sum of squares $\sum_i(y_i/x_i-\beta)^2$. Of course $x_i$ cannot be zero in such case. So the main idea under choosing the estimator what error exactly you want to minimize.

$\endgroup$
  • $\begingroup$ I think it should be $\varepsilon_i/x_i$ in the equation. $\endgroup$ – user103828 Aug 13 '14 at 18:27
  • $\begingroup$ No. $\epsilon_i$ is not observable. This problem is not equivalent to the original. If you divide $\epsilon_i$ by $x_i$ ($x_i\ne 0$ ) you don't change the original problem. $\endgroup$ – Alexander Vigodner Aug 13 '14 at 18:38
  • $\begingroup$ sorry, misread.... i thought you were saying that they were equivalent. $\endgroup$ – user103828 Aug 14 '14 at 10:41
1
$\begingroup$

It's not 'wrong' to use that estimator, which is the mean of the ratios given by each point.

But they are not the same; the first one is the unique value of $\beta$ which minimizes

$$ E(\beta) = \sum_i \left( \beta x_i - y_i \right)^2 $$ and will give a different result in the general case.

In the case where all the $x_i, y_i$ can be exactly fitted with a specific $\beta$ value, both estimates will yield that value.

Qualitatively, $\beta_2$ will tend to give larger fitting errors for points with larger values of $x$, and a better fit for smaller values, compared to $\beta_1$

Also, in the case where all your $x$ fall in a narrow relative range, e.g. $100 \le x \le 104 $; so the ratio $\max (|x_i|) / \min(|x_i|)$ is not much larger than 1: there will be very little difference between the two estimates, even when the points don't fall close to a straight line.

$\endgroup$
  • $\begingroup$ How do I know that $\hat{\beta_1}$ is a better estimator than $\hat{\beta_2}$? It must be more stable, I am very confused? $\endgroup$ – trienko Aug 13 '14 at 17:20
  • $\begingroup$ The 'quality' of an estimator is defined by what you want to do with the estimate, and how well it works there. 'Least-squares error' is simply one very commonly used approach. $\endgroup$ – greggo Aug 13 '14 at 17:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.