1
$\begingroup$

Assume that I have data that can be described by:

$y_i = \beta x_i + \epsilon_i, \epsilon_i \sim (0,\sigma_{\epsilon})$,

then the least squares estimator is given by

$\hat{\beta_1} = \frac{\sum_{i=1}^N x_iy_i}{\sum_{i=1}^N x_i^2}$.

Why is it wrong to use the following estimator?

$\hat{\beta_2} = \frac{1}{N}\sum_{i=1}^N \frac{y_i}{x_i}$.

Do they not estimate the same parameter?

$\endgroup$
4
  • 1
    $\begingroup$ Consider case where (x,y) = (0,0.003) is in the set. $\endgroup$
    – greggo
    Commented Aug 13, 2014 at 17:02
  • $\begingroup$ I see your point, but if $x_i \neq 0$ do they give you the same estimate? $\endgroup$
    – trienko
    Commented Aug 13, 2014 at 17:04
  • 1
    $\begingroup$ No, the first one gives you the estimate that minimizes the sum of error squared - which includes ignoring all (0,y) points, since the value of beta has no effect on those. Likewise (x,y) where x is small will have lower relative effect on beta than where x is large. The second one will give a different result in the general case. $\endgroup$
    – greggo
    Commented Aug 13, 2014 at 17:08
  • $\begingroup$ Isn't the $\mathbb{E}[\hat{\beta_1}] = \beta$, and the $\mathbb{E}[\hat{\beta_2}]= \beta$. If I calculate the two expectations it seems to be true. Only if $x_i \neq 0$. $\endgroup$
    – trienko
    Commented Aug 13, 2014 at 17:12

3 Answers 3

1
$\begingroup$

The ordinary least squares estimator, $\widehat \beta_{OLS}=\sum \frac{x_i y_i}{x_i^2}$, as others have mentioned minimizes the sum of squares error $\widehat \beta_{OLS}=argmin_b \sum(y_i -bx_i)^2$. The major reason that it is so widely used is because it is BLUE, best linear unbiased estimator (http://en.wikipedia.org/wiki/Gauss%E2%80%93Markov_theorem) i.e. among all unbiased estimators is has lowest variance ($var(\widehat \beta_{OLS})$ is smallest among all unbiased estimators). Hence, your estimator is unbiased but $\widehat \beta_{OLS}$ is also unbiased and has lower variance.

There are other estimators that are sometimes thought to be better than $\widehat \beta_{OLS}=\sum \frac{x_i y_i}{x_i^2}$. For example, an estimator that minimizes the absolute difference $\widehat \beta_{LAD}=argmin_b \sum |y_i -bx_i|$ is less sensitive to outliers http://en.wikipedia.org/wiki/Least_absolute_deviations.

Actually, I think the estimator $\widehat \beta_1=\frac{1}{N}\sum \frac{y_i}{x_i}$ is not a very good one because it is sensitive to small values of $x_i$ while the other estimator are not... if anything I think $\widehat \beta_2=\frac{\sum y_i}{\sum x_i}$ is better.... Also it is hard to see how to extend the estimator $\widehat \beta_1$ to a multivariate regression.

Let's calculate the variances. Assume for simplicity that the $x_i$ are nonrandom. Then, $$var(\widehat \beta_1)=var(\frac{1}{N}\sum \frac{y_i}{x_i}) =\frac{1}{N^2}var( \sum \frac{(x_i\beta+\varepsilon_i)}{x_i}) =\frac{\sigma^2_{\varepsilon}}{N^2} \sum \frac{1}{x_i^2} \\ var(\widehat \beta_2)=var( \frac{\sum y_i}{\sum x_i})=var( \frac{\sum(x_i\beta+\varepsilon_i)}{\sum x_i}) = \frac{N\sigma^2_{\varepsilon}}{(\sum x_i)^2} \\ var(\widehat \beta_{OLS})=var( \frac{\sum x_iy_i}{\sum x_i^2}) =var( \frac{\sum x_i(x_i\beta+\varepsilon_i)}{\sum x_i^2}) =var( \frac{\sum x_i\varepsilon_i}{\sum x_i^2}) = \frac{\sigma^2_{\varepsilon} }{\sum x_i^2}$$ we can see that the issue with $\widehat \beta_1$ is when $x_i$ is small... I'll let you apply Cauchy-Swartz to actually prove that $var(\widehat \beta_{OLS})\leq var(\widehat \beta_{1})$.

$\endgroup$
3
  • $\begingroup$ How can I calculate the variance of $\hat{\beta_2}$ and compare it with $\hat{\beta_1}$? The variance of $\hat{\beta_1} = \frac{\sigma_{\epsilon}^2}{\sum x_i^2}$? I am really trying to understand this? $\endgroup$
    – trienko
    Commented Aug 14, 2014 at 7:35
  • 1
    $\begingroup$ I will add the calculations of variance to answer. $\endgroup$
    – user103828
    Commented Aug 14, 2014 at 9:33
  • $\begingroup$ I think you missed a square in your last variance calculation... $\endgroup$
    – trienko
    Commented Aug 14, 2014 at 11:31
1
$\begingroup$

The estimator you define also makes sense. But like others said this is the different estimator. Actually it solves the different problem: $$ y_i/x_i = \beta +\epsilon_i $$ and minimizes the different sum of squares $\sum_i(y_i/x_i-\beta)^2$. Of course $x_i$ cannot be zero in such case. So the main idea under choosing the estimator what error exactly you want to minimize.

$\endgroup$
3
  • $\begingroup$ I think it should be $\varepsilon_i/x_i$ in the equation. $\endgroup$
    – user103828
    Commented Aug 13, 2014 at 18:27
  • $\begingroup$ No. $\epsilon_i$ is not observable. This problem is not equivalent to the original. If you divide $\epsilon_i$ by $x_i$ ($x_i\ne 0$ ) you don't change the original problem. $\endgroup$ Commented Aug 13, 2014 at 18:38
  • $\begingroup$ sorry, misread.... i thought you were saying that they were equivalent. $\endgroup$
    – user103828
    Commented Aug 14, 2014 at 10:41
1
$\begingroup$

It's not 'wrong' to use that estimator, which is the mean of the ratios given by each point.

But they are not the same; the first one is the unique value of $\beta$ which minimizes

$$ E(\beta) = \sum_i \left( \beta x_i - y_i \right)^2 $$ and will give a different result in the general case.

In the case where all the $x_i, y_i$ can be exactly fitted with a specific $\beta$ value, both estimates will yield that value.

Qualitatively, $\beta_2$ will tend to give larger fitting errors for points with larger values of $x$, and a better fit for smaller values, compared to $\beta_1$

Also, in the case where all your $x$ fall in a narrow relative range, e.g. $100 \le x \le 104 $; so the ratio $\max (|x_i|) / \min(|x_i|)$ is not much larger than 1: there will be very little difference between the two estimates, even when the points don't fall close to a straight line.

$\endgroup$
2
  • $\begingroup$ How do I know that $\hat{\beta_1}$ is a better estimator than $\hat{\beta_2}$? It must be more stable, I am very confused? $\endgroup$
    – trienko
    Commented Aug 13, 2014 at 17:20
  • $\begingroup$ The 'quality' of an estimator is defined by what you want to do with the estimate, and how well it works there. 'Least-squares error' is simply one very commonly used approach. $\endgroup$
    – greggo
    Commented Aug 13, 2014 at 17:23

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .