I have a situation which can be handled well through a simple linear regression model. That is, I have data points with known x values, y values with a given amount of error, and an ideal fit of the form $y = \alpha + \beta x$. It's easily possible to get unbiased (but correlated) estimators for $\alpha$ and $\beta$ through linear regression, but I have a case where it would be useful to have an unbiased estimator of $\beta^{-1}$, and I haven't been able to figure out if this is possible.

Some things I've tried which don't work:

  1. Using the inverse of the estimator, $\hat{\beta}^{-1}$. This can be shown through Taylor expansion to be biased. Eg. if $\beta > 0$, then this will be biased high proportional to $\sigma^2 (\hat{\beta})$ to lowest order.
  2. Performing the inverse linear regression, instead trying to fit $x = -\alpha/\beta + y/\beta$ for $-\alpha/\beta$ and $\beta^{-1}$. The problem here is that this breaks the assumption going into the linear regression model, that error is on the dependent variable only. Instead of getting an unbiased estimator of $\beta^{-1}$, you end up getting an unbiased estimator of $\beta\sigma^2(x)/\sigma^2(y)$.

So, is there any known way to do this?

  • $\begingroup$ Not confident enough to make it an answer, but one thing you could try is looking at the eigendecomposition of the covariance matrix of your data. Then the eigenvector associated with the smallest eigenvalue will point in the direction of the opposite of the reciprocal of $\beta$. That is, you would have found the direction of $-\frac{1}{\beta}x$. $\endgroup$ – dt688 May 26 '17 at 11:25
  • $\begingroup$ Have you tried using maximum likelihood? $\endgroup$ – dreamer May 26 '17 at 14:06
  • $\begingroup$ I did look into a maximum likelihood solution, but it ended up with either a ridiculously complicated formula, or an approximation which isn't significantly different from iteratively correcting for the calculated bias of using eg. the inverse of the estimator. As for the eigendecomposition possibility, I haven't tried that, so I'll have to see what it ends up giving me. Thanks for the suggestions! $\endgroup$ – Bryan Gillis May 30 '17 at 16:01

As you say, a second-order Taylor expansion for $g(\hat{\beta})=1/\hat{\beta}$ around $\beta$ gives: \begin{equation} \frac 1 {\hat{\beta}} = \frac 1 \beta - \frac{\hat{\beta}-\beta}{\beta^2} + \frac{(\hat{\beta}-\beta)^2}{\beta^3} \end{equation} Hence, $\operatorname E(1/\hat{\beta})=(1/\beta)+\operatorname{Var}(\hat{\beta})/\beta^3$ and is upward bias. So $(1/\hat{\beta})- \operatorname{Var}(\hat{\beta})/\hat{\beta}^3$ should, in principle, give a better estimate for $1/\beta$. Probably this is what you had thought of already and rejected. So apologies if so.

  • $\begingroup$ That's roughly what I ended up doing, since I haven't found a way to get a perfectly unbiased estimator. This requires an iterative approach to successively reduce bias (and the calculations start to rely on the assumption of Gaussianity in the error on $\beta$), but seems to be the most practical approach. $\endgroup$ – Bryan Gillis Feb 5 '18 at 20:49
  • $\begingroup$ Was thinking again on this and was reminded of the invariance property of MLE i.e. MLE of $g(\beta)$ = $g($MLE of $\beta)$. $\endgroup$ – ELevy Mar 2 '18 at 19:56
  • $\begingroup$ @ELevy : What is your criterion for goodness of an estimator? If one goes by mean squared error, an estimator with lower bias is not always a better estimator. And perhaps one should note some incompleteness in this answer: $\operatorname{var}(\widehat\beta)$ depends both on the distribution of the observed $x$ values and on an estimate of the variance of the errors. $\endgroup$ – Michael Hardy Aug 15 '18 at 21:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.