Take the 2-minute tour ×
Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. It's 100% free, no registration required.

Let $ x_i $ be iid observations in a sample from a uniform distribution over $ \left[ 0, \theta \right] $. Now I need to estimate $ \theta $ based on $N$ observations and I want the estimator to be unbiased.

I thought about simple estimator $ \hat{\theta} = \max \left( x_i \right) $.

Based on simulation it is not biased, yet I couldn't show it analytically.

Could anyone, please, show it is unbiased?

BTW, I could easily find another, easy to prove, unbiased estimator, $ \hat{\theta} = 2 \mathrm{mean} \left( {x}_{i} \right) $

share|improve this question
    
I have corrected the terminology in the question, replacing the word "samples" with "observations in a sample". –  Michael Hardy Aug 30 '11 at 19:19

4 Answers 4

up vote 12 down vote accepted

Of course $\hat\theta=\max\{x_i\}$ is biased, simulations or not, since $\hat\theta<\theta$ with full probability hence one can be sure that, for every $\theta$, $E_\theta(\hat\theta)<\theta$.

One usually rather considers $\hat\theta_N=\dfrac{N+1}N\max\{x_i\}$, then $E_\theta(\hat\theta_N)=\theta$ for every $\theta$.

To show that $\hat\theta_N$ is unbiased, one must compute the expectation of $M_N=\max\{x_i\}$, and the simplest way to do that might be to note that for every $x$ in $(0,\theta)$, $P_\theta(M_N\le x)=(x/\theta)^N$, hence $$ E_\theta(M_N)=\int\limits_0^{\theta}P_\theta(M_N\ge x)\text{d}x=\int\limits_0^{\theta}(1-(x/\theta)^N)\text{d}x=\theta-\theta/(N+1)=\theta N/(N+1). $$

Edit (Below is a remark due to @cardinal, which completes nicely this post.)

It may be worth noting that the maximum $M_N=\max\limits_{1\le k\le N}X_k$ of an i.i.d. Uniform$(0,θ)$ random sample is a sufficient statistic for $θ$ and that it is one of the few statistics for distributions outside the exponential family which is also complete.

An immediate consequence is that $\hat\theta_N=(N+1)M_N/N$ is the uniformly minimum variance unbiased estimator for $θ$, that is, that any other unbiased estimator for $θ$ is a worse estimator in the $L^2$ sense.

share|improve this answer
    
Thank for the answer. What if $ {x}_{i} $ are distributed uniformly on $ [-5, \theta] $ so you couldn't use the expectation formula you used. How would you calculate it then? Meaning, could you solve it by using a pdf instead of the probability function? –  Drazick Aug 29 '11 at 13:59
    
If I wanted to stick to the PDF of M, I could write E(M+5) as the integral of P(M>x) from x=-5 to x=+infty (or to x=theta, this gives the same result) and P(M<x)=((x+5)/(theta+5))^N. This would get me right away that E(M+5) is (theta+5)N/(N+1) hence that an unbiased estimator of theta is ((N+1)M+5)/N. –  Did Aug 29 '11 at 17:02
    
I've moved it. () –  Michael Hardy Aug 30 '11 at 19:20
    
(+1) As you're aware (obviously!), there are pretty strong theoretical reasons for using the precise estimator $\hat{\theta}$ that you consider as opposed to others. Maybe that's worth a mention. See, also, my comment to @Michael regarding this point. –  cardinal Aug 30 '11 at 19:30
    
@cardinal, yes. Here is a suggestion: I copy your comment to Michael and paste it at the end on my post, with due mention of authorship naturally. What do you think about it? –  Did Aug 30 '11 at 21:23

To derive Didier's result, observe that the cumulative distribution function for the maximum $m$ is given by the ratio of the volume of $[0,m]^N$ to the volume of $[0,\theta]^N$, which is $(m/\theta)^N$. So the density is the derivative of that, and we can calculate the expectation value of $m$ as

$$ \begin{eqnarray} \int_0^\theta m\frac{\mathrm d}{\mathrm dm}\left(\frac m\theta\right)^N\mathrm dm &=& \frac N{\theta^N}\int_0^\theta m^N\mathrm dm \\ &=& \frac N{N+1}\theta\;, \end{eqnarray} $$

so we get an unbiased estimator for $\theta$ by multiplying by $(N+1)/N$.

share|improve this answer
    
Could you explain the intuition behind you first statement: "To derive Didier's result, observe that the cumulative distribution function for the maximum m is given by the ratio of the volume of [0,m]N to the volume of [0,θ]N, which is (m/θ)N.". Thank You. –  Drazick Aug 29 '11 at 13:48
    
@Drazick: The value of the cumulative distribution function for the maximum at $m$ is the probability that the maximum is less than or equal to $m$. That is the case if and only if all $N$ samples are in $[0,m]$, and the probability for that is $(m/\theta)^N$, the fraction of the volume of the total space $[0,\theta]^N$ occupied by the event $[0,m]^N$. I hope that made it clearer? –  joriki Aug 29 '11 at 13:55
    
Yes, indeed. Thank you. –  Drazick Aug 29 '11 at 15:07

Another way to look at the result derived by Joriki:

It is known that if you order $N$ uniform observations in a sample from a given interval, then the resulting partition of the interval gives a point sampled uniformly on the $(N+1)$-simplex.

In particular, the remaining difference $\theta-\hat\theta$ has the same distribution of any component of such a random point. In particular, it is clear that it has expectancy $\theta/(N+1)$ (since by symmetry, all components have the same expectation).

All in all: $$\mathbb{E}[\theta-\hat\theta] = \frac{\theta}{N+1},$$ and indeed

$$\mathbb{E}[\hat\theta] = \frac{N}{N+1}\theta.$$

Edit: If the minimum of the interval is unknown as well (say $[a,b]$), then by calling $\hat a$ the minimum of the sample and $\hat b$ the maximum, the same reasoning shows that $$\mathbb{E}[b-\hat b]=\frac{b-a}{N+1},$$ $$\mathbb{E}[\hat b]=\frac{N}{N+1}b+\frac{1}{N+1}a.$$

And so to have an unbiased estimator of the maximum $b$, one could use for example $$\frac{N\hat b-\hat a}{N-1}.$$

share|improve this answer
    
Nice :-) ${}{}$ –  joriki Aug 29 '11 at 12:19
    
Could you please try to explain it more intuitively? I couldn't understand how you derived the expectancy of the difference. Thank You! –  Drazick Aug 29 '11 at 14:02
    
@Drazick: If you sample $N$ points from the interval $[0,\theta]$ and sort them, the successive differences will all have the same distribution. So if you rename your samples according to their order $X_1<X_2<...X_N$, then $\mathbb{E}[X_1-0] = \mathbb{E}[X_2-X_1]=...=\mathbb{E}[\theta-X_N]=\theta/(N+1)$. –  FelixCQ Aug 29 '11 at 15:44
    
This posting incorrectly uses the word "sample" in the same way in which the original question did. I wonder if it will ever be possible to talk mathematicians out of that one. What are called samples here should be called observations in a sample. –  Michael Hardy Aug 30 '11 at 13:02
    
@FelixCQ - that is really neat. How do you prove that successive differences have the same distribution? –  robinson Aug 30 '11 at 16:51

It may be worth noticing a couple of things about the other unbiased estimator mentioned in the question: $2\times\text{the sample mean}$.

(1) If one finds the conditional expected value of $2\times\text{the sample mean}$ given the sample maximum, one gets $(N+1)/N$ times the sample maximum.

(2) In some cases, $2\times\text{the sample mean}$ is actually smaller than the sample maximum. Thus although it is on average the right amount, there are instances where the data themselves tell you that it's nowhere near the right amount.

share|improve this answer
4  
It may be worth noting that the maximum $M_n = \max_k X_k$ of an iid $\mathcal{U}(0,\theta)$ random sample is a sufficient statistic for $\theta$ and is one of the few statistics for distributions outside the exponential family that can also be shown to be complete. An immediate consequence of this is that $(N+1) M_n / N$ is the uniformly minimum variance unbiased estimator for $\theta$, i.e., any other unbiased estimator for $\theta$ is a "worse" estimator. –  cardinal Aug 30 '11 at 19:26
    
1. Could you show the "Sufficient Statistics" property? 2. Could you show the Conditional Expected Value you mentioned in (1)? Thanks. –  Drazick Sep 9 '11 at 16:56
1  
@Drazick, The sufficiency follows directly from the Factorization Theorem since the joint density is $\theta^{-n} \prod_{i=1}^n 1_{(0 \leq X_i \leq \theta)} = \theta^{-n} 1_{(0 < X_{(n)})} 1_{(0 < X_{(1)} < X_{(n)})} = g(\theta,X_{(n)}) h(X_{(1)},X_{(n)})$. –  cardinal Sep 11 '11 at 16:37
1  
Also, (1) in the answer is simply an example of the effects of the Rao-Blackwell theorem, which is intimately tied to sufficiency. –  cardinal Sep 11 '11 at 16:43

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.