Unbiased Estimator for a Uniform Variable Support

Let $x_i$ be iid observations in a sample from a uniform distribution over $\left[ 0, \theta \right]$. Now I need to estimate $\theta$ based on $N$ observations and I want the estimator to be unbiased.

I thought about simple estimator $\hat{\theta} = \max \left( x_i \right)$.

Based on simulation it is not biased, yet I couldn't show it analytically.

Could anyone, please, show it is unbiased?

BTW, I could easily find another, easy to prove, unbiased estimator, $\hat{\theta} = 2 \mathrm{mean} \left( {x}_{i} \right)$

• I have corrected the terminology in the question, replacing the word "samples" with "observations in a sample". Aug 30, 2011 at 19:19
• math.stackexchange.com/q/313390/321264 Aug 10, 2020 at 15:30

Of course $\hat\theta=\max\{x_i\}$ is biased, simulations or not, since $\hat\theta<\theta$ with full probability hence one can be sure that, for every $\theta$, $E_\theta(\hat\theta)<\theta$.

One usually rather considers $\hat\theta_N=\frac{N+1}N\cdot\max\{x_i\}$, then $E_\theta(\hat\theta_N)=\theta$ for every $\theta$.

To show that $\hat\theta_N$ is unbiased, one must compute the expectation of $M_N=\max\{x_i\}$, and the simplest way to do that might be to note that for every $x$ in $(0,\theta)$, $P_\theta(M_N\le x)=(x/\theta)^N$, hence $$E_\theta(M_N)=\int\limits_0^{\theta}P_\theta(M_N\ge x)\text{d}x=\int\limits_0^{\theta}(1-(x/\theta)^N)\text{d}x=\theta-\theta/(N+1)=\theta N/(N+1).$$

Edit (Below is a remark due to @cardinal, which completes nicely this post.)

It may be worth noting that the maximum $M_N=\max\limits_{1\le k\le N}X_k$ of an i.i.d. Uniform$(0,θ)$ random sample is a sufficient statistic for $θ$ and that it is one of the few statistics for distributions outside the exponential family which is also complete.

An immediate consequence is that $\hat\theta_N=(N+1)M_N/N$ is the uniformly minimum variance unbiased estimator (UMVUE) for $θ$, that is, that any other unbiased estimator for $θ$ is a worse estimator in the $L^2$ sense.

• Thank for the answer. What if ${x}_{i}$ are distributed uniformly on $[-5, \theta]$ so you couldn't use the expectation formula you used. How would you calculate it then? Meaning, could you solve it by using a pdf instead of the probability function?
– Royi
Aug 29, 2011 at 13:59
• If I wanted to stick to the PDF of M, I could write E(M+5) as the integral of P(M>x) from x=-5 to x=+infty (or to x=theta, this gives the same result) and P(M<x)=((x+5)/(theta+5))^N. This would get me right away that E(M+5) is (theta+5)N/(N+1) hence that an unbiased estimator of theta is ((N+1)M+5)/N.
– Did
Aug 29, 2011 at 17:02
• I've moved it. () Aug 30, 2011 at 19:20
• (+1) As you're aware (obviously!), there are pretty strong theoretical reasons for using the precise estimator $\hat{\theta}$ that you consider as opposed to others. Maybe that's worth a mention. See, also, my comment to @Michael regarding this point. Aug 30, 2011 at 19:30
• Can someone please explain the $P_θ(M_N \leq x)=(x/θ)^N$ part.
– Kuai
Feb 24, 2014 at 9:05

Another way to look at the result derived by Joriki:

It is known that if you order $N$ uniform observations in a sample from a given interval, then the resulting partition of the interval gives a point sampled uniformly on the $(N+1)$-simplex.

In particular, the remaining difference $\theta-\hat\theta$ has the same distribution of any component of such a random point. In particular, it is clear that it has expectancy $\theta/(N+1)$ (since by symmetry, all components have the same expectation).

All in all: $$\mathbb{E}[\theta-\hat\theta] = \frac{\theta}{N+1},$$ and indeed

$$\mathbb{E}[\hat\theta] = \frac{N}{N+1}\theta.$$

Edit: If the minimum of the interval is unknown as well (say $[a,b]$), then by calling $\hat a$ the minimum of the sample and $\hat b$ the maximum, the same reasoning shows that $$\mathbb{E}[b-\hat b]=\frac{b-a}{N+1},$$ $$\mathbb{E}[\hat b]=\frac{N}{N+1}b+\frac{1}{N+1}a.$$

And so to have an unbiased estimator of the maximum $b$, one could use for example $$\frac{N\hat b-\hat a}{N-1}.$$

• Nice :-) ${}{}$ Aug 29, 2011 at 12:19
• Could you please try to explain it more intuitively? I couldn't understand how you derived the expectancy of the difference. Thank You!
– Royi
Aug 29, 2011 at 14:02
• @Drazick: If you sample $N$ points from the interval $[0,\theta]$ and sort them, the successive differences will all have the same distribution. So if you rename your samples according to their order $X_1<X_2<...X_N$, then $\mathbb{E}[X_1-0] = \mathbb{E}[X_2-X_1]=...=\mathbb{E}[\theta-X_N]=\theta/(N+1)$. Aug 29, 2011 at 15:44
• This posting incorrectly uses the word "sample" in the same way in which the original question did. I wonder if it will ever be possible to talk mathematicians out of that one. What are called samples here should be called observations in a sample. Aug 30, 2011 at 13:02
• @FelixCQ - that is really neat. How do you prove that successive differences have the same distribution? Aug 30, 2011 at 16:51

To derive Didier's result, observe that the cumulative distribution function for the maximum $m$ is given by the ratio of the volume of $[0,m]^N$ to the volume of $[0,\theta]^N$, which is $(m/\theta)^N$. So the density is the derivative of that, and we can calculate the expectation value of $m$ as

$$\begin{eqnarray} \int_0^\theta m\frac{\mathrm d}{\mathrm dm}\left(\frac m\theta\right)^N\mathrm dm &=& \frac N{\theta^N}\int_0^\theta m^N\mathrm dm \\ &=& \frac N{N+1}\theta\;, \end{eqnarray}$$

so we get an unbiased estimator for $\theta$ by multiplying by $(N+1)/N$.

• Could you explain the intuition behind you first statement: "To derive Didier's result, observe that the cumulative distribution function for the maximum m is given by the ratio of the volume of [0,m]N to the volume of [0,θ]N, which is (m/θ)N.". Thank You.
– Royi
Aug 29, 2011 at 13:48
• @Drazick: The value of the cumulative distribution function for the maximum at $m$ is the probability that the maximum is less than or equal to $m$. That is the case if and only if all $N$ samples are in $[0,m]$, and the probability for that is $(m/\theta)^N$, the fraction of the volume of the total space $[0,\theta]^N$ occupied by the event $[0,m]^N$. I hope that made it clearer? Aug 29, 2011 at 13:55
• Yes, indeed. Thank you.
– Royi
Aug 29, 2011 at 15:07

It may be worth noticing a couple of things about the other unbiased estimator mentioned in the question: $2\times\text{the sample mean}$.

(1) If one finds the conditional expected value of $2\times\text{the sample mean}$ given the sample maximum, one gets $(N+1)/N$ times the sample maximum.

(2) In some cases, $2\times\text{the sample mean}$ is actually smaller than the sample maximum. Thus although it is on average the right amount, there are instances where the data themselves tell you that it's nowhere near the right amount.

• It may be worth noting that the maximum $M_n = \max_k X_k$ of an iid $\mathcal{U}(0,\theta)$ random sample is a sufficient statistic for $\theta$ and is one of the few statistics for distributions outside the exponential family that can also be shown to be complete. An immediate consequence of this is that $(N+1) M_n / N$ is the uniformly minimum variance unbiased estimator for $\theta$, i.e., any other unbiased estimator for $\theta$ is a "worse" estimator. Aug 30, 2011 at 19:26
• 1. Could you show the "Sufficient Statistics" property? 2. Could you show the Conditional Expected Value you mentioned in (1)? Thanks.
– Royi
Sep 9, 2011 at 16:56
• @Drazick, The sufficiency follows directly from the Factorization Theorem since the joint density is $\theta^{-n} \prod_{i=1}^n 1_{(0 \leq X_i \leq \theta)} = \theta^{-n} 1_{(0 < X_{(n)})} 1_{(0 < X_{(1)} < X_{(n)})} = g(\theta,X_{(n)}) h(X_{(1)},X_{(n)})$. Sep 11, 2011 at 16:37
• Also, (1) in the answer is simply an example of the effects of the Rao-Blackwell theorem, which is intimately tied to sufficiency. Sep 11, 2011 at 16:43