$f(x, \theta)= \frac{\theta}{x^2}$ with $x\geq\theta$ and $\theta>0$, find the MLE

Let $X$ be a random variable with density $$f(x, \theta)= \frac{\theta}{x^2}$$ with $x\geq\theta$ and $\theta>0$.

a) Show if $S=\min\{x_1,\cdots, x_n\}$ is a sufficient statistics and if it is minimal.

b) Find the Maximum Likelihood Estimator of $\theta$ and tell if it is unbiased.

c) Find the distribution of $S$ and tell if there is an unbiased estimator of the form $cS$ for some $c$.

attempt: There are several problems. $S$ doens't look like a sufficient statistics cause $L(\theta|x_1, \cdots, x_n)= \frac{\theta^n}{(x_1\cdots x_n)^2}$ doesn't seem to be known if we know the minimum. Morover there is no maximum for that function so I can't find MLE.

Thanks!

• – Saad Feb 21 '18 at 0:56
• See Wikipedia on 'Pareto distribution': Their $\alpha = 1,$ their $x_m = \theta.$ – BruceET Feb 21 '18 at 4:12
• In @AlexFrancisco's link, the task is to find an MLE for $\alpha$ (here set to $\alpha=1)$ with your $\theta$ (minimum possible value) specified. So I'm not sure how much help that link is going to be. // In the Wikipedia article, both parameters are estimated. Ignore estimation of $\alpha.$ – BruceET Feb 21 '18 at 5:17

I have mentioned this elsewhere, but it bears repeating because it is such an important concept:

Sufficiency pertains to data reduction, not parameter estimation per se. Sufficiency only requires that one does not "lose information" about the parameter(s) that was present in the original sample.

Students of mathematical statistics have a tendency to conflate sufficient statistics with estimators, because "good" estimators in general need to be sufficient statistics: after all, if an estimator discards information about the parameter(s) it estimates, it should not perform as well as an estimator that does not do so. So the concept of sufficiency is one way in which we characterize estimators, but that clearly does not mean that sufficiency is about estimation. It is vitally important to understand and remember this.

That said, the Factorization theorem is easily applied to solve (a); e.g., for a sample $\boldsymbol x = (x_1, \ldots, x_n)$, the joint density is $$f(\boldsymbol x \mid \theta) = \prod_{i=1}^n \frac{\theta}{x_i^2} \mathbb 1 (x_i \ge \theta) \mathbb 1 (\theta > 0) = \mathbb 1 (x_{(1)} \ge \theta > 0) \, \theta^n \prod_{i=1}^n x_i^{-2},$$ where $x_{(1)} = \min_i x_i$ is the minimum order statistic. This is because the product of the indicator functions $\mathbb 1 (x_i \ge \theta)$ is $1$ if and only if all of the $x_i$ are at least as large as $\theta$, which occurs if and only if the smallest observation in the sample, $x_{(1)}$, is at least $\theta$. We see that we cannot separate $x_{(1)}$ from $\theta$, so this factor must be part of $g(\boldsymbol T(\boldsymbol x) \mid \theta)$, where $\boldsymbol T(\boldsymbol x) = T(\boldsymbol x) = x_{(1)}$. Note that in this case, our sufficient statistic is a function of the sample that reduces a vector of dimension $n$ to a scalar $x_{(1)}$, so we may write $T$ instead of $\boldsymbol T$. The rest is easy: $$f(\boldsymbol x \mid \theta) = h(\boldsymbol x) g(T(\boldsymbol x) \mid \theta),$$ where $$h(\boldsymbol x) = \prod_{i=1}^n x_i^{-2}, \quad g(T \mid \theta) = \mathbb 1 (T \ge \theta > 0) \theta^n,$$ and $T$, defined as above, is our sufficient statistic.

You may think that $T$ estimates $\theta$--and in this case, it happens to--but just because we found a sufficient statistic via the Factorization theorem, this doesn't mean it estimates anything. This is because any one-to-one function of a sufficient statistic is also sufficient (you can simply invert the mapping). $T^2 = x_{(1)}^2$ is also sufficient (note while $m : \mathbb R \to \mathbb R$, $m(x) = x^2$ is not one-to-one in general, in this case it is because the support of $X$ is $X \ge \theta > 0$).

Regarding (b), MLE estimation, we express the joint likelihood as proportional to $$\mathcal L(\theta \mid \boldsymbol x) \propto \theta^n \mathbb 1(0 < \theta \le x_{(1)}).$$ We simply discard any factors of the joint density that are constant with respect to $\theta$. Since this likelihood is nonzero if and only if $\theta$ is positive but not exceeding the smallest observation in the sample, we seek to maximize $\theta^n$ subject to this constraint. Since $n > 0$, $\theta^n$ is a monotonically increasing function on $\theta > 0$, hence $\mathcal L$ is greatest when $\theta = x_{(1)}$; i.e., $$\hat \theta = x_{(1)}$$ is the MLE. It is trivially biased because the random variable $X_{(1)}$ is never smaller than $\theta$ and is almost surely strictly greater than $\theta$; hence its expectation is almost surely greater than $\theta$.

Finally, we can explicitly compute the density of the order statistic as requested in (c): $$\Pr[X_{(1)} > x] = \prod_{i=1}^n \Pr[X_i > x],$$ because the least observation is greater than $x$ if and only if all of the observations are greater than $x$, and the observations are IID. Then $$1 - F_{X_{(1)}}(x) = \left(1 - F_X(x)\right)^n,$$ and the rest of the computation is left to you as a straightforward exercise. We can then take this and compute the expectation $\operatorname{E}[X_{(1)}]$ to ascertain the precise amount of bias of the MLE, which is necessary to answer whether there is a scalar value $c$ (which may depend on the sample size $n$ but not on $\theta$ or the sample $\boldsymbol x$) such that $c\hat \theta$ is unbiased.

• Thank you very much for the perfect explaination! – Maffred Feb 21 '18 at 6:07
• Nicely done! (+1). – BruceET Feb 21 '18 at 6:35
• I was wondering about minimality of our sufficient statistic. Why is it minimal? I thought it was not, due to the minimality condition of $S(x)=S(y) \iff \frac{L(x)}{L(y)}$ doesn't depend on $\theta$. – Maffred Feb 21 '18 at 7:09
1. Use the indicator function in the factorization criteria, as $$I\{\cap_{i=1}^n \{X_i \ge \theta\}\} = I\{X_{(1)}\ge \theta\} \prod_{i=2}^nI\{ X_i\ge X_{(1)}\} \, .$$
2. $f(x;\theta)$ is monotonic decreasing function so the solution is on the boundary of $\Theta$, thus $X_{(1)} = \hat{\theta}_n$.

3. Use the fact that $$F_S(s) = 1 - (1 - F_X(s))^n = 1 - \left(1 - \int_{\theta}^{s}\frac{\theta}{x^2} dx \right)^n,$$ and $$f_S(s)=F'_S(s).$$

• For 1. $L$ becomes $L(\theta|x_1, \cdots, x_n)= \frac{\theta^n}{(x_1\cdots x_n)^2}\{X_{(1)}\ge \theta\} \prod_{i=2}^nI\{ X_i\ge X_{(1)}\}$. $\theta^n$ isn't a problem for the factorization criteria? – Maffred Feb 21 '18 at 1:45
• No. $h(X) = (\prod X_i)^{-2}\prod_{i=2}I\{X_i \ge X_{(1)}\}$ and $g(\theta; X_{(1)})=\theta^n I\{X_{(1)} \ge \theta\}$ – V. Vancak Feb 21 '18 at 1:47
• Oh ok! Btw in $I\{\cap_{i=1}^n \{X_i \ge \theta\}\} = I\{X_{(1)}\ge \theta\} \prod_{i=2}^nI\{ X_i\ge X_{(1)}\} \, .$ asking for $I\{X_{(1)}\ge \theta\}$ doesn't already provide the other part? Why we need to insert the product? – Maffred Feb 21 '18 at 1:53
• We don't as $\prod I_i =1$ – V. Vancak Feb 21 '18 at 1:59
• It is necessary minimal as $\dim(X_{(1)}) =1 = \dim(\theta)$. – V. Vancak Feb 21 '18 at 2:06

Comment. This is for intuition only. It seems your conversation with @V.Vancak (+1) has taken care of (a).

Hints for the rest: Using the 'quantile' (inverse CDF) method, an observation $X$ from your Pareto distribution can be simulated as $X = \theta/U,$ where $U$ is standard uniform.

In the simulation let $\theta = 2$ and $n = 10.$ Sample a million sample minimums $S.$ Then the average of the $S$'s approximates $E(S).$ Clearly the minimum is a biased estimator of $\theta.$ In this example $E(S) \approx 2.222 \pm 0.004.$ This makes intuitive sense because the minimum must always be at least a little larger than $\theta.$

m = 10^5; th = 2;  n = 10
s = replicate(m, min(th/runif(n)))
mean(s)
## 2.221677      # aprx E(S) I will leave the mathematical derivation of $E(S)$ and looking for an 'unbiasing' constant (that may depend on $n$) to you.

Addendum: (One more hint per Comments.) According to Wikipedia the CDF of $X$ is $F_X(x) = 1 - \theta/x,$ for $x > \theta.$ [This is the CDF I inverted in order to simulate, noting that $U = 1 - U^\prime$ is standard uniform if $U^\prime$ is.]

Thus for $n \ge 2,$ $$1 - F_S(s) = P(X > s) = P(X_1 > s, \dots, X_n > x) = P(X_1 > s) \cdots P(X_n > s) = (\theta/s)^n.$$ So $F_S(s) = P(S \le s) = 1 - (\theta/s)^n,$ for $s > \theta.$ From there you should be able able to find $f_S(s),\,$ $E(S),$ and the unbiasing constant.

• Thanks! I thought your same reasoning of the unbiaseness but I am not able to compute it. I tried the integral of $xs(s)$ route but am not able to solve it! – Maffred Feb 21 '18 at 5:19
• Do you know how to find the dist'n of the minimum? Hope you're not trying to find $E(X) = \infty.$. My guess is that unbiasing const is something like $c(n) = \frac{n-1}{n}.$ – BruceET Feb 21 '18 at 5:50
• No i'm trying with the distribution of the minimum! I also think to some similar coefficient like uniform distribution $(0,1)$ – Maffred Feb 21 '18 at 5:54
• Had no problem (even at this time of night after too much celebrating over dinner) getting $E(S)$ from my CDF, just added to Answer. (Check for typos.) Bedtime. Good luck with this! – BruceET Feb 21 '18 at 6:24
• I find $E(S) = \int_{\theta}^{\infty} s\frac{n\theta^n}{s^{n+1}}ds = \frac{n}{n-1}\theta$ as desired! Thanks! ^^ – Maffred Feb 21 '18 at 7:04