Let $X_1,..., X_n$ be a random sample from a distribution with probability density function $f(x;\theta) = (1/2\theta) exp(-|x|/\theta)$ for $-\infty<x<\infty$.

(a) Find the maximum likelihood estimator of $\theta$ and calculate the Fisher (expected) information in the sample.

I've calculated the MLE to be $\sum |X_i|/n$ and I know the definition of Fisher expectation, but I'm getting really stuck with calculating it. I think the $|X_i|$ terms are throwing me off.

Any help in doing this problem would be much appreciated!



Recall that $$I(\theta)=-\mathbb{E}\left[\frac{\partial^2}{\partial \theta^2}l(X\,| \,\theta)\right]\,$$ under certain regularity conditions (that apply here), where $I$ is the Fisher information and $l$ is the log-likelihood function of $X$. The log-likelihood function in this case is given by $$\begin{align} l(X\,|\,\theta) &=\text{log}f(X\,|\,\theta) \\&=\log\left(\frac{1}{2\theta}\text{exp}\left(-\frac{|X|}{\theta}\right)\right) \\ &= -\frac{|X|}{\theta} - \text{log}(2\theta)\,\,. \end{align}$$ It follows that $$\frac{\partial}{\partial \theta}l(X \,|\,\theta) = \frac{|X|}{\theta^2}-\frac{1}{\theta} \implies \frac{\partial^2}{\partial \theta^2}l(X \,|\,\theta) = -\frac{2|X|}{\theta^3}+\frac{1}{\theta^2}\,.$$ So, we have $$I(\theta)=-\mathbb{E}\left[-\frac{2|X|}{\theta^3}+\frac{1}{\theta^2}\right]=\mathbb{E}\left[\frac{2|X|}{\theta^3}-\frac{1}{\theta^2}\right]=\frac{2}{\theta^3}\mathbb{E}(\,|X|\,)-\frac{1}{\theta^2}\,.$$ It remains to compute the expectation of $|X|$. To this end, I will set up the integral. By definition of expected value for transformations of continuous random variables, we have $$\mathbb{E}(\,|X|\,)=\int_{-\infty}^{\infty}|x|\,f(x \,|\, \theta)\,\text{d}x=\int_{-\infty}^{\infty}\frac{|x|}{2\theta}\text{exp}\left(-\frac{|x|}{\theta}\right)\,\text{d}x = \theta\,.$$ Note: to compute the integral, alter its form by taking advantage of the fact that $|x|$ is symmetric (and, you can also decompose the integral based on cases for $|x|$).

Thus, the Fisher information is $$I(\theta)= \frac{2}{\theta^3}\mathbb{E}(\,|X|\,)-\frac{1}{\theta^2} = \frac{2}{\theta^2}-\frac{1}{\theta^2}=\frac{1}{\theta^2}$$ For a sample $X_1,X_2,...,X_n$ of size $n$, the Fisher information is then $$I(\theta \,|\,n)=nI(\theta)=\frac{n}{\theta^2}\,.$$ Therefore, by the Cramér–Rao inequality, the variance of any unbiased estimator $\hat{\theta}$ of $\theta$ is bounded by the reciprocal of the Fisher information (this includes the MLE that you have computed, which achieves this lower bound, and is said to be an efficient estimator). In other words, $$\text{Var}(\hat{\theta}) \geq \frac{1}{nI(\theta)} = \frac{\theta^2}{n}\,\,.$$

  • $\begingroup$ but how do I differentiate $|x|$? $\endgroup$ – Brian May 19 '14 at 10:48
  • $\begingroup$ Simplify the integral using a symmetric argument, so that you won't have to...but for future reference, look up the signum function $\text{sgn}(x)$. $\endgroup$ – afedder May 19 '14 at 11:01
  • $\begingroup$ The substitution I mentioned is one way to compute the integral, but there are other ways...the symmetry of $|x|$ yields the best approach. @Brian $\endgroup$ – afedder May 19 '14 at 11:02
  • $\begingroup$ @Brian, in regards to making the integral manageable: what is $|x|$ for $x \geq 0$ and for $x < 0$? You can split the integral into these two cases and then apply the substitution and integration by parts. $\endgroup$ – afedder May 19 '14 at 11:20
  • $\begingroup$ thanks, I understand now. Looked up $sgn(x)$ and see why it works now! $\endgroup$ – Brian May 19 '14 at 11:24

To find MLE of $\theta$ we take likelihood function as $$L(x,\theta) = \frac{1}{(2\theta)^n} \exp \left( \frac{-\sum |x_i|}{\theta} \right).$$ Next we take its $\log$ as $$l(x,\theta) = - n \log 2 \theta - \frac{\sum_{i=1}^n |x_i|}{\theta}.$$ Differentiate w.r.t $\theta$ and equate to zero. $$\frac{\partial l (x,\theta)}{\partial \theta} = \frac{-n}{\theta} + \frac{\sum |x_i|}{\theta^2} = 0$$ yields $$-n \theta + \sum |x_i| = 0$$ hence MLE is $$\theta = \frac{\sum_{i=1}^n |x_i|}{n}.$$

The Fisher information is given as $$I(\theta) = -E \left[ \frac{\partial^2 l(\theta)}{\partial \theta^2} \right]$$ i.e., expected value of the second derivative of the log likelihood $l(\theta)$. $$\frac{\partial^2 l(\theta)}{\partial \theta^2} = \frac{n}{\theta^2} - \frac{ 2 \sum_{i=1}^n | x_i|}{\theta^3}$$ Taking expectation we have $$I(\theta) = - E \left[ \frac{\partial^2 l(\theta)}{\partial \theta^2} \right] = - \left[ \frac{n}{\theta^2} - \frac{2n\theta}{\theta^3} \right] = \frac{n}{\theta^2}.$$

Original images: one, two.

  • $\begingroup$ No I am not claiming E(sigma[x])=n theta. I have just edited my answer to make my point clear and that is sigma x(i)=n theta. $\endgroup$ – SA-255525 May 19 '14 at 11:30
  • $\begingroup$ @afedder I had not taken expectation of the first partial derivative of theta w.r.t. "x". I had equated it to zero. $\endgroup$ – SA-255525 May 19 '14 at 11:38
  • $\begingroup$ @ afedder Do check my answer. $\endgroup$ – SA-255525 May 19 '14 at 11:56
  • $\begingroup$ You are correct, but there is no justification of the equalities after "taking the expectation we have", and hence, it is unhelpful to the OP. $\endgroup$ – afedder May 19 '14 at 17:52
  • $\begingroup$ I have attempted to typeset your images. Please check to ensure that no errors have been introduced. $\endgroup$ – user642796 May 19 '14 at 18:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.