Suppose that $a(\cdot)$ and $b(\cdot)$ are two non-negative functions such that $$f(x;\theta)=a(\theta)b(x)$$ is a probability density function for each $\theta > 0$. Find the maximum likelihood estimator of $\theta$.

My try: Our likelihood function is given by $$L(\theta) = \prod_{i=1}^n a(\theta)b(x_i) = a(\theta)^nb(x)^n$$ The log likelihood function is given by $$\ln L(\theta) = n\ln a(\theta) + n \ln b(x)$$ Equating it to zero we get $$\ln a (\theta) = - \ln b(x)$$ which obviously leads to nowhere.

Moreover, the question itself seems weird to me. I am used to the form of "Given a random sample $X_1,...,X_n$ of size $n$ (...)", since this is missing now, does this imply that I cannot use the usual method I demonstrated above?

Lastly, if you want to, could you check the exercise below for errors?

Let $X_1,...,X_n$ denote a random sample from $$f(x;\theta) = f_\theta (x) = \theta f_1(x) + (1-\theta)f_0 (x)$$ where $0 \leq \theta \leq 1$ and $f_0(\cdot)$ and $f_1(\cdot)$ are known densities, estimate $\theta$ by the method of moments.

Answer: First, we need to write $E[X]$ in a better form: \begin{align*} E[x] &= \int_{-\infty}^\infty x\cdot f(x;\theta)dx = \int_{-\infty}^\infty x(\theta f_1(x) + (1-\theta)f_0 (x))dx \\ &= \theta \int_{-\infty}^\infty x\cdot f_1 (x)dx + (1-\theta)\int_{-\infty}^\infty x\cdot f_0 (x)dx \\ &= \theta \int x(f_1-f_0)dx + \int x f_0 dx \\ &=\theta \left(E_1\left[x\right] - E_0\left[x\right]\right) + E_0\left[x\right] \end{align*} Equating this to the first sample moment ($m_1'$) we get: \begin{align*} m_1'= \theta \left(E_1\left[x\right] - E_0\left[x\right]\right) + E_0\left[x\right] \end{align*} which is equivalent to \begin{align*} \theta = \dfrac{m_1' - E_0[x]}{E_1[x] - E_0[x]} \end{align*} Hence, our method of moments estimator for $\theta$ is given by: $$\hat{\theta} = \dfrac{m_1' - E_0[x]}{E_1[x] - E_0[x]}$$

Both are questions from "Introduction to the theory of statistics" by Mood, Graybill and Boes.

Thanks in advance!

  • 1
    $\begingroup$ $\prod_{i=1}^n b(x_i)=b(x)^n$ doesn't make sense. $\endgroup$ – Stefan Hansen Jun 17 '14 at 14:35
  • $\begingroup$ @StefanHansen When looking back, you're obviously right. What would you suggest how I should continue? $\endgroup$ – Nigel Overmars Jun 17 '14 at 14:39
  • 1
    $\begingroup$ Continue by keeping $\prod_{i=1}^n b(x_i)$ as it is. Moreover, you want to solve $l'(\theta)=0$, not $l(\theta)=0$, where $l$ is the log-likelihood. $\endgroup$ – Stefan Hansen Jun 17 '14 at 14:49
  • $\begingroup$ @StefanHansen Yes, I also spotted that mistake. It has been some time since I had to do this. Continuing on your advice: $\ln L(\theta) = n \ln a (\theta) + \sum_i \ln b(x_i)$. Differentiate wrt to $\theta$ and set equal to zero to get: $\dfrac{n\cdot a'(\theta)}{a(\theta)} = 0$ since $\sum_i \ln b(x_i)$ is just a constant. Which gives $a'(\theta)=0$. Do you think this is the answer? $\endgroup$ – Nigel Overmars Jun 17 '14 at 14:54
  • $\begingroup$ This assumes that $a$ is differentiable. Are you sure that you haven't incorrectly stated the problem? Exercise 39 in the book you mention has $f(x;\theta)=a(\theta)b(x)I_{(0,\theta)}(x)$. $\endgroup$ – Stefan Hansen Jun 17 '14 at 15:13

Another example where including the (crucial) indicator functions in the densities simplifies everything... Here the PDF is $$ f(x;\theta)=a(\theta)b(x)\mathbf 1_{[0,\theta]}(x), $$ where $$ \frac1{a(\theta)}=\int_0^\theta b(x)\,\mathrm dx, $$ hence the likelihood of a sample $\mathbf x=(x_k)$ is $$ L(\mathbf x,\theta)=\prod_kf(x_k;\theta)=a(\theta)^n\,\mathbf 1_{\theta\geqslant m(\mathbf x)}\,\prod_kb(x_k), $$ where $$ m(\mathbf x)=\max_kx_k. $$ The last product does not depend on $\theta$ hence one can forget it. The indicator function shows that $L(\mathbf x,\theta)$ can be nonzero only when $\theta\geqslant m(\mathbf x)$. And $\theta\mapsto a(\theta)$ is nonincreasing hence one looks for $\theta$ as small as possible. Finally, $L(\mathbf x,\theta)$ is maximal when $\theta=\hat\theta(\mathbf x)$ with $$ \hat\theta(\mathbf x)=m(\mathbf x). $$

  • $\begingroup$ Could you explain how you got the $1/a(\theta) = \int_0^\theta b(x) dx$ equality? Or (I think equivalently), why including the indicator function is necessary $\endgroup$ – Nigel Overmars Jun 17 '14 at 17:38
  • $\begingroup$ Because for every $\theta$, the function $f(\ ;\theta)$ must integrate to $1$ and $$\int_\mathbb Rf(x;\theta)\mathrm dx=a(\theta)\int_0^\theta b(x)\mathrm dx.$$ $\endgroup$ – Did Jun 17 '14 at 18:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.