2
$\begingroup$

Suppose I have a Bernoulli distribution. It is discrete, so the semantics of derivation of MLE as a joint pmf is clear. For sample set $X_1, X_2,\cdots,X_m$,

$$ L(p) = P(X_1=x_1;X_2=x_2;\cdots;X_n=x_m) = \prod_{i=1}^mP(X_i=x_i) = \prod_{i=1}^mp^{x_i}(1-p)^{1-x_i} \tag{1} $$

And then we derive $L_{max}(P) = \hat{p} = \dfrac{\sum\limits_{i=1}^mx_i}{m}$. So far so good.

We actually started with finding out the joint probability mass function of all our sample data occurrences. Since each occurrence is independent, we simply multiplied individual pmf.

I am unable to take this same notion to pdf. For eg, normal distribution. Let $X_1, X_2, \cdots, X_m$ bea random sample from a normal distribution $N(\theta_1, \theta_2)$.

Then,

$$ L(\theta_1,\theta_2) = P(X_1=x_1;X_2=x_2;\cdots;X_n=x_m) = \prod_{i=1}^{m} \dfrac{1}{\sqrt{2\pi\theta_2}}{\text{exp}}{\Big[ -\dfrac{ (x_i-\theta_1)^2 }{2\theta_2} \Big]} \tag{2} $$

Question 1:
It is here I am stuck. The individual probabilities of $P(X_i=x_i)$ are $0$ for continuous pdf without a continuity correction. So how do we justify doing above step? What is the notion I am missing here?

My take: Here is my take so far but I doubt if its correct. Unlike a pmf which directly gives $P(X_i=x_i)$, a pdf only a function and always needs integration to find the probability area. That is,

If $x_1$ is a sample observation from $N(\theta_1, \theta_2)$, then of course $P(X_1=x_1)=0$, and we are not interested in that in particular (which was a wrong notion implicitly implanted while attempting joint pmf). Instead we are interested in a collective probability density function of all samples' individual probability densities.

That is, below is a continuous pdf for sample $X_1$

$$ A = f(x_1; \theta_1, \theta_2) = \dfrac{1}{\sqrt{2\pi\theta_2}}{\text{exp}}{\Big[ -\dfrac{ (x_1-\theta_1)^2 }{2\theta_2}}\Big] \tag{3} $$

But when we want to find a probability with above pdf its always in a range. For example,

$$ P(X_1 \leq a) = \int_{-\infty}^{a} f(x_1; \theta_1, \theta_2)dx_1 \tag{4} $$

Similarly, for another sample $X_2$ from same pdf,

$$ B = f(x_2; \theta_1, \theta_2) = \dfrac{1}{\sqrt{2\pi\theta_2}}{\text{exp}}{\Big[ -\dfrac{ (x_2-\theta_1)^2 }{2\theta_2}}\Big] \tag{5} $$

And for that, for an interesting range, the probability could be something like below.

$$ P(X_2 \leq b) = \int_{-\infty}^{b} f(x_2; \theta_1, \theta_2)dx_2 \tag{6} $$

Note A and B are the functions while, eq. $4$ and $6$ denote a probability calculated out of those functions. When we say, we are interested in joint pdf, we are interested in the multiplication of the functions A and B (because they are independent), and not probabilities like $4$ and $6$. The probability of any joint interested event could be calculated in resultant function AB. That is,

$$ AB = f(x_1,x_2;\theta_1,\theta_2) = \prod\limits_{i=1}^{2} \dfrac{1}{\sqrt{2\pi\theta_2}}{\text{exp}}{\Big[ -\dfrac{ (x_i-\theta_1)^2 }{2\theta_2}}\Big] \tag{7} $$

And then, in this joint pdf I could calculate interested probabilities, for example,

$$ P(X_1 \leq a; X_2 \leq b) = \int_{-\infty}^{x_1=a}\int_{-\infty}^{x_2=b} \prod\limits_{i=1}^{2} \dfrac{1}{\sqrt{2\pi\theta_2}}{\text{exp}}{\Big[ -\dfrac{ (x_i-\theta_1)^2 }{2\theta_2}}\Big] \tag{8} $$

Generalizing,

$$ P(X_1 \leq x_1; X_2 \leq x_2) = \prod\limits_{i=1}^{2} \int_{-\infty}^{x_i} \dfrac{1}{\sqrt{2\pi\theta_2}}{\text{exp}}{\Big[ -\dfrac{ (x_i-\theta_1)^2 }{2\theta_2}}\Big] \tag{9} $$

Not just left area, but any probability of interest could be calculated after this step. For example,

$$ P(X_1 \geq x_1; X_2 \geq x_2) = \prod\limits_{i=1}^{2} \int_{x_i}^{\infty} \dfrac{1}{\sqrt{2\pi\theta_2}}{\text{exp}}{\Big[ -\dfrac{ (x_i-\theta_1)^2 }{2\theta_2}}\Big] \tag{10} $$

This is why, unlike pmf, for a pdf,

$$ f(x_1,x_2;\theta_1,\theta_2) = f(x_1;\theta_1,\theta_2)f(x_2;\theta_1,\theta_2) \\ \neq P(X_1 \leq x_1; X_2 \leq x_2) \\ \neq P(X_1 \geq x_1; X_2 \geq x_2) \\ \neq P(X_1 = x_1; X_2 = x_2) $$

Question 2: Can you please confirm if this understanding is correct and why if not so, and what am I still missing?

Note:

  1. I am aware of another question which discusses similar issue, but I could not yet find convincing answer there, so I am asking again here in a way I understood the problem and to add my understanding. I also could not find any convincing answer anywhere else. :(
$\endgroup$
5
  • $\begingroup$ The likelihood function is often taken to be the joint distribution of $X_1,\ldots,X_n$ . So joint pmf and joint pdf are both valid, depending on whether the $X_i$'s have discrete or absolutely continuous distributions. $\endgroup$ – StubbornAtom Oct 18 '18 at 7:43
  • $\begingroup$ As for the pdf you mention, if it is the normal distribution you are referring to, then the pdf is not correct. $\endgroup$ – StubbornAtom Oct 18 '18 at 7:46
  • $\begingroup$ Hi, am no statistician (and have always found MLE a bit odd), but heuristically, if the the joint density is smooth enough, one could reasonably state that for a given $\epsilon$, the finite probability in an $\epsilon$-neighbourhood of the (global) maximum $\mathbf{x_0}$ say is maximised over all such neighbourhoods. At least that's how I've intuitively, hand-wavily understood it without bothering to prove the details... Perhaps someone can give you a rigorous answer! $\endgroup$ – Mehness Oct 18 '18 at 8:04
  • $\begingroup$ @StubbornAtom which pdf or step is not correct? can you please point to eq. number? $\endgroup$ – Parthiban Rajendran Oct 18 '18 at 10:42
  • $\begingroup$ @StubbornAtom I just fixed a careless mistake, are the equations ok now? $\endgroup$ – Parthiban Rajendran Oct 18 '18 at 10:44
0
$\begingroup$

It is here I am stuck. The individual probabilities of P(Xi=xi) are 0 for continuous pdf without a continuity correction. So how do we justify doing above step? What is the notion I am missing here?

You need to use probability density instead of probabilities. Definition of MLE:

The likelihood function is always defined as a function of the parameter ${\displaystyle \theta }$ equal to (or sometimes proportional to) the density of the observed data with respect to a common or reference measure, for both discrete and continuous probability distributions. So, your step is completely justified.

True that the individual probabilities are zero and to get probabilities from pdfs you need to integrate over some interval. Have a look at this answer on CV.

$L(\theta_1,\theta_2) = p(X_1=x_1;X_2=x_2;\cdots;X_m=x_m|\theta_1,\theta_2) = \prod_{i=1}^{m} \dfrac{1}{\sqrt{2\pi\theta_2}}{\text{exp}}{\Big[ -\dfrac{ (x_i-\theta_1)^2 }{2\theta_2} \Big]} \tag{1}$

where $\theta_1,\theta_2$ are mean and variance respectively.

And $p(X_1=x_1;X_2=x_2;\cdots;X_m=x_m)$ is the joint density.

And, the likelihood will be:

$L(\theta_1,\theta_2) = p(X_1=x_1;X_2=x_2;\cdots;X_m=x_m|\theta_1,\theta_2) =\dfrac{1}{\sqrt{(2\pi\theta_2)^m}}{\text{exp}}{-\frac{1}{2\theta_2}}{\sum_{i = 1}^{m}{(x_i-\theta_1)^2 }} \tag{2}$

To get the values of $\theta_1,\theta_2$, maximize the likelihood by taking derivatives w.r.t the desired variable.

Instead we are interested in a collective probability density function of all samples' individual probability densities.

I doubt this statement. You are interested in pdf of the sample from a normal population. But, you are missing that sample points are realizations of a distribution. They are not random variables but rather a realization of random variables.

From wiki:

A sample concretely represents the results of n experiments in which the same quantity is measured. For example, if we want to estimate the average height of members of a particular population, we measure the heights of n individuals. Each measurement is drawn from the probability distribution F characterizing the population, so each measured height ${\displaystyle x_{i}} $ is the realization of a random variable ${\displaystyle X_{i}}$ with distribution F. Note that a set of random variables (i.e., a set of measurable functions) must not be confused with the realizations of these variables (which are the values that these random variables take). In other words, ${\displaystyle X_{i}}$ is a function representing the measurement at the i-th experiment and ${\displaystyle x_{i}=X_{i}(\omega )}$ is the value obtained when making the measurement.

$\endgroup$
17
  • 1
    $\begingroup$ If individual probabilities are 0, how is it ok to write $L(\theta_1,\theta_2) = P(X_1=x_1;X_2=x_2;\cdots;X_m=x_m)$, shouldn't there an alternate notation for pdf? It is not making sense to say $P(X_1=x_1) = 0$ for pdf, and then write as (1) above?! $\endgroup$ – Parthiban Rajendran Oct 18 '18 at 10:39
  • $\begingroup$ Apologies for the notation. What I really meant was joint density. Have edited the answer. $\endgroup$ – naive Oct 18 '18 at 10:52
  • 1
    $\begingroup$ No apologies please, I highly appreciate people here taking time to clarify :) By the way, what I meant was, and also my understanding, that for pdf, we should not have probability notation at all $P(X_1=x_i;..)$ because it is not joint "probability", the probability is realized only after an integration later. So will it be better to stop at $f(x_1, x_2,..x_m | \theta_1,\theta_2)$ and proceed to multiplying individual pdfs? And my "take" I explained in my question was correct? $\endgroup$ – Parthiban Rajendran Oct 18 '18 at 11:16
  • $\begingroup$ I agree with you on the notation part. That is why I have use $'p'$ instead of$'P'$ to denote joint density. And this is a practice followed widely. Yes, sure the multiplication is of individual densities only. $\endgroup$ – naive Oct 18 '18 at 11:43
  • $\begingroup$ oh ok, I did not know that, As per wiki, $f(x)$ or $f_X(x)$ is used. $\endgroup$ – Parthiban Rajendran Oct 18 '18 at 12:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.