Take the 2-minute tour ×
Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. It's 100% free, no registration required.

I am struggling with the semantics of continuous random variables.

For example, we do maximum likelihood estimation, in which we try to find the parameter $\theta$ which, for some observed data $D$, maximizes the likelihood $P(\theta|D)$.

But my understanding of this is $$P(\theta = x) = P(x\leq\theta\leq x) = \int_x^xp(t)dt = 0$$ so I am not sure how any $\theta$ can result in a non-zero probability.

Intuitively I understand what it means to find the "most probable" $\theta$, but I am having trouble uniting it with the formal definition.


EDIT: In my class we defined $L(\theta:D)=P(D|\theta)=\prod_i P(D_i|\theta)$ (assuming i.i.d, where $D_i$ are the observations). Then we want to find $\text{argmax}_\theta \prod_i P(D_i|\theta)$.

I was incorrect above about finding $P(\theta)$, but it seems to me we're still trying to find the maximal probability, where all probabilities are zero. Some answerers suggested that we're actually trying to find the max probability density but I don't understand why this is true.

share|improve this question
1  
In the frequentist theory of statistics where maximum likelihood estimators (MLE) are defined, $\theta$ is an unknown parameter, not a random variable or event, and so $\theta$ cannot have probability associated with it. –  Dilip Sarwate Jun 5 '12 at 2:03
1  
"maximum likelihood" is not "maximum probability" –  leonbloy Jun 5 '12 at 2:29

5 Answers 5

up vote 3 down vote accepted

It seems to me like whoever defined it for you was being hand-wavy (and, I would argue, careless). For continuous random variables, the likelihood is defined to be the joint density of the data $\mathcal D$ when taken as a function of the unknown parameter $\theta$, i.e. for a vector-valued observation $\mathbf x$ we have $$L(\theta | \mathbf x) := f(\mathbf x | \theta) $$ where $f$ is the density of a random vector $\mathbf X$. For discrete random vectors, replace take $f$ to be a probability mass function. This is the definition of likelihood that you should take. Then, to get the MLE, take an $\arg \max$ over $\theta$. To get a definition that is less arbitrary in the distinction between continuous and discrete rvs, one needs to introduce measure theory and the notion of Radon-Nikodym derivative, in which case we can generalize the notion of density so that mass functions are a type of density, and the arbitrariness vanishes.

share|improve this answer
    
This looks different than the definition on p. 860 here. Can you take a look and see if I've transcribed it wrong? –  Xodarap Jun 12 '12 at 23:36
1  
@Xodarap I would just ignore that. I'm familiar with how Koller presents things (I actually suspected at the outset that this is what you were studying), and the best thing to do is probably to use a different reference for a few of the statistical concepts presented e.g. likelihood and especially sufficiency. –  guy Jun 13 '12 at 0:58

When using the maximum likelihood estimation principle, the parameter that you are trying to estimate is not a random variable. That is, there is some true parameter $\theta^*$, which is a fixed (non-random), but unknown quantity.

The maximum likelihood estimator is formed as $\hat \theta = \arg\min_\theta \prod_{i=1}^n f(x_i|\theta)$, which is a random variable as it depends on ${\cal D} = \{x_i\}_{i=1}^n$.

Now, as you pointed out $P(\hat \theta = \theta^*) = 0$. However, there are a number of ways in which you can evaluate goodness of your estimator. For example, under some conditions, you could show that $E(\hat \theta - \theta^*)^2$ is small, or that $\hat \theta \xrightarrow{a.s.} \theta^*$ as $n \rightarrow \infty$.

share|improve this answer

"The most probable $\theta$" is a misleading way of saying it, although it is very frequently encountered. The MLE is actually the value of $\theta$ that makes the observed data more probable than it would be with any other value of $\theta$. Except that, as you note, with continuous random variables, the probability of the observed data is always $0$.

"$P(\theta=x)$" is nowhere involved.

Suppose we will observe the realized value of the random variable $X$. Given the value of $\theta$, the probability that $X$ is in any measurable set $A$ is $\displaystyle\int_A f(x \mid \theta)\,dx$. Given the observation $X=x$, the MLE is the value of $\theta$ that maximizes the likelihood function $L(\theta) = f(x\mid\theta)$.

Notice that, for example of $X$ has a Poisson distribution with expected value $\theta$, then $\theta$ could be, for example $3.2781$, but $X$ must always be in $\{0,1,2,3,\ldots\}$. If it is observed that $X=4$, it makes no sense to say that then we're considering $P(\theta=4)$. So "$P(\theta=x)$" does not enter what we're doing.

share|improve this answer

Although the others are correct, to address your confusion, the likelihood function is the joint probability density for the observed x as a function of the parameter. So finding the maximum likelihood estimate you find the parameter value that makes this density the highest. The density function will be positive even though the probability of exactly observing the given x is 0. The likelihood for continuous distributions is a density function. Your misunderstanding of this fact is what is causing you confusion. For discrete distributions it is the probability mass function at the observed values of the data as a function of the parameter theta. Also L(θ:D)=P(D|θ) is the joint density. It only factors into the product of the marginal densities when the observations are independent (which is the usual case).

share|improve this answer
    
Likelihood was not defined to me this way (see my edit). Or am I missing something? –  Xodarap Jun 12 '12 at 14:03
    
The definition can be given in a variety of ways. How was it defined to you? Give me your definition and maybe I can show you why the two are equivalent (assuming that you were taught correctly)> –  Michael Chernick Jun 12 '12 at 14:35
    
I edited the question to include my understanding of the definition. –  Xodarap Jun 12 '12 at 16:02

Here's still another way to view the MLE, that really helped clarify it for me:

You're taking the derivative of the pmf (With respect to whatever variable you're trying to isolate) and finding a local maximum by setting that derivative equal to 0.

That's what the MLE is. To look at it from the viewpoint of a normal distribution, you're finding the exact value (Or the formula for it) of the peak (highest probability of occurring - the mean, in the case of the normal), because that's where its derivative changes direction (So it, for an instant, is 0 there)

The log step is because all taking a log does is flattens a curve (Such as you see here, in situations where it won't just flatten the curve you can't take the log) which doesn't change the local maximum at all, just lowers it -- but it almost always makes the derivative more straightforward.

Hope this helps!

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.