Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have the problem to understand the following simple Maximum (Log) Likelihood example. Let $X$ be a discrete variable with domain $\{1,\dots,K\}$ and the discrete distribution is parametrized

$$P(X=k;\pi) = \pi_k$$

With parameters $\pi = (\pi_1,\dots,\pi_K)$ that are constrainted to fulfill $\sum_k \pi_k = 1$ and there is some data $D = \{x_i\}_{i = 1}^n$

What is the log likelihood $\mathcal{L}(\pi)$ of the data under the model?

I have applied the definition which gives me:

$$\mathcal{L}(\pi) = \log P(x_{1:n};\pi) = \sum_{i = 1}^n \log P(x_i;\pi)$$

At first I thought that it must sum up to 1 and the liklihood is 0, but this does not make sense, this the sum is over the data set which can have different occurences of different $X=k$ values, also the $\log$ is applied every time. The only thing I can think of is that $$\mathcal{L}(\pi) \leq 0$$ since there is no value which is $\geq$ 1.

share|cite|improve this question
Your likelihood is correct to begin with, but you must now write it explicitly. Hint: when you are at a loss, try with some concrete example. Say, take K=2, an n=5, and write down the likelihood. – leonbloy May 31 '12 at 20:17
@leonbloy Does this involve making use of the Lagrange multiplier? – Mahoni May 31 '12 at 21:28
Solving some concrete examples just leads me to likelihood with a negative result, but I lack the idea to generalize this. – Mahoni May 31 '12 at 21:49
It is not the likelihood that is negative, it is the log likelihood that is working out to be negative. – Dilip Sarwate May 31 '12 at 22:46
It does not matter at all that the log-likelihood is negative, the log-lilelihood is not a probability density (not even the likelihood is), you just want to consider it as a function with the parameter/s as variable, and find its maximum – leonbloy May 31 '12 at 23:07
up vote 2 down vote accepted

Suppose $K=3$ and you data consists of 5 samples ${\mathbb X} =\{ 1, 3, 1, 1, 2 \}$ The likelihood of this realization would be $P(X=1)P(X=3)P(X=1)P(X=1) P(X=2) = \pi_1 \pi_3\pi_1\pi_1\pi_2 = \pi_1^3 \pi_2\pi_3 $ Calling $n_i$ the number of samples with value $i$, this can be written in general as $\pi_1^{n_1} \pi_2^{n_2} \pi_3^{n_3}$

In general, then $$\mathcal{L}(\pi) = \sum_{i = 1}^n \log \pi_{x_i} = \sum_{j = 1}^K \log \pi_{j}^{n_j} = \sum_{j = 1}^K {n_j} \log \pi_{j} $$

Now, you must consider this as a function of $\pi=\{\pi_j\}$ ($n_j$ are given by the realization) and find the value of $\pi$ that maximizes this - subjected to the restriction that $\sum \pi_j=1$ and $\pi_j \ge 0$.

This is now a typical problem of multivariate Calculus (maximize a differentiable funcion of several variables subjected to a restriction given as another function) Lagrange multipliers is the standard method . Can you go on from here?

share|cite|improve this answer
Thanks a lot these were many hints, but they helped me to understand the problem at least in terms of the notation. I applied Lagrange multiplier and my result is $\pi_{k}^{ML} = \frac{m_k}{N}$, where $N$ is the size of the given data. – Mahoni Jun 1 '12 at 11:03
Anyway, can be said, that $\mathcal{L}(\pi)$ is the same as $\pi_k^{ML}$ ? – Mahoni Jun 1 '12 at 11:12
I meant is $\pi_k^{ML}$ the result for maximizing $\mathcal{L}(\pi)$? I forgot to say, that my $m_k$ is your $n_i$ – Mahoni Jun 1 '12 at 11:22
Yes, that's the definition of Maximum Likelihood (the (log)likelihood is a function that has the parameter as variable; the ML estimator is the value of the variable that maximizes the function). BTW, your result is correct, and it's good to understand that it is intuitively satisfactory. – leonbloy Jun 1 '12 at 11:42

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.