# Maximum Likelihood Estimator of parameters of multinomial distribution

Suppose that 50 measuring scales made by a machine are selected at random from the production of the machine and their lengths and widths are measured. It was found that 45 had both measurements within the tolerance limits, 2 had satisfactory length but unsatisfactory width, 2 had satisfactory width but unsatisfactory length, 1 had both length and width unsatisfactory. Each scale may be regarded as a drawing from a multinomial population with density

$$\pi_{11}^{x_{11}} \pi_{12}^{x_{12}} \pi_{21}^{x_{21}}(1-\pi_{11}-\pi_{12}-\pi_{21})^{x_{22}}$$

Obtain the maximum likelihood estimates of the parameters.

I have tried this by the following way:

the likelihood function is

\begin{align} L & =L(\pi_{11},\pi_{12},\pi_{21},(1-\pi_{11}-\pi_{12}-\pi_{21})) \\[8pt] & =\prod_{i=1}^{50}[\pi_{11}^{x_{11}} \pi_{12}^{x_{12}} \pi_{21}^{x_{21}}(1-\pi_{11}-\pi_{12}-\pi_{21})^{x_{22}}] \\[8pt] & =[\pi_{11}^{x_{11}} \pi_{12}^{x_{12}} \pi_{21}^{x_{21}}(1-\pi_{11}-\pi_{12}-\pi_{21})^{x_{22}}]^{50} \\[8pt] & =[\pi_{11}^{45}\pi_{12}^{2} \pi_{21}^{2}(1-\pi_{11}-\pi_{12}-\pi_{21})^{1}]^{50} \\[8pt] & =\pi_{11}^{2250}\pi_{12}^{100} \pi_{21}^{100}(1-\pi_{11}-\pi_{12}-\pi_{21})^{50} \end{align}

Taking logarithm of the likelihood function yields,

\begin{align} L^*& =\log L=\log \left[\pi_{11}^{2250} \pi_{12}^{100} \pi_{21}^{100}(1-\pi_{11}-\pi_{12}-\pi_{21})^{50}\right] \\[8pt] & =2250\log [\pi_{11}]+100\log [\pi_{12}]+100\log [\pi_{21}]+50\log (1-\pi_{11}-\pi_{12}-\pi_{21}) \end{align}

Now taking the first derivative of $$L^*$$ with respect to $$\pi_{11}$$

$$\frac{\partial L^*}{\partial \pi_{11}}$$ $$=\frac{2250}{\pi_{11}}-\frac{50}{(1-\pi_{11}-\pi_{12}-\pi_{21})}$$

setting $$\frac{\partial L^*}{\partial \pi_{11}}$$ equal to $$0$$,

$$\frac{\partial L^*}{\partial \hat\pi_{11}}=0$$

$$\Rightarrow\frac{2250}{\hat\pi_{11}}-\frac{50}{(1-\hat\pi_{11}-\hat\pi_{12}-\hat\pi_{21})}=0$$

$$\Rightarrow \hat\pi_{11}=\frac{45(1-\hat\pi_{12}-\hat\pi_{21})}{44}$$

$$\bullet$$Are the procedure and estimate of $$\pi_{11}$$ correct?

$$\bullet$$I have another question that if it is multinomial then where the term $$\binom{n}{x_{11}x_{12}x_{21}x_{22}}=\binom{50}{45,2,2,1}$$?

Consider a positive integer $n$ and a set of positive real numbers $\mathbf p=(p_x)$ such that $\sum\limits_xp_x=1$. The multinomial distribution with parameters $n$ and $\mathbf p$ is the distribution $f_\mathbf p$ on the set of nonnegative integers $\mathbf n=(n_x)$ such that $\sum\limits_xn_x=n$ defined by $$f_\mathbf p(\mathbf n)=n!\cdot\prod_x\frac{p_x^{n_x}}{n_x!}.$$ For some fixed observation $\mathbf n$, the likelihood is $L(\mathbf p)=f_\mathbf p(\mathbf n)$ with the constraint $C(\mathbf p)=1$, where $C(\mathbf p)=\sum\limits_xp_x$. To maximize $L$, one asks that the gradient of $L$ and the gradient of $C$ are colinear, that is, that there exists $\lambda$ such that, for every $x$, $$\frac{\partial}{\partial p_x}L(\mathbf p)=\lambda\frac{\partial}{\partial p_x}C(\mathbf p).$$ In the present case, this reads $$\frac{n_x}{p_x}L(\mathbf p)=\lambda,$$ that is, $p_x$ should be proportional to $n_x$. Since $\sum\limits_xp_x=1$, one gets finally $\hat p_x=\dfrac{n_x}n$ for every $x$.

• "To maximize L, one asks that the gradients of L of C are colinear" -- why is that sentence true? Oct 1, 2014 at 21:52
• – Did
Oct 2, 2014 at 0:00
• Why $\frac{n_x}{p_x}L(\mathrm{\mathbf{p}})= \lambda$ implies $p_x$ should be proportional to $n_x$? Should'n it be $\frac{n_x}{p_x}=constant$ to reach that conclusion? I had this minor concern about this answer, but it must be right since GyuHyeon Choi's below gives the same result.
– abcd
Dec 7, 2019 at 22:38
• great answer! can you please take a look at my question? math.stackexchange.com/questions/4598359/… thanks! Dec 14, 2022 at 4:46
• @abcd Since $\frac{n_x}{p_x}L(\mathbf p)=\lambda$ for each $p_x$, that means $\frac{n_x}{\lambda}L(\mathbf p)=p_x$. Summation to one means $1 = \sum_x \frac{n_x}{\lambda}L(\mathbf p) = \frac{L(\mathbf p)}{\lambda} \sum_x n_x = \frac{L(\mathbf p)}{\lambda} n$, so $\lambda = L(\mathbf p) n$ and hence $n_x/p_x = n$ or $p_x = n_x/n$.
– Mew
Mar 27, 2023 at 10:24

If an observation is

\begin{align} p_1 = P(X_1) &= \frac{x_1}{n} \\ &...\\ p_m = P(X_m) &= \frac{x_m}{n} \end{align}

then the likelihood which can be described as joint probability is (https://en.wikipedia.org/wiki/Multinomial_theorem)

\begin{align} L(\mathbf{p}) &= {{n}\choose{x_1, ..., x_m}}\prod_{i=1}^m p_i^{x_i} \\ &= n! \prod_{i=1}^m \frac{p_i^{x_i}}{x_i!} \end{align}

and the log-likelihood is

\begin{align} l(\mathbf{p}) = \log L(\mathbf{p}) &= \log \bigg( n! \prod_{i=1}^m \frac{p_i^{x_i}}{x_i!} \bigg)\\ &= \log n! + \log \prod_{i=1}^m \frac{p_i^{x_i}}{x_i!} \\ &= \log n! + \sum_{i=1}^m \log \frac{p_i^{x_i}}{x_i!} \\ &= \log n! + \sum_{i=1}^m x_i \log p_i - \sum_{i=1}^m \log x_i! \end{align}

Posing a constraint ($$\sum_{i=1}^m p_i = 1$$) with Lagrange multiplier

\begin{align} l'(\mathbf{p},\lambda) &= l(\mathbf{p}) + \lambda\bigg(1 - \sum_{i=1}^m p_i\bigg) \end{align}

To find $$\arg\max_\mathbf{p} L(\mathbf{p},\lambda)$$

\begin{align} \frac{\partial}{\partial p_i} l'(\mathbf{p},\lambda) = \frac{\partial}{\partial p_i} l(\mathbf{p}) + \frac{\partial}{\partial p_i} \lambda\bigg(1 - \sum_{i=1}^m p_i\bigg) &= 0\\ \frac{\partial}{\partial p_i} \sum_{i=1}^m x_i \log p_i - \lambda \frac{\partial}{\partial p_i} \sum_{i=1}^m p_i &= 0 \\ \frac{x_i}{p_i}- \lambda &= 0 \\ p_i &= \frac{x_i}{\lambda} \\ \end{align}

Thus, \begin{align} p_i &= \frac{x_i}{n} \end{align}

because

\begin{align} p_i &= \frac{x_i}{\lambda} \\ \sum_{i=1}^m p_i &= \sum_{i=1}^m \frac{x_i}{\lambda} \\ 1 &= \frac{1}{\lambda} \sum_{i=1}^m x_i \\ \lambda &= n \end{align}

Finally, the probability distribution that maximizes the likelihood of observing the data

\begin{align} \mathbf{p} = \bigg( \frac{x_1}{n}, ..., \frac{x_m}{n} \bigg) \end{align}

• Great answer! Can you please show the formula for the variance of these probabilities? Thanks! (I have a related question here: math.stackexchange.com/questions/4598359/…) Dec 14, 2022 at 6:28
• This blog: blog.jakuba.net/maximum-likelihood-for-multinomial-distribution appears to be a fan of your derivation! Jan 19, 2023 at 19:02
• Nevertheless, the one part of the derivation that I would change is breaking up the multinomial coefficient in the likelihood. Rather, keep the multinomial coefficient in tact, then take the natural logarithm to form the log-likelihood. The natural logarithm of the multinomial coefficient separates from $\sum_{i=1}^{m} x_{i} ln(p_{i}),$ and maximum likelihood estimation only considers the latter due to argmax. Now, the benefit is that there is an immediate correspondence with math.stackexchange.com/questions/2725539/…. Jan 19, 2023 at 19:17

Let $\mathbf{X}$ be a RV following multinomial distribution. Then, \begin{align}P(\mathbf{X} = \mathbf{x};n,\mathbf{p}) &= n!\,\Pi_{k=1}^K \frac{p_k^{x_k}}{x_k!} \end{align} $x_i$ is the number of success of the $k^{th}$ category in $n$ random draws, where $p_k$ is the probability of success of the $k^{th}$ category. Note that, \begin{align}\sum_{k=1}^K x_k &= n\\ \sum_{k=1}^{K} p_k &=1 \end{align}

For the estimation problem, we have $N$ samples $\mathbf{X_1}, \ldots,\mathbf{X_N}$ drawn independently from above multinomial distribution. The log-liklihood is given as $$\mathcal{L}(\mathbf{p},n) = \sum_{i=1}^N \log P(\mathbf{x_i},n,\mathbf{p})$$ where \begin{align}\log P(\mathbf{x_i},n,\mathbf{p}) &= \log \frac{n!}{\Pi_k x_{ik}!} + \sum_{k=1}^{K} x_{ik} \log p_k \\ \sum_{i=1}^N \log P(\mathbf{x_i},n,\mathbf{p}) &= C + \sum_{k=1}^{K} N_k \log p_k \end{align} where $N_k = \sum_{i=1}^{N} x_{ik}$, is the total number of success of $k^{th}$ category in $N$ samples.

For MLE estimate of $\mathbf{p}$, assuming $n$ is known, we solve the following optimization problem: \begin{align} \max_{\mathbf{p}} &\,\, \mathcal{L}(\mathbf{p},n) \\ s.t. & \,\, \sum_{k=1}^{K} p_k \,\,=1\end{align} Using equality constraint for variable reduction, $$p_K\,=\, 1 - \sum_{k=1}^{K-1} p_k$$ We have an unconstrained problem in $K-1$ variables. Compute gradient for stationary point computation as, \begin{align}\frac{\partial\mathcal{L}(\mathbf{p},n)}{\partial p_k} &= \frac{N_k}{p_k} - \frac{N_K}{p_K}\,\,=\,\, 0 \\ p_k &= \frac{N_k\,p_K}{N_K}\end{align} Solving, with $\sum_{k=1}^{K} p_k\,=\, 1$ gives MLE estimate for $p_k$, $$p_k = \frac{N_k}{nN}$$

• Great answer! Can you please show the formula for the variance of these probabilities? Thanks! (I have a related question here: math.stackexchange.com/questions/4598359/…) Dec 14, 2022 at 6:29

From the beginning, you assume a multinomial (not product) kdistribution of $$n=50$$ i.e., for counts $$x_{11}, x_{12},x_{21},x_{22}$$ from the $$2×2$$ contingency table of four possible events. Therefore, $$X\sim Multn(n, p_{11},p_{12},p_{21},p_{22})$$ The PMF~ with interest to maximize the parameter p={p_i: I=1 to q categories/outcomes}: $$L(p) = P_X(x_{11}, x_{12},x_{21},x_{22}: n, p_{11},p_{12},p_{21},p_{22})$$ Which is, $$= \frac{n! p_{11}^{x_{11}} \cdot p_{12}^{x_{12}} \cdot p_{21}^{x_{21}} \cdot p_{22}^{x_{22}} }{x_{11}! x_{12}! x_{21}! x_{22}! }$$

Hence, we can write $$Log_e{L(p)} = \log_e { \frac{50! p_{11}^{x_{11}} \cdot p_{12}^{x_{12}} \cdot p_{21}^{x_{21}} \cdot p_{22}^{x_{22}} }{x_{11}! x_{12}! x_{21}! x_{22}! }}$$

That is, $$Log_e(L(p)) = \log_e{\frac{50! p_{11}^45 p_{12}^2 p_{21}^2 p_{22}^1}{45! 2! 2! 1! }}$$

From there you equate the slope [differentiation with respect to p] to zero, ... the process of differentiation you did was correct, but not for the right pmf. And, combination $$nC_{***}$$ should be included. That is, $$∆(Log(L))= ∆(\log_e{ n!} + \log_e{p_{11}^45} + \log_e{p_{12}^2} + \log_e{p_{21}^2} + \log_e{p_{22}^1} - \log_e{45!} - \log_e{ 2!} - \log_e{2!} - \log_e{1!})$$ These reduce to only expressions with p, that we are maximizing, that is $$∆[ \log_e{p_{11}^45} + \log_e{p_{12}^2} + \log_e{p_{21}^2} + \log_e{p_{22}^1}]= 0$$ Sum of $$P_{ij}$$ equal 1, $$p_i= \frac{n_{ij}}{N}$$ , hence $$p_{21}=\frac{2}{50}$$ Hope this help.