# Function proportional to the log likelihood for the Gaussian distribution

The following question has been crossposted to CrossValidated upon recommendation from the community and a lack of responses here.

Consider the following problem from a course on statistical inference:

If we generate a sample $$x_i$$ for $$i \in$$ {$$1 ... n$$} from $$p(x_i) = \sum_{k=1}^2 w_kp(x_i| \mu _k, \sigma^2_k)$$ where $$p(x_i| \mu _k, \sigma ^2_k)$$ are Gaussian densities and $$w_2 = 1 - w_1$$ (where we assume that all the parameters are unknown)

Find the log-likelihood function for the sample $$x_{1:n}$$.

The log-likelihood is defined as:

$$l( \theta | x_{1:n}) = \log(p(x_{1:n})) = \log( \Pi _{i=1}^n p(x_i)) = \sum _{i=1}^n \log(p(x_i))$$

Now substituting in for $$p(x_i)$$ we get:

$$l( \theta | x_{1:n}) = \sum_{i=1}^n \log \Big{(} \sum_{k=1}^2 w_k p(x_i | \mu _k , \sigma^2_k) \Big{)}$$

We can now substitute in the Gaussian densities which gives us:

$$l( \theta | x_{1:n}) = \sum_{i=1}^n \log \Big{[} w_1(2 \pi \sigma ^2 _1)^{-1/2} \exp \Big{(}\frac{(x - \mu _1)^2}{2 \sigma ^2 _1} \Big{)} + w_2 (2 \pi \sigma ^2 _2)^{-1/2} \exp \Big{(}\frac{(x - \mu _2)^2}{2 \sigma ^2 _2} \Big{)} \Big{]}$$

The following solution aligns with my working, however, there is a proportionality relation on the last line that is unclear to me.

Why does the final line hold? I can't see a clear and obvious way to get from the point at which I have substituted the Gaussian densities into the sum to it being proportional to the given double summation.

I understand that we can exclude any multiplicative constants, however, it seems as though the summation has been taken out of the logarithm at some stage in order to derive a proportionality relation. Although I'm not sure if this holds and if it does, why does it hold?

I am quite certain that the formula on the grey background is false. This would mean that the log-likelihood of a single sample $$x$$ (i.e. the log of the density of the mixture of two gaussians) is a polynomial in $$x$$ of degree two, i.e. that the mixture of two gaussians is a gaussian. This is obviously false, see https://www.wolframalpha.com/input?i=plot+ln%28%281%2F2%29+e%5E%28-%28x%2B3%2F2%29%5E2%29+%2B+%281%2F2%29+e%5E%28-%28x-3%2F2%29%5E2%29%29+for+x%3D-4..4

Note that the Expectation-Maximization (EM) algorithm – which is where the formula in the grey background seems to come from, as is somewhat visible in the CrossValidated post – avoids this (and keeps simple formulae) by first estimating from which gaussian the sample was sampled before estimating the log-likelihood of the sample together with the number of the gaussian it was sampled from. Let us try to find out what this means.

To find a formula similar to what you are looking for, define first $$z_1, \dots, z_n$$ to be the number of the gaussian from which you sampled $$x_1, \dots, x_n$$. Note that $$z_i$$ is not present in the data. The log-likelihood of the joint variable $$(z_1,x_1)$$ is \begin{align} \mathcal L\mathcal L(z, x) := \ln w_{z} + \ln p(x | \mu_z, \sigma^2_z) ) = \ln w_z - \ln(\sqrt{2\pi\sigma_z^2}) - \frac{(x-\mu_z)^2}{2\sigma_z^2} . \qquad (1) \end{align} On the other hand, $$\mathbb P(z_1 = k | x_1) = \frac{w_k p(x_1 | \mu_k, \sigma_k^2)}{\sum_{k’} w_{k’} p(x_1 | \mu_{k’}, \sigma_{k’}^2)} .$$ You are given $$x_1, \dots, x_n$$ generated according to an unknown mixture of gaussians. If, somehow, you knew $$\mathbb P(z_1 = k | x_1)$$ exactly, then you could generate the $$z_i$$ to obtain a sample $$(z_i, x_i)$$ that has the same distribution as the original data. Unfortunately, $$\mathbb P(z_1 = k | x_1)$$ is unknown because the $$\mu_i, \sigma_i^2$$ are unknown.

Here comes the trick: the EM algorithm assumes you have pretty good estimators for the $$\mu_k, \sigma_k^2$$, so you have a pretty good estimate of $$\mathbb P(z_1 = k | x_1)$$. So you can generate $$z_i$$ in this manner, then compute the log-likelihood of the joint series $$(z_i, x_i)$$ which has a very simple expression $$(1)$$.

In fact, sampling the $$z_i$$ is not needed: you can compute the expected log-likelihood you would obtain in this manner (conditionally on the $$x_i$$), which is given by $$\sum_{k_1, \dots, k_n} \sum_i \mathcal L\mathcal L(k_i, x_i) \mathbb P(z_i = k_i | x_i) .$$ This is where every formula with the sum outside the log ([2] in the CrossValidated post) come from, and as you can see, they do not actually give the true log-likelihood.

Note 1: There is a slight difference in [2], where $$\ln(\sqrt{2\pi\sigma_z^2})$$ is replaced by $$\frac{1}{2}\ln(\sigma_z^2)$$. This is okay because we do not care about additive constants.

Note 2: To be convinced of just how sketchy our reasoning was, be aware that the EM algorithm, which we derived to try and find the maximum likelihood estimator (MLE) via a couple of heuristics, might fail to converge to the MLE, see these lecture notes.