Gaussian mixture model and compound multivariate normal distribution

I am new to Machine Learning with a bit of background in engineering and statistics, probability and stochastic processes.

I am currently reading Ian Goodfellow's Deep Learning book, section 3.9.6, and trying to fathom his explanation of the Gaussian Mixture Model.

I know about compound probability distributions that are distributions whose parameters are themselves random variables, and that's how i understand his definition of "Mixture of Distributions". Correct me if i am wrong.

When it comes down to the Gaussian Mixture Model, that's where things get tricky. The notation and terminology I find confusing.

The intuition I get from what's written is that a Gaussian Mixture Model is simply a compound Multivariate Normal distribution for which $\mathbf{\mu}$ is a vector-valued random variable and $\mathbf{\Sigma}$ a matrix-valued random variable.

Please enlighten me, this machine-learning terminology and notation confuses me coming from a engineering background.

Mixture models are models that have joint distributions given by:

$$p(x,z) = p(x|z)p(z)$$

where $x$ are your observations, and $z$ are hidden/latent variables. In the ML literature, it is very often the case that the $z$ variables follow a discrete categorical distribution (they can take one of K possible values), which is a model for the class they belong to. Then $x$ can be taken as any distribution (referred to as the base distribution), in the case of the Gaussian mixture model, $x$ is taken as a Gaussian. We therefore have

\begin{align*} p(x) &= \sum_{k=1}^K p(x|z=k)p(z=k)\\ &= \sum_{k=1}^K \pi_k p(x|z=k)\\ &= \sum_{k=1}^K \pi_k \mathcal{N}(x|\mu_k, \Sigma_k) \end{align*}

where the first equality follows from the law of total probability, and we write $\pi_k = p(z=k)$. Note that this tells us that the probability of observing $x$ is a weighted combination of observing $x$ under each of the $K$ possible classes that $z$ belongs to. Note further that since the $\pi_k$'s are probabilities, we have that:

$$\pi_k \in [0,1], \qquad \sum_{k=1}^K\pi_k = 1$$

and so this is in fact a convex combination of the base distribution.

I like to think of this in the context of the MNIST dataset, in which we have 10 classes, (the digits [0-9]), so our latent variable models the prior probability of each class (There are $K=10$ classes). Now, within each class, the observations are normally distributed with their own mean vector and covariance matrices - naturally we expect the parameters of the gaussian distribution that generated an image of a zero digit to differ from the gaussian that generated the digit seven for example.

Hopefully this clarifies what is meant by a GMM, and it is indeed an example of a compound/mixture distribution from the statistics literature.

As a side note, the book by Goodfellow has some useful deep learning materials, but is very vague when it comes to the more meaty statistical concepts, you should look at one of these books for more solid explanations: Bishop, Murphy, ESL

• Thank you for the reference books. ESL seems like a solid pick. – Roulbacha Jan 29 '18 at 3:30