# GMM with full and diagonal covariances

I have Gaussian Mixture Model-- distribution with probability density function, that is a weighted sum of Gaussian probability density functions: $$p(X)=\sum_{i=1}^k \omega_i\mathcal{N}(X,\mu_i,\Sigma_i)=\sum_{i=1}^k \omega_ip_i(X),$$

where $k$ is the number of components, $\mathcal{N}(X,\mu_i,\Sigma_i), i=1,...,k$ are Gaussian densities with expectations (vectors) $\mu_i,i=1,...,k$ and covariance matrices $\Sigma_i,i=1,...,k$,

$\omega_i,i=1,...,k$ are weights: $\sum_{i=1}^k \omega_i=1.$

Covariance matrices $\Sigma_i,i=1,...,k$,are full -- have correlation elements (non-zero non-diagonal elements).
How I can approximate this GMM via GMM with components with diagonal covariances. It is understood, that it will be more components in the weighted sum, but they will be diagonal. Here on page 2 in is written, that it is possible (but without proof) :

https://www.ll.mit.edu/mission/cybersec/publications/publication-files/full_papers/0802_Reynolds_Biometrics-GMM.pdf

"It is also important to note that because the component Gaussian are acting together to model the overall feature density, full covariance matrices are not necessary even if the features are not statistically independent. The linear combination of diagonal covariance basis Gaussians is capable of modeling the correlations between feature vector elements. The effect of using a set of M full covariance matrix Gaussians can be equally obtained by using a larger set of diagonal covariance Gaussians. "

But how it can be done and what can be say if to compare cost of calculations for these 2 cases? Is it faster to use in calculations more components, but diagonal? Thank you.

• I stumbled upon the same phrase in the same paper and performed the search, only to find your question without answer :) Have found the proof? Commented Jun 30, 2016 at 10:55
• I've found similar question at researchgate.net/post/…, but I do not understand the answer Commented Jun 30, 2016 at 11:13

I don't know if this helps you. But the same claim has been made in

the exact passage is:

While the general model form supports full covariance matrices, that is, a covariance matrix with all its elements, typically only diagonal covariance matrices are used. This is done for three reasons. First, the density modeling of an Mth-order full covariance GMM can equally well be achieved using a larger-order diagonal covariance GMM.

with the explanation being:

GMMs with M > 1 using diagonal covariance matrices can model >distributions of feature vectors with correlated elements. Only in the degenerate case of M = 1 is the use of a diagonal covariance matrix incorrect for >feature vectors with correlated elements.

• The link is broken Commented Aug 27, 2023 at 17:12

This is a "standard" problem in machine learning.

You typically use the Expectation Maximization algorithm. You find complete derivations of it in many books, such as "Pattern Recognition and Machine Learning" by C.M. Bishop. The derivation is essentially the same for diagonal or general covariance matrices.

You can find complete software implementations in many languages, e.g. https://scikit-learn.org/stable/modules/generated/sklearn.mixture.GaussianMixture.html

The performance difference depends on the number of components, the dimensionality of the data, and the distribution of the data. The computation speed also scales differently with the number of data samples. It is hard to say something general, and optimal parameters are selected by cross validation rather than theoretical considerations.

Consider posting this on CrossValidated.SE or StackOverflow, since those sites might provide better fit with the question.