The entropy of a multivariate Gaussian is given at https://en.wikipedia.org/wiki/Multivariate_normal_distribution#Entropy

as $$\frac{1}{2}\ln((2\pi e)^n |\Sigma|).$$

Here $n$ is the dimension of the random vector.

What happens when the covariance matrix $\Sigma$ is singular? Imagine, for example that two rows of the matrix are identical. Intuitively I feel this shouldn't reduce the entropy by too much if $n$ is large but the formula no longer applies.

  • $\begingroup$ Have you tried case $n=2$? In general, it will probably be absolute value of the product of non-zero eigenvalues of $\Sigma$ instead of $|\Sigma|$ and corresponding reduction in $n$. $\endgroup$ – A.S. Dec 21 '15 at 16:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.