0
$\begingroup$

I was playing with generating 2D data from a Gaussian distribution with a covariance matrix that has non-zero off-diagonal elements (i.e., a generic covariance matrix).

I noticed that, whatever covariance matrix I use for generating data, I get

$$\det(\text{diag}(\hat{\Sigma})) > \det{(\hat{\Sigma})}$$

where $\hat{\Sigma}$ is the empirical covariance matrix.

Here's a short code example in Python, in case it's helpful:

import numpy as np

data = np.random.multivariate_normal(mean=[0, 0, 0], cov=np.array([[2, 0.9, 0.3], 
                                                                   [0.9, 4, 0.4], 
                                                                   [0.3, 0.4, 3]]), size=10000)

Σ_full = 1 / len(data) * data.T @ data
Σ_diag = Σ_full * np.eye(3)

print(np.linalg.det(Σ_diag) - np.linalg.det(Σ_full))  # <- this is positive

Since $\hat{\Sigma} \approx \Sigma$, on hindsight, my question is really: given a positive definite matrix, is it true that the determinant of its diagonal version is greater than or equal to the determinant of itself?


I'm developing an intuition of this phenomenon as follows.

We know that the determinant of $\text{diag}{(\Sigma)}$ is just the product of its diagonal entries (i.e., the variances), and the determinant of $\Sigma$ is just the determinant of its eigenvalues. This leads to a geometric argument that, if I can argue geometrically that $\Sigma$ scales a unit space less than $\text{diag}(\Sigma)$, then the claim is obviously true.

A thought along this approach is that the column vectors of $\text{diag}{(\Sigma)}$ are orthogonal, yet those of $\Sigma$ are not. Therefore, if their column vectors were all unit length, then clearly $\Sigma$ scales a unit space less than $\text{diag}(\Sigma)$ because the column vectors of $\Sigma$ are pointing closer together (all in the positive quadrant) in space and constructs a "flatter" parallelepiped. But the problem is that their column vectors are not of unit length.

$\endgroup$
3
  • $\begingroup$ For example, for a 2-dim one, det(covariance)=var X * var Y - (cov(X,Y))^2 which is clearly <= det(diag) $\endgroup$
    – Paul
    Commented Nov 20, 2021 at 0:14
  • $\begingroup$ math.stackexchange.com/questions/556486/… $\endgroup$
    – user140541
    Commented Nov 20, 2021 at 0:19
  • $\begingroup$ For reference, this is related to Exercise 4.12 of Machine Learning: A Probabilistic Perspective by Kevin Murphy. $\endgroup$ Commented Nov 20, 2021 at 0:41

1 Answer 1

0
$\begingroup$

This result is known as Hadamard's inequality. First, notice that any $n\times n$ covariance matrix $\Sigma$ is positive semi-definite. If $\Sigma$ is singular, the inequality holds trivially. So, we may assume that $\Sigma$ is positive definite. Let $A=\operatorname{diag}(\Sigma)^{-1/2}$. Then $\operatorname{det}(A\Sigma A)\le 1$ iff $\operatorname{det}(\Sigma)\le \prod_{i=1}^n\Sigma_{ii}$ and, therefore, we may assume that $\Sigma_{11}=\cdots=\Sigma_{nn}=1$. Finally, $$ \operatorname{det}(\Sigma)=\prod_{i=1}^n \lambda_i\le \left(\frac{1}{n}\sum_{i=1}^n\lambda_i\right)^n=\left(\frac{1}{n}\operatorname{tr}(\Sigma)\right)^n=1 $$ by the AM-GM inequality, where $\lambda_1,\ldots,\lambda_n$ are the eigenvalues of $\Sigma$.

$\endgroup$
1
  • $\begingroup$ Very cool. For those who are interested, Wikipedia's page on Hadamard's inequality also offer some great geometric intuition. $\endgroup$ Commented Nov 20, 2021 at 0:53

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .