# Proof that covariance matrix is positive semi-definite (and not positive definite)

The covariance matrix is given $$\Sigma_{\textbf{XX}} = \text{cov}[\textbf{X}, \textbf{X}] = E[(\textbf{X} - \mu_{\textbf{X}})(\textbf{X} - \mu_{\textbf{X}})^\top]$$ We want to prove that $$\Sigma_{\textbf{XX}}$$ is positive semi-definite. For this, we can say that if $$\Sigma_{\textbf{XX}} \in \mathbb{R}^{n\times n}$$, $$\forall \textbf{u} \in \mathbb{R}^n$$ \begin{align} \textbf{u}^\top \Sigma_{\textbf{XX}} \textbf{u} &=\textbf{u}^\top E[(\textbf{X} - \mu_{\textbf{X}})(\textbf{X} - \mu_{\textbf{X}})^\top]\textbf{u}\\ &= E[\textbf{u}^\top(\textbf{X} - \mu_{\textbf{X}})(\textbf{X} - \mu_{\textbf{X}})^\top\textbf{u}]\\ &= E[((\textbf{X} - \mu_{\textbf{X}})^\top\textbf{u})^\top(\textbf{X} - \mu_{\textbf{X}})^\top\textbf{u}]\quad \textbf{(1)}\\ &= E[((\textbf{X} - \mu_{\textbf{X}})^\top\textbf{u})^2]\ge 0 \quad \textbf{(2)}\\ \end{align}

Here are my 2 questions:

1. What is the linear algebra property allowing to go from $$\textbf{(1)}$$ to $$\textbf{(2)}$$?

2. What is the information that allows to state in the last equation that $$E[((\textbf{X} - \mu_{\textbf{X}})^\top\textbf{u})^2]\ge 0$$ and not $$E[((\textbf{X} - \mu_{\textbf{X}})^\top\textbf{u})^2]> 0$$ $$\iff$$ how can we state that $$\Sigma_{\textbf{XX}}$$ is positive semi-definite and not positive definite?

Edit (thanks to John Hughes) \begin{align} \textbf{u}^\top \Sigma_{\textbf{XX}} \textbf{u} &= E[\underbrace{((\textbf{X} - \mu_{\textbf{X}})^\top\textbf{u})^\top}_{\in \mathbb{R}}\underbrace{(\textbf{X} - \mu_{\textbf{X}})^\top\textbf{u}}_{\in \mathbb{R}}]\\ &= E[((\textbf{X} - \mu_{\textbf{X}})^\top\textbf{u})^2]\ge 0 \\ \end{align}

## 1 Answer

Look at line 1 as having the form $$E[s^t s],$$ where $$s = (X - \mu_X)^t u$$. Now $$s^t s$$ is, in general, the same as $$s \cdot s$$, so we can say that $$s^t s = \| s \|^2$$ (by definition of length!). In this case, it appears that $$s$$ is just a number, so $$s^t s$$ is simple $$s^2$$.

For your second question, look at the number $$s$$: it might always be zero, in which case $$s^2$$ would always be zero, so the expected value would always be $$0$$. When you have a non-negative random variable, the expected value is also non-negative, but not necessarily positive!

• Many thanks @JohnHughes. Is the edit of my question reflects your point accurately regarding the first question? Regarding the question you mean $\text{X}$ could be the zero vector?
– ecjb
Sep 22 '19 at 14:27
• Sure .. the everywhere zero vector is a random variable, with mean $\mu_X$ being the zero vector, so you get a covariance of zero. Sep 22 '19 at 14:36
• many thanks @JohnHughes
– ecjb
Sep 22 '19 at 15:17