The covariance matrix is given $$ \Sigma_{\textbf{XX}} = \text{cov}[\textbf{X}, \textbf{X}] = E[(\textbf{X} - \mu_{\textbf{X}})(\textbf{X} - \mu_{\textbf{X}})^\top] $$ We want to prove that $\Sigma_{\textbf{XX}}$ is positive semi-definite. For this, we can say that if $\Sigma_{\textbf{XX}} \in \mathbb{R}^{n\times n}$, $\forall \textbf{u} \in \mathbb{R}^n$ \begin{align} \textbf{u}^\top \Sigma_{\textbf{XX}} \textbf{u} &=\textbf{u}^\top E[(\textbf{X} - \mu_{\textbf{X}})(\textbf{X} - \mu_{\textbf{X}})^\top]\textbf{u}\\ &= E[\textbf{u}^\top(\textbf{X} - \mu_{\textbf{X}})(\textbf{X} - \mu_{\textbf{X}})^\top\textbf{u}]\\ &= E[((\textbf{X} - \mu_{\textbf{X}})^\top\textbf{u})^\top(\textbf{X} - \mu_{\textbf{X}})^\top\textbf{u}]\quad \textbf{(1)}\\ &= E[((\textbf{X} - \mu_{\textbf{X}})^\top\textbf{u})^2]\ge 0 \quad \textbf{(2)}\\ \end{align}

Here are my 2 questions:

  1. What is the linear algebra property allowing to go from $\textbf{(1)}$ to $\textbf{(2)}$?

  2. What is the information that allows to state in the last equation that $E[((\textbf{X} - \mu_{\textbf{X}})^\top\textbf{u})^2]\ge 0$ and not $E[((\textbf{X} - \mu_{\textbf{X}})^\top\textbf{u})^2]> 0$ $\iff$ how can we state that $\Sigma_{\textbf{XX}}$ is positive semi-definite and not positive definite?

Edit (thanks to John Hughes) \begin{align} \textbf{u}^\top \Sigma_{\textbf{XX}} \textbf{u} &= E[\underbrace{((\textbf{X} - \mu_{\textbf{X}})^\top\textbf{u})^\top}_{\in \mathbb{R}}\underbrace{(\textbf{X} - \mu_{\textbf{X}})^\top\textbf{u}}_{\in \mathbb{R}}]\\ &= E[((\textbf{X} - \mu_{\textbf{X}})^\top\textbf{u})^2]\ge 0 \\ \end{align}


Look at line 1 as having the form $$ E[s^t s], $$ where $s = (X - \mu_X)^t u$. Now $s^t s$ is, in general, the same as $s \cdot s$, so we can say that $s^t s = \| s \|^2$ (by definition of length!). In this case, it appears that $s$ is just a number, so $s^t s$ is simple $s^2$.

For your second question, look at the number $s$: it might always be zero, in which case $s^2$ would always be zero, so the expected value would always be $0$. When you have a non-negative random variable, the expected value is also non-negative, but not necessarily positive!

  • $\begingroup$ Many thanks @JohnHughes. Is the edit of my question reflects your point accurately regarding the first question? Regarding the question you mean $\text{X}$ could be the zero vector? $\endgroup$
    – ecjb
    Sep 22 '19 at 14:27
  • $\begingroup$ Sure .. the everywhere zero vector is a random variable, with mean $\mu_X$ being the zero vector, so you get a covariance of zero. $\endgroup$ Sep 22 '19 at 14:36
  • $\begingroup$ many thanks @JohnHughes $\endgroup$
    – ecjb
    Sep 22 '19 at 15:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.