# 'Bounds' on the Covariance Matrix

We define covariance of random vector ${\bf X}$ as \begin{align} Cov({\bf X})=E \left[ \left( {\bf X}-E[{\bf X}] \right) \left( {\bf X}-E[{\bf X}] \right)^T \right]. \end{align}

In the scalar case there exists the following inequality \begin{align} Var(X)= E[ (X-E[X])^2] \le E[ (X-c)^2] \end{align} for any $c \in \mathbb{R}$.

My question does there exist an equivalent inequality (in positive semidefinite sence) for covarience matrix that is \begin{align} Cov({\bf X})=E \left[ \left( {\bf X}-E[{\bf X}] \right) \left( {\bf X}-E[{\bf X}] \right)^T \right] \preceq E \left[ \left( {\bf X}-{\bf c} \right) \left( {\bf X}-{\bf c}] \right)^T \right] (*) \end{align} for any ${\bf c} \in \mathbb{R}^n$?

Note that, via orthogonality principle, it is not difficult to show the following inequality \begin{align} {\rm Tr} \left(Cov({\bf X})\right) \le {\rm Tr} \left(E \left[ \left( {\bf X}-{\bf c} \right) \left( {\bf X}-{\bf c}] \right)^T \right]\right) (**) \end{align}

Note that if (*) is true it would imply (**) by monotonicity of trace operator. However, the inverse is not true.

I was thinking that the proof would go something like this.

W.l.o.g. assume that $E[{\bf X}]=0$
Then we have to show that \begin{align} z^T \left(E \left[ \left( {\bf X}-{\bf c} \right) \left( {\bf X}-{\bf c}] \right)^T \right]- E \left[ {\bf X} {\bf X} ^T \right] \right)z \ge 0 \end{align}

which simplifies to

\begin{align} z^T \left({\bf c} {\bf c}^T \right)z \ge 0 \end{align}

For $E[(X-E[X])^2]$, the minimality of $c=E[X]$ can be understood from a number of vantage points. On the one hand, it is directly related to the abstract definition of conditional expectation, which asks to minimize $E[(X-Y)^2|F]$ where $F$ is the sigma algebra of constant functions. On the other hand,

$$E[(X-c)^2]=E[(X-E[X]+E[X]-c)^2]=E[(X-E[X])^2]+(E[X]-c)^2,$$

from which $c=E[X]$ gives minimality.

For the multidimensional case, we have:

\begin{align*} E[(X-c)(X-c)^T]&=E[(X-E[X]+E[X]-c)(X-E[X]+E[X]-c)^T]\\ &=E[(X-E[X])(X-E[X])^T]+(E[X]-c)(E[X]-c)^T. \end{align*}

The problem is that $(E[X]-c)(E[X]-c)^T$ is not necessarily positive in all its entries, as the example of $yy^T$ shows for $y^T=(1,-2)$. It is however positive definite, in that for any $z$ and $c$:

$$z^TE[(X-E[X])(X-E[X])^T]z\leq z^T(E[X]-c)(E[X]-c)^Tz.$$

Now try to work out the result for the covariance matrix $E[(X-c)(Y-d)^T]$.

• Thanks. A few questions? Just to summarize the inequality that was proposed holds, right? Then the suggestion you have $E[(X-c)(Y-d)]$ it's just for exercise, right? Also, should it be $E[(X-c)(Y-d)^T]$ ? – Boby Jun 14 '16 at 22:28