We define covariance of random vector ${\bf X}$ as \begin{align} Cov({\bf X})=E \left[ \left( {\bf X}-E[{\bf X}] \right) \left( {\bf X}-E[{\bf X}] \right)^T \right]. \end{align}

In the scalar case there exists the following inequality \begin{align} Var(X)= E[ (X-E[X])^2] \le E[ (X-c)^2] \end{align} for any $c \in \mathbb{R}$.

My question does there exist an equivalent inequality (in positive semidefinite sence) for covarience matrix that is \begin{align} Cov({\bf X})=E \left[ \left( {\bf X}-E[{\bf X}] \right) \left( {\bf X}-E[{\bf X}] \right)^T \right] \preceq E \left[ \left( {\bf X}-{\bf c} \right) \left( {\bf X}-{\bf c}] \right)^T \right] (*) \end{align} for any ${\bf c} \in \mathbb{R}^n$?

Note that, via orthogonality principle, it is not difficult to show the following inequality \begin{align} {\rm Tr} \left(Cov({\bf X})\right) \le {\rm Tr} \left(E \left[ \left( {\bf X}-{\bf c} \right) \left( {\bf X}-{\bf c}] \right)^T \right]\right) (**) \end{align}

Note that if (*) is true it would imply (**) by monotonicity of trace operator. However, the inverse is not true.

I was thinking that the proof would go something like this.

W.l.o.g. assume that $E[{\bf X}]=0$
Then we have to show that \begin{align} z^T \left(E \left[ \left( {\bf X}-{\bf c} \right) \left( {\bf X}-{\bf c}] \right)^T \right]- E \left[ {\bf X} {\bf X} ^T \right] \right)z \ge 0 \end{align}

which simplifies to

\begin{align} z^T \left({\bf c} {\bf c}^T \right)z \ge 0 \end{align}


For $E[(X-E[X])^2]$, the minimality of $c=E[X]$ can be understood from a number of vantage points. On the one hand, it is directly related to the abstract definition of conditional expectation, which asks to minimize $E[(X-Y)^2|F]$ where $F$ is the sigma algebra of constant functions. On the other hand,


from which $c=E[X]$ gives minimality.

For the multidimensional case, we have:

\begin{align*} E[(X-c)(X-c)^T]&=E[(X-E[X]+E[X]-c)(X-E[X]+E[X]-c)^T]\\ &=E[(X-E[X])(X-E[X])^T]+(E[X]-c)(E[X]-c)^T. \end{align*}

The problem is that $(E[X]-c)(E[X]-c)^T$ is not necessarily positive in all its entries, as the example of $yy^T$ shows for $y^T=(1,-2)$. It is however positive definite, in that for any $z$ and $c$:

$$z^TE[(X-E[X])(X-E[X])^T]z\leq z^T(E[X]-c)(E[X]-c)^Tz.$$

Now try to work out the result for the covariance matrix $E[(X-c)(Y-d)^T]$.

  • $\begingroup$ Thanks. A few questions? Just to summarize the inequality that was proposed holds, right? Then the suggestion you have $E[(X-c)(Y-d)]$ it's just for exercise, right? Also, should it be $E[(X-c)(Y-d)^T]$ ? $\endgroup$ – Boby Jun 14 '16 at 22:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.