$\newcommand{\cov}{\operatorname{cov}}\newcommand{\corr}{\operatorname{corr}}$I have two dependent multivariate random variables, or random vectors, $X$ and $Y$, their respective covariance matrices $\Sigma_X = \cov(X,X)$, $\Sigma_Y = \cov(Y,Y)$ and a non-zero correlation matrix $P_{XY} = \corr(X, Y)$.

I would like to compute the covariance matrix of the sum of the two random vectors:

\begin{align} Z {}={}& X + Y \\ \Sigma_Z {}={}& \Sigma_X + \Sigma_Y + 2 \cov(X, Y) \end{align}

Intuitively, I would assume the cross-covariance can be computed as follows:

$$\cov(X, Y) = P_{XY} \sqrt{\Sigma_X} \sqrt{\Sigma_Y}$$

But I can see that this matrix is not necessarily symmetric, and therefore the resulting $\Sigma_Z$ wouldn't be either. Am I wrong? Is it at all possible to compute $\Sigma_Z$ from the information I have? If not, is there some way to approximate it?

EDIT: I think it isn't correct to write:

$$ \Sigma_Z = \Sigma_X + \Sigma_Y + 2 \cov(X, Y) $$

as $\cov(X,Y) \ne \cov(Y,X)$, but instead $\cov(X,Y) = \cov(Y,X)^T$.

Rewriting the equation for $\Sigma_Z$ to:

\begin{align} \Sigma_Z {}={}& \Sigma_X + \Sigma_Y + \cov(X, Y) + \cov(Y, X) \\ {}={}& \Sigma_X + \Sigma_Y + \cov(X, Y) + \cov(X, Y)^T \end{align}

I can now prove that indeed $\Sigma_Z$ is guaranteed to be symmetric. As antkam points out in the chat below, it isn't necessary that $\cov(X, Y) + \cov(X, Y)^T$ be positive semi-definite for $\Sigma_Z$ to be positive semi-definite.

  • $\begingroup$ Wait... what exactly is your question? I thought you're trying to calculate $\Sigma_Z = cov(Z,Z)$? Of course this matrix is symmetric and psd. Why would you need to prove this? $\endgroup$ – antkam May 9 '19 at 14:42
  • $\begingroup$ Of course $\Sigma_Z$ is symmetric and PSD, which is why I was concerned about my above definition of $\Sigma_Z$ . I saw this cannot possibly be right as I wrote it, hence my question here :) I understand your answer below and think you are correct. Now I'm trying to fit this into my algorithm (obtaining $P_{XY}$ is another step I have to solve, but unrelated to the question here). I'll upvote/accept your answer once I've got a good feeling I understand the full picture :) $\endgroup$ – lcbuai May 9 '19 at 14:49
  • $\begingroup$ hahaha, ok I see now. so your OP actually contains two errors: (1) the equation $\Sigma_Z = ...$ (which I did not address) and (2) the equation $cov(X,Y)=...$ (which I did address). anyway, you can calculate $cov(X,Y)$ using my method and then $cov(Y,X)$ is just the transpose. $\endgroup$ – antkam May 9 '19 at 14:53
  • $\begingroup$ BTW, in general if matrices $A = B + C + D$ and $A,B,C$ are symmetric and psd, that implies $D$ is symmetric but does not imply $D$ is also psd. (E.g. $A = B=C = I$, the identity, then $D = -I$.) I don't know offhand whether $cov(X,Y) + cov(Y,X)$ is psd, but perhaps I'm just missing something obvious. $\endgroup$ – antkam May 9 '19 at 14:56
  • $\begingroup$ Thanks for the pointer. Perhaps it isn't the case that $cov(X, Y) + cov(Y, X)$ is PSD then. $\endgroup$ – lcbuai May 9 '19 at 15:01

Your original intuition does not work, i.e., $cov(X, Y) \neq P_{XY} \sqrt{\Sigma_X} \sqrt{\Sigma_Y}$, because the definition of $corr(X_i, Y_j)$ involves scalar divisions, which cannot be reversed like this. For one thing, you have three matrices, so how do you pick the order of multiplication?

Instead, I would go back to the per-entry definitions:

$$[cov(X,Y)]_{ij} = cov(X_i, Y_j) = corr(X_i, Y_j) \ \sigma(X_i) \ \sigma(Y_j) = [P_{XY}]_{ij} \sqrt{ [\Sigma_X]_{ii}} \sqrt{[\Sigma_Y]_{jj}}$$

As you can see, the non-diagonal entries of $\Sigma_X, \Sigma_Y$ don't enter the calculations at all. So I'd think you cannot write this in terms of standard matrix multiplications.

However, if you have a $Diag$ function that keeps the diagonal entries but zeroes out all the non-diagonal entries, then you can write:

$$Cov(X,Y) = \sqrt{Diag(\Sigma_X)} \ P_{XY} \ \sqrt{Diag(\Sigma_Y)}$$

The multiplication is in that order because left-multiplying by a diagonal matrix scales the rows and right-multiplying scales the columns. (I know some typical software packages have a similar function although in the case of python np.diag you actually have to call it twice, so be careful!)

| cite | improve this answer | |
  • $\begingroup$ Actually, I initially implemented $cov(X,Y) = diag(P_{XY})\sqrt{diag(\Sigma_X)}\sqrt{diag(\Sigma_Y)}$ which is close to your suggestion, but realized experimentally that this constrained $cov(X,Y)$ to be axis-aligned (if I e.g. look at a covariance-ellipse plot). Need to think about this some more. Thanks! $\endgroup$ – lcbuai May 9 '19 at 14:00
  • $\begingroup$ The product $diag(.) diag(..) diag(...)$ would still be diagonal, which $cov(X,Y)$ isn't (necessarily). Anyway, if you have further questions, feel free to comment again. You do understand the per-entry equations, right? In the worst (most tedious) case you can always write a loop to calculate entry-by-entry... $\endgroup$ – antkam May 9 '19 at 14:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.