Eigenvalue decomposition of block covariance matrix for Canonical Correlation Analysis (CCA)

Edited:

My question is related to a tutorial I was reading.

The covariance matrix is a block matrix where $C_{xx}$ and $C_{yy}$ are within-set covariance matrices and $C_{xy} = C_{yx}^T$ are between-sets covariance matrices.

$$\left[\begin{array}{r r} C_{xx} & C_{xy}\\ C_{yx} & C_{yy} \end{array}\right]$$

The tutorial says that the cannonical correlations between $x$ and $y$ can be found by solving the eigenvalue equations

$$C_{xx}^{-1}C_{xy}C_{yy}^{-1}C_{yx} \hat w_x = \rho^2 \hat w_x \\ C_{yy}^{-1}C_{yx}C_{xx}^{-1}C_{xy} \hat w_y = \rho^2 \hat w_y$$

where the eigenvalues are the squared canonical correlations and the eigenvectors and are the normalized canonical correlation basis vectors.

What I do not understand is how the eigenvalue equations are found by using the covariance matrix? Can someone please explain how we get those sets of equations?

Thanks.

-

2 Answers

Canonical correlation between two random vectors $X$ and $Y$ is obtained as the maximal correlation between $a^TX$ and $b^TY$, where the maximum is taken over vectors $a$ and $b$. We can assume without loss of generality that $a^T \Sigma_x a = b^T \Sigma_y b = 1$. Assume for simplicity also that $E(X) = 0$ and $E(y) = 0$. The the correlation between $a^TX$ and $b^TY$ is just $$E(a^T X)(b^TY) = E(a^T X)(Y^Tb) = a^T E(XY^T) b = a^T \Sigma_{xy}b.$$

You can now use either Lagrange duality or Cauchy-Schwarz. Say we use Lagrange duality. The optimal should maximize $$a^T \Sigma_{xy}b -\frac12\mu (a^T \Sigma_x a) - \frac12\lambda (b^T \Sigma_y b)$$ over $a$ and $b$. ($\frac12$s in the above are for convenience.) Differentiating with respect to $a$ and $b$ gives \begin{align*} \Sigma_{xy} b - \mu \Sigma_x a &= 0 \\ \Sigma_{yx} a- \lambda \Sigma_y b &= 0, \end{align*} Multiplying the first by $a^T$ and the second by $b^T$ and enforcing the constraints shows that $\mu = \lambda$. Then, if $\Sigma_x$ and $\Sigma_y$ are invertible you can solve the equations for what you have. That is, \begin{align*} \Sigma_x^{-1} \Sigma_{xy} b - \mu a &= 0 \\ \Sigma_y^{-1} \Sigma_{yx} a- \mu b &= 0, \end{align*} implying \begin{align*} \frac{1}{\mu} \Sigma_x^{-1} \Sigma_{xy} \Sigma_y^{-1} \Sigma_{yx} a - \mu a &= 0 \end{align*}

-

You can find the eigenvalue equation from the CCA's Lagrangian form. Hope it help.

PS: I don't know how to write latex in reply:(

-
Hi Dylan, you seem to be new here. For some basic information about writing math at this site see e.g. here, here, here and here. It's not always needed if answers are in the form of hints like this one is, but you may find that using LaTeX increases the readability of your answers and makes them easier to understand! –  Tom Oldfield Dec 20 '12 at 4:24