Eigenvalue decomposition of block covariance matrix for Canonical Correlation Analysis (CCA)

Question

Edited:

My question is related to a tutorial I was reading.

The covariance matrix is a block matrix where $C_{xx}$ and $C_{yy}$ are within-set covariance matrices and $C_{xy} = C_{yx}^T$ are between-sets covariance matrices.

$$ \left[\begin{array}{r r} C_{xx} & C_{xy}\\ C_{yx} & C_{yy} \end{array}\right] $$

The tutorial says that the canonical correlations between $x$ and $y$ can be found by solving the eigenvalue equations

$$ C_{xx}^{-1}C_{xy}C_{yy}^{-1}C_{yx} \hat w_x = \rho^2 \hat w_x \\ C_{yy}^{-1}C_{yx}C_{xx}^{-1}C_{xy} \hat w_y = \rho^2 \hat w_y $$

where the eigenvalues are the squared canonical correlations and the eigenvectors and are the normalized canonical correlation basis vectors.

What I do not understand is how the eigenvalue equations are found by using the covariance matrix? Can someone please explain how we get those sets of equations?

Thanks.

Answer 1

Canonical correlation between two random vectors $X$ and $Y$ is obtained as the maximal correlation between $a^TX$ and $b^TY$, where the maximum is taken over vectors $a$ and $b$. We can assume without loss of generality that $a^T \Sigma_x a = b^T \Sigma_y b = 1$. Assume for simplicity also that $E(X) = 0$ and $E(y) = 0$. The the correlation between $a^TX$ and $b^TY$ is just $$E(a^T X)(b^TY) = E(a^T X)(Y^Tb) = a^T E(XY^T) b = a^T \Sigma_{xy}b.$$

You can now use either Lagrange duality or Cauchy-Schwarz. Say we use Lagrange duality. The optimal should maximize $$a^T \Sigma_{xy}b -\frac12\mu (a^T \Sigma_x a) - \frac12\lambda (b^T \Sigma_y b)$$ over $a$ and $b$. ($\frac12$s in the above are for convenience.) Differentiating with respect to $a$ and $b$ gives $$ \begin{align*} \Sigma_{xy} b - \mu \Sigma_x a &= 0 \\ \Sigma_{yx} a- \lambda \Sigma_y b &= 0, \end{align*} $$ Multiplying the first by $a^T$ and the second by $b^T$ and enforcing the constraints shows that $\mu = \lambda$. Then, if $\Sigma_x$ and $\Sigma_y$ are invertible you can solve the equations for what you have. That is, $$ \begin{align*} \Sigma_x^{-1} \Sigma_{xy} b - \mu a &= 0 \\ \Sigma_y^{-1} \Sigma_{yx} a- \mu b &= 0, \end{align*} $$ implying $$ \begin{align*} \frac{1}{\mu} \Sigma_x^{-1} \Sigma_{xy} \Sigma_y^{-1} \Sigma_{yx} a - \mu a &= 0 \end{align*} $$

Answer 2

You can find the eigenvalue equation from the CCA's Lagrangian form. Hope it help.

PS: I don't know how to write latex in reply:(