My question is related to a tutorial I was reading.

The covariance matrix is a block matrix where $C_{xx}$ and $C_{yy}$ are within-set covariance matrices and $C_{xy} = C_{yx}^T$ are between-sets covariance matrices.

$$ \left[\begin{array}{r r} C_{xx} & C_{xy}\\ C_{yx} & C_{yy} \end{array}\right] $$

The tutorial says that the canonical correlations between $x$ and $y$ can be found by solving the eigenvalue equations

$$ C_{xx}^{-1}C_{xy}C_{yy}^{-1}C_{yx} \hat w_x = \rho^2 \hat w_x \\ C_{yy}^{-1}C_{yx}C_{xx}^{-1}C_{xy} \hat w_y = \rho^2 \hat w_y $$

where the eigenvalues are the squared canonical correlations and the eigenvectors and are the normalized canonical correlation basis vectors.

What I do not understand is how the eigenvalue equations are found by using the covariance matrix? Can someone please explain how we get those sets of equations?



2 Answers 2


Canonical correlation between two random vectors $X$ and $Y$ is obtained as the maximal correlation between $a^TX$ and $b^TY$, where the maximum is taken over vectors $a$ and $b$. We can assume without loss of generality that $a^T \Sigma_x a = b^T \Sigma_y b = 1$. Assume for simplicity also that $E(X) = 0$ and $E(y) = 0$. The the correlation between $a^TX$ and $b^TY$ is just $$E(a^T X)(b^TY) = E(a^T X)(Y^Tb) = a^T E(XY^T) b = a^T \Sigma_{xy}b.$$

You can now use either Lagrange duality or Cauchy-Schwarz. Say we use Lagrange duality. The optimal should maximize $$a^T \Sigma_{xy}b -\frac12\mu (a^T \Sigma_x a) - \frac12\lambda (b^T \Sigma_y b)$$ over $a$ and $b$. ($\frac12$s in the above are for convenience.) Differentiating with respect to $a$ and $b$ gives $$ \begin{align*} \Sigma_{xy} b - \mu \Sigma_x a &= 0 \\ \Sigma_{yx} a- \lambda \Sigma_y b &= 0, \end{align*} $$ Multiplying the first by $a^T$ and the second by $b^T$ and enforcing the constraints shows that $\mu = \lambda$. Then, if $\Sigma_x$ and $\Sigma_y$ are invertible you can solve the equations for what you have. That is, $$ \begin{align*} \Sigma_x^{-1} \Sigma_{xy} b - \mu a &= 0 \\ \Sigma_y^{-1} \Sigma_{yx} a- \mu b &= 0, \end{align*} $$ implying $$ \begin{align*} \frac{1}{\mu} \Sigma_x^{-1} \Sigma_{xy} \Sigma_y^{-1} \Sigma_{yx} a - \mu a &= 0 \end{align*} $$

  • $\begingroup$ Thanks for the derivation. Where does the $\mu$ go? I'm confused because in the final answer of $a$, there is no $\mu$ $\endgroup$ Apr 24, 2018 at 6:15
  • $\begingroup$ Ah, I just found this transformed problem is equivalent to generalized eigenvalue problem. Maybe you can add it as well. $\endgroup$ Apr 24, 2018 at 6:24

You can find the eigenvalue equation from the CCA's Lagrangian form. Hope it help.

PS: I don't know how to write latex in reply:(

  • $\begingroup$ Hi Dylan, you seem to be new here. For some basic information about writing math at this site see e.g. here, here, here and here. It's not always needed if answers are in the form of hints like this one is, but you may find that using LaTeX increases the readability of your answers and makes them easier to understand! $\endgroup$ Dec 20, 2012 at 4:24

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .