My question is related to a tutorial I was reading.

The covariance matrix is a block matrix where $C_{xx}$ and $C_{yy}$ are within-set covariance matrices and $C_{xy} = C_{yx}^T$ are between-sets covariance matrices.

$$ \left[\begin{array}{r r} C_{xx} & C_{xy}\\ C_{yx} & C_{yy} \end{array}\right] $$

The tutorial says that the canonical correlations between $x$ and $y$ can be found by solving the eigenvalue equations

$$ C_{xx}^{-1}C_{xy}C_{yy}^{-1}C_{yx} \hat w_x = \rho^2 \hat w_x \\ C_{yy}^{-1}C_{yx}C_{xx}^{-1}C_{xy} \hat w_y = \rho^2 \hat w_y $$

where the eigenvalues are the squared canonical correlations and the eigenvectors and are the normalized canonical correlation basis vectors.

What I do not understand is how the eigenvalue equations are found by using the covariance matrix? Can someone please explain how we get those sets of equations?



Canonical correlation between two random vectors $X$ and $Y$ is obtained as the maximal correlation between $a^TX$ and $b^TY$, where the maximum is taken over vectors $a$ and $b$. We can assume without loss of generality that $a^T \Sigma_x a = b^T \Sigma_y b = 1$. Assume for simplicity also that $E(X) = 0$ and $E(y) = 0$. The the correlation between $a^TX$ and $b^TY$ is just $$E(a^T X)(b^TY) = E(a^T X)(Y^Tb) = a^T E(XY^T) b = a^T \Sigma_{xy}b.$$

You can now use either Lagrange duality or Cauchy-Schwarz. Say we use Lagrange duality. The optimal should maximize $$a^T \Sigma_{xy}b -\frac12\mu (a^T \Sigma_x a) - \frac12\lambda (b^T \Sigma_y b)$$ over $a$ and $b$. ($\frac12$s in the above are for convenience.) Differentiating with respect to $a$ and $b$ gives $$ \begin{align*} \Sigma_{xy} b - \mu \Sigma_x a &= 0 \\ \Sigma_{yx} a- \lambda \Sigma_y b &= 0, \end{align*} $$ Multiplying the first by $a^T$ and the second by $b^T$ and enforcing the constraints shows that $\mu = \lambda$. Then, if $\Sigma_x$ and $\Sigma_y$ are invertible you can solve the equations for what you have. That is, $$ \begin{align*} \Sigma_x^{-1} \Sigma_{xy} b - \mu a &= 0 \\ \Sigma_y^{-1} \Sigma_{yx} a- \mu b &= 0, \end{align*} $$ implying $$ \begin{align*} \frac{1}{\mu} \Sigma_x^{-1} \Sigma_{xy} \Sigma_y^{-1} \Sigma_{yx} a - \mu a &= 0 \end{align*} $$

  • $\begingroup$ Thanks for the derivation. Where does the $\mu$ go? I'm confused because in the final answer of $a$, there is no $\mu$ $\endgroup$ – xiaohan2012 Apr 24 '18 at 6:15
  • $\begingroup$ Ah, I just found this transformed problem is equivalent to generalized eigenvalue problem. Maybe you can add it as well. $\endgroup$ – xiaohan2012 Apr 24 '18 at 6:24

You can find the eigenvalue equation from the CCA's Lagrangian form. Hope it help.

PS: I don't know how to write latex in reply:(

  • $\begingroup$ Hi Dylan, you seem to be new here. For some basic information about writing math at this site see e.g. here, here, here and here. It's not always needed if answers are in the form of hints like this one is, but you may find that using LaTeX increases the readability of your answers and makes them easier to understand! $\endgroup$ – Tom Oldfield Dec 20 '12 at 4:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.