# Maximizing generalized Rayleigh quotient with constraints

Let $$A$$ and $$B$$ be $$n \times n$$ symmetric matrices with real entries and let $$k \geq 2$$ be an integer.

I want to find the maximum of $$\frac{\sum_{i=1}^k X_i^{\mathrm T}\,A\,X_i}{\sum_{i=1}^k X_i^{\mathrm T}\,B\,X_i}$$ where the unknowns $$X_i$$ are vectors, under the constraints that the $$X_i$$ are normalized ($$X_i^{\mathrm T}\,X_i = 1$$ for all $$i$$) and orthogonal ($$X^{\mathrm T}_i\,X_j = 0$$ if $$i \neq j$$).

The problem I am facing (and that is why I add the orthogonal constraints) is that, classically, one will get the eigenvectors of $$B^{-1}\,A$$ (see here and here) but there is no reason that these eigenvectors will be orthogonal (since $$B^{-1}\,A$$ has no reason to be symmetric). Maybe I am missing something...

Assuming that $$B$$ p.s.d., note that $$B^{-\frac 1 2} A B^{-\frac 1 2}$$ is symmetric, so it has $$n$$ eigenvectors $$y_1,\dots,y_n$$ that are orthogonal. Then if you consider $$x_i=B^{-\frac 1 2}y_i$$, then $$x_1,\dots x_n$$ are $$n$$ eigenvectors of $$B^{-1}A$$: $$B^{-1}A x_i = B^{-\frac 1 2} \left ( B^{-\frac 1 2} A B^{-\frac 1 2}\right)B^{\frac 1 2}x_i = \lambda_i B^{-\frac 1 2}y_i=\lambda_i x_i$$ Those eigenvectors are not orthogonal w.r.t. the regular inner product. That's because: $$x_i^Tx_j = y_i^TB^{-\frac 1 2}B^{-\frac 1 2}y_j=y_i^TB^{-1}y_j$$ But they are orthogonal under the inner product defined by $$B$$: $$x_i^TBx_j = x_i^TB^{\frac 1 2}B^{\frac 1 2}x_j=y_i^Ty_j$$
• Thanks! So, if I understand correctly, in PCA, we get eigenvectors that are orthogonal not with the regular inner product but with the inner product defined by $B$? May 14 at 5:03
• For the regular PCA, matrix $A$ is the covariance matrix of your (mean-centered) data set, and matrix $B$ is the identity. So your eigenvectors are orthogonal with the regular inner product. May 14 at 13:44