How does this problem specifically correspond to a generalized eigenvalue problem? Given two symmetrical square matrices $A,B \in \mathbb R^{n\times n}$ and a rectangular matrix $W\in \mathbb R^{n,k}$. I want to maximize
$$\max_{W} \frac{\det (W^TAW)}{\det(W^TBW)}.$$
I read that the $W$ the maximizes this consists of the generalized eigenvectors that correspond to the largest eigenvalue in $$Aw_i=\lambda_iBw_i.$$
I know that this is true if we want to maximize $\frac{w^TAw}{w^TBw}$, $w \in \mathbb R^n$. However, I am not sure how this corresponds to the first optimization problem where we have a rectangular matrix $W$ and the determinant involved. Is there a way I can see this?
 A: According to matrix calculus the derivative of what you want to maximize with respect to $W$ is
$$ \frac{2 |W^TAW| |W^TBW| \left( AW (W^TAW)^{-1} - BW (W^TBW)^{-1} \right)}{|W^TBW|^2} = 0 $$
where $|\cdot|$ represents the determinant. So you need
$$ AW = BW (W^TBW)^{-1} (W^TAW) \tag{1} $$
It is easy to see that the solution satisfies $AW = BWX$ for some $X \in \mathbb{R}^{k \times k}$ and also
$$ \frac{|W^TAW|}{|W^TBW|} = |X| $$
Since the determinant is the product of eigenvalues you get your result.
However, one needs to take into the sign of the eigenvalues. For example, let the eigenvalues be $-1, -2, 1$. For $k = 1$, the maximizer is $w_3$, while for $k=2$, $W=[w_1 ~~ w_2]$ is the maximizer.
Edit: The specific equation in matrix calculus wiki is in the "Identities: scalar-by-matrix" table one previous from last row: "A is not a function of X, X is non-square, A is symmetric".
If $AW=BWX$ then from equation $(1)$
$$\begin{align}
AW &= BW (W^TBW)^{-1} (W^TAW)\\
&= BW (W^TBW)^{-1} W^T(AW)\leftarrow\text{Plug in here}\\
&= BW (W^TBW)^{-1} W^T(BWX)\\
&= BW (W^TBW)^{-1} (W^TBW)X\\
&= BWX
\end{align}$$
Let $X = T \tilde{X} T^{-1}$, then
$$\begin{align}
AW &= BW X\\
AW &= BW T \tilde{X} T^{-1}\\
AWT &= BW T \tilde{X}\\
A \tilde{W} &= B \tilde{W} \tilde{X}
\end{align}$$
where $\tilde{W} := WT$. So, without losing generality, we can assume that $X$ is in Jordan normal form. In particular, if $X$ is diagonal, then its elements correspond to the generalized eigenvalues of $(A,B)$. So, maximization of determinant can be done by selecting the eigenvalues that has maximum multiplication value. (If $X$ is upper triangular, then its diagonal elements correspond to the eigenvalues.)
Edit 2. Let $X(W) := (W^TBW)^{-1} (W^TAW)$. We want to solve
$$ \max_W |X(W)| $$
Therefore we need,
$$ \frac{\partial |X(W)|}{\partial W} = 2 |X(W)| \left( AW (W^TAW)^{-1} - BW (W^TBW)^{-1} \right) = 0$$
and consequently $W$ and $X(W)$ must satisfy
$$ AW = BW X(W) $$
