What is $\det (W^T A W)$ for $W$ not square? I am sure the answer to this question is out there, but I cannot find it, maybe because I don't know the correct torm for 'matrix multiplication from left and right'.
Consider a matrix $W \in \mathbb{R}^{m \times n}$ and a square matrix $A \in \mathbb{R}^{m \times m}$. What can we say about the determinant
$$\det (W^TAW)?$$
In the case that $W$ is also square, since the determinant then commutes, we have
$$\det (W^TAW) = \det (A WW^T) = \det (A) \det (WW^T) = \det(A) \det(W^TW).$$
Do we have a similar result when $W$ is not square? We can not expect the two last equalities on the left generally hold, since one of $\det (WW^T)$ and $ \det(W^TW)$ will be equal to $0$, but maybe it holds for the respective one that might have full rank?
Edit:
Maybe this works indeed, using singular value decomposition:
Let $W = U \Sigma V$ and assume $W$ has full rank, and assume that $m<n$, such that $U$ is an orthogonal $m\times m$ matrix, $V$ is an orthogonal $n \times n$ matrix, and $\Sigma \in \mathbb{R}^{m\times n}$ is of the form $\begin{pmatrix}D & 0\end{pmatrix}$ with $D$ diagonal.
Then $$\det (W^T AW) = \det (V^T \Sigma^T U^T A U \Sigma V) = \det ( \Sigma^T U^T A U \Sigma VV^T) = \det ( \Sigma^T U^T A U\Sigma)$$
Bot now if we denote $U^TAU=B$, then $$\Sigma^T B \Sigma = \begin{pmatrix}D \\ 0\end{pmatrix} B \begin{pmatrix}D & 0\end{pmatrix} = DBD $$. But then
$$\det(\Sigma^T U^T A U\Sigma) = \det (\Sigma^TB\Sigma) = \det (\Sigma^T\Sigma B) = \det(D^2) \det(B) = \det(D^2) \det(U^TAU) = \det(D^2) \det(A) =\det(WW^T)\det(A).$$
Is this correct?
second edit: the 'solution' above must contain an error, since we obviously have that the determinant is zero in the case that $m<n$...
 A: An analysis similar to yours works for the case where $m>n$. Let $U \Sigma V^T$ be the SVD of $W$, where $\Sigma$ has the form
$$
\Sigma = \pmatrix{D\\0}.
$$
Denote $B = U^TAU$. Partition $B$ into
$$
B = \pmatrix{B_{11} & B_{12}\\ B_{21} & B_{22}}, 
$$
where $B$ is square of size $n$. We have
$$
\Sigma^TB\Sigma = \pmatrix{D & 0}\pmatrix{B_{11} & B_{12}\\ B_{21} & B_{22}} \pmatrix{D \\ 0} = DB_{11}D.
$$
Thus, we have
$$
\det(W^TAW) = \det(\Sigma^TB\Sigma) = \det(DB_{11}D) = \det(B_{11})\det(D)^2\\
= \det(B_{11}) \det(W^TW). 
$$
This isn't as "nice" as the formula for the case of $m = n$, but perhaps you will find it interesting nevertheless.
A: I suggest to use the generalization of the Matrix determinant lemma:
$$
\det\left(\mathbf{Q} + \mathbf{UAV}^\textsf{T}\right) = \det\left(\mathbf{A}^{-1} + \mathbf{V}^\textsf{T}\mathbf{Q}^{-1}\mathbf{U}\right)\det\left(\mathbf{A}\right)\det\left(\mathbf{Q}\right).
$$
with $\mathbf{Q} = \varepsilon {\mathbf{I}}_n$, $\mathbf{V}^\textsf{T} = \mathbf{U}^\textsf{T} = \mathbf{W}\in \mathbb C^{m\times n}$:
$$
\det\left(\varepsilon {\mathbf{I}}_n + \mathbf{W}^\textsf{T}\mathbf{AW}\right) = \det\left(\mathbf{A}^{-1} + \varepsilon^{-1} \mathbf{W}\mathbf{W}^\textsf{T}\right)\det\left(\mathbf{A}\right)\det\left(\mathbf{\varepsilon {\mathbf{I}}_n}\right).
$$
Note that here the determinants are calculated for matrices with different sizes.
Having this in mind we write
$$
\det\left(\varepsilon {\mathbf{I}}_n + \mathbf{W}^\textsf{T}\mathbf{AW}\right) = 
\varepsilon^{n-m}
\det\left(\varepsilon \mathbf{A}^{-1} + \mathbf{W}\mathbf{W}^\textsf{T}\right)\det\left(\mathbf{A}\right).
$$
Now let us tend $\varepsilon\to 0$. When $n>m$ we obtain
$$
\det\left(\mathbf{W}^\textsf{T}\mathbf{AW}\right) = 0.
$$
When $n=m$ we obtain
$$
\det\left(\mathbf{W}^\textsf{T}\mathbf{AW}\right) = \det\left( \mathbf{W}\mathbf{W}^\textsf{T}\right)\det\left(\mathbf{A}\right).
$$
When $m>n$ we obtain
$$
\det\left( \mathbf{W}^\textsf{T}\mathbf{AW}\right) = 
\det\left(\mathbf{A}\right)
\lim_{\varepsilon\to0}
\frac{
\det\left(\varepsilon \mathbf{A}^{-1} + \mathbf{W}\mathbf{W}^\textsf{T}\right)
}{\varepsilon^{m-n}}.
$$
I do not know if this expression is somehow helpful but we can use it to obtain a closed-form expression in the particular case of $m=n+1$ by differentiating the determinant using Jacobi formula:
$$
\det\left( \mathbf{W}^\textsf{T}\mathbf{AW}\right) = 
\det\left(\mathbf{A}\right)
{\rm tr}[ {\rm adj}(\mathbf{W}\mathbf{W}^\textsf{T}) \mathbf{A}^{-1} ]
={\rm tr}[ {\rm adj}(\mathbf{W}\mathbf{W}^\textsf{T}) \cdot {\rm adj}\mathbf{A} ]
.
$$
