What is the relation between $\det(A^TA)$ and $\det(AA^T)$ for an $m\times n$ matrix $A$? Let $A$ be an $m\times n$ real matrix, and $\det(\cdot)$ the determinant. It is rather trivial that when $A$ is of size $1\times 1$, then $\det(A^TA)=\det(AA^T)$. 

In general, what is the relation between $\det(A^TA)$ and $\det(AA^T)$?

 A: Case 1: $A$ is a square matrix
If $A$ is a square matrix, then $\det(A^T)=\det(A)$, so
$$
\det(AA^T)=\det(A)\det(A^T)=(\det(A))^2
$$
and the same holds for $\det(A^TA)$.
Case 2: $A$ is not a square matrix
Suppose $A$ is $m\times n$, with $m\ne n$. If $m>n$, then the rank of $AA^T$ is at most $n$ and therefore $\det(AA^T)=0$. If $A$ has rank $n$, then $A^TA$ has rank $n$ (see later), so $\det(A^TA)\ne0$. Similarly if $m<n$.
Thus, the equality $\det(AA^T)=\det(A^TA)$ only holds if and only if the rank of $A$ is strictly less than $\min\{m,n\}$.
Exercise
The rank of $A$ equals the rank of $AA^T$ and of $A^TA$.
A: Summary: If $n \neq m$, one will be $0$, while the other will be non-zero iff the rows or the columns of $A$ are linearly independent.  If $n = m$, then the determinants are equal and both will be non-zero iff $A$ is invertible.
For parts A, B and C below, assume $m \neq n$.
A) Proof of "one will be $0$": 
$A B$'s columns are a linear combination of the columns of $A$. Thus if $A B$ has more columns than $A$, then the columns of $A B$ will not be linearly independent.
In $A A^T$ and $A^T A$, one of these will have more columns than $A$ or $A^T$ respectively.  Thus for one of these the columns will not be linearly independent.  Now both of these are square matrices, thus one of these will be singular, thus $\det(A A^T) = 0$ or $\det(A^T A) = 0$.
B) Proof of "the other will be non-zero only if the rows or the columns of $A$ are linearly independent", i.e. proof of "the other will be zero if the rows and the columns of $A$ are not linearly independent":
Now $A (c_1 {\bf x} + c_2 {\bf y}) = c_1 (A {\bf x}) + c_2 (A {\bf y})$, more generally a matrix times a linear combination of column vectors is a linear combination of the matrix times the column vectors.
The $i$-th column of $A B$'s columns will be $A$ times the $i$-th column of $B$.
Thus if $i$-th column of $B$ is a linear combination of $B$'s other columns, then the $i$-th column of $A B$ will be a linear combination of $A B$'s other columns.
Thus if the columns of $B$ are not linearly independent, then the columns of $A B$ will not be linearly independent.  
If the columns and rows of $A$ are not linearly independent, then the columns of $A$ and the columns of $A^T$ will not be linearly independent, thus the columns of $A A^T$ and $A^T A$ will not be linearly independent, thus both are singular, thus $\det(A A^T) = 0 = \det(A^T A)$.
C) Proof of "the other will be non-zero if the rows or the columns of $A$ are linearly independent", i.e. if the other is 0, then the rows and the columns of $A$ are not linearly independent.
The other is zero iff $\det(A^T A) = 0 = \det(A A^T) = 0$.
Assume that $\det(A^T A) = 0$, thus $A^T A$ is singular, thus there exists an non zero ${\bf x}$ such that $A^T A {\bf x} = {\bf 0}$.
Let ${\bf b} = A {\bf x}$, thus ${\bf b}^T = {\bf x}^T A^T$, thus ${\bf x}^T A^T A {\bf x} = {\bf b}^T {\bf b} = \|{\bf b}\|^2$. 
But  ${\bf x}^T A^T A {\bf x} = {\bf x}^T {\bf 0} = 0$.
Now $\|{\bf b}\|^2 = 0$ iff ${\bf b} = {\bf 0}$ for any vector ${\bf b}$.
Thus $A {\bf x} = {\bf 0}$ where $x$ is non-zero.
Thus the columns of $A$ are not linearly independent.
Similarly if $\det(A A^T) = 0$, then the columns of $A^T$ are not linearly independent, which implies the rows of $A$ are not linearly independent.
Thus if $\det(A^T A) = 0 = \det(A A^T) = 0$, then the rows and the columns of $A$ are not linearly independent.
D) Proof of "If $n = m$, then the determinants are equal and both will be non-zero iff $A$ is invertible."
Assume $m = n$, thus $A$ and $A^T$ are square matrices, thus $\det(A A^T) = \det(A) \det(A^T) = \det(A)\det(A) = \det(A)^2$.
Similarly $\det(A^T A) = \det(A)^2$.
Now $\det(A^T A) = 0$ iff $\det(A)^2 = 0$ iff $\det(A) = 0$ iff $A$ is singular.
A: The statement isn't true for generic rectangular matrices $m\neq n$ even though both expressions $\det(AA^T)$, $\det (A^T T)$ are well-defined.
For example, if $A$ is the row containing the entries $(a,b)$, then $AA^T$ is the $1\times 1$ matrix with the entry $a^2+b^2$ whose determinant is $a^2+b^2$.
On the other hand, $A^TA$ is the $2\times 2$ matrix with the rows $(a^2,ab)$, $(ab,b^2)$ whose determinant is zero. Not surprising because the rows aren't linearly independent. Note that the terms of this determinant, $a^2 b^2$, don't even have the same "units" as the terms in the first determinant. The vanishing of the "larger" among the two matrices holds in general.
For $m=n$, $\det A=\det A^T$ is well-defined and one may use
$$\det(AA^T)=\det(A^T A) = \det(A)\det(A^T)=[\det(A)]^2$$
so the identity holds.
A: For $n\times n$ matrices $A,B$, you have that $\det(AB)=\det(A)\det(B)=\det(B)\det(A)=\det(BA)$.  However, for non-square matrices, without being able to take the determinant of the individual terms, you cannot do the switching in this way.
However, by looking at characteristic polynomials, you can say something quite good about how $AB$ and $BA$ relate. Write $p_A(t)=\det(tI-A)$, so the characteristic polynomial can be thought of as an extension of determinant. Then $p_{AB}(t)$ and $p_{BA}(t)$ will differ only by a power of $t$ (whatever power is required to make the polynomials of the same degree).  In particular, they share they same eigenvalues, and they are of the same multiplicity, with the exception of $0$.   Thus, while one of $AB$ and $BA$ will have zero determinant, they are still very closely related.
