when does $\det(AB^T+BA^T)\le \det(AA^T+BB^T)$ hold? When does the following matrix inequality hold? 
$$\det(AB+B^TA^T)\le \det(AA^T+BB^T)$$
$A$ and $B$ are any real matrices.
My reply gives a counter example. The question is under what condition does that hold?
[Update]
The inequality actually I would like to prove is not the above one. It should be 

$$\det(AB^T+BA^T)\le \det(AA^T+BB^T)$$

I also changed the title. Thanks julien.
 A: I assume that $A,B$ are real $n\times m$ matrices. But the same proof gives the same result as below for complex matrices with the adjoint instead of the transpose. 

For every real $n\times m$ matrices $A,B$, we have
  $$
|\det (AB^T+BA^T)|\leq \det(AA^T+BB^T).
$$

Proof: if $x\in\ker (AA^T+BB^T)$, then $\|A^Tx\|^2+\|B^Tx\|^2=0$, whence $A^Tx=B^Tx=0$ and so $x\in \ker(AB^T+BA^T)$. In other words, if $\det(AA^T+BB^T)=0$, we do have $\det(AB^T+BA^T)=0$ and the inequality holds. So we will assume that $AA^T+BB^T$ is invertible from now on.
The matrix $C=(A-B)(A-B)^T$ is positive semidefinite. So
$$
0\leq (Cx,x)=((AA^T+BB^T)x,x)-((AB^T+BA^T)x,x)
$$
whence
$$
((AB^T+BA^T)x,x)\leq ((AA^T+BB^T)x,x).
$$
Changing $B$ for $-B$, we get altogether

$$
|((AB^T+BA^T)x,x)|\leq ((AA^T+BB^T)x,x).
$$

Denote $C:=AB^T+BA^T$ which is symmetric, and $D:=AA^T+BB^T$ which is positive definite and has a unique positive definite square root $D^{1/2}$. Now the inequality above entails, with $x=D^{-1/2}y$:
$$
|(D^{-1/2}CD^{-1/2}y,y)|\leq(y,y)=\|y\|^2\qquad\forall y\in\mathbb{R}^n.
$$ 
Since $T=D^{-1/2}CD^{-1/2}$ is symmetric, it is diagonalizable in an orthonormal basis and the above shows that its spectrum is contained in $[-1,1]$. A fortiori
$$
|\det (D^{-1/2}CD^{-1/2})|\leq 1\qquad \Rightarrow \qquad |\det C|\leq \det D.
$$
QED.
Note: the inequality you wanted initially requires that $A,B$ be square matrices of the same size. As you observed, the inequality is not true in general. The transpose was not at the right place.  It follows in particular from the inequality above that the inequality you first wanted holds when $A,B$ are both symmetric, or both skew-symmetric. Of course, it is true also if $AB=0$. One can add some more solutions. All $(A,B)$ such that the symmetric matrix $AB+B^TA^T$ is singular or has an odd number of negative eigenvalues. But I doubt there is a nice general characterization. Although I might be wrong.
A: I found a counter example by Matlab. I know that is cheap..... :-|
$A = 0.4582 \quad  0.1567  \quad  0.0519\\
     0.8204  \quad  0.9111  \quad  0.0512 \\
     0.9831  \quad  0.1020  \quad  0.6691$
$B = 0.8492  \quad  0.0900 \quad  0.0787\\
    0.8890  \quad  0.0126  \quad  0.7477\\
    0.7770  \quad  0.2320 \quad   0.0607$
f = @(A,B)(det(A*A'+B*B')-det(A*B+B'*A'))
f(A, B) gives -1.9356
A: As $AA^T+BB^T\succeq0$, if $(AA^T+BB^T)x=0$, we must have $A^Tx=B^Tx=0$ and $(AB^T+BA^T)x=0$ too. So, it suffices to prove the inequality when $AA^T+BB^T\succ0$.
Write $AA^T+BB^T=\pmatrix{A&B}\pmatrix{A^T\\ B^T}$ and $AB^T+BA^T=\pmatrix{A&B}\pmatrix{0&I\\ I&0}\pmatrix{A^T\\ B^T}$. Let $\pmatrix{A&B}=U\pmatrix{\Sigma&0}V^T$ be a singular value decomposition, where $\Sigma$ is a diagonal matrix. Since $AA^T+BB^T$ is invertible, so is $\Sigma$. Consequently, the inequality in question can be rewritten as
$$
\det\left[\pmatrix{I&0}\ \underbrace{V^T\pmatrix{0&I\\ I&0} V}_S\ \pmatrix{I\\ 0}\right]\le1.
$$
Hence it suffices to prove that for any symmetric square root $S$ of $I_{2n}$, its leading principal $n\times n$ minor is bounded above by $1$. This actually follows immediately from the interlacing inequality for eigenvalues of submatrices of Hermitian matrices, but we will offer another proof here.
Let $S=\pmatrix{X&Y^T\\ Y&Z}$, where $X$ and $Z$ are symmetric matrices of the same sizes. By comparing the first subblocks on both sides of $S^2=I_{2n}$, we get $X^TX+Y^TY=I$. Therefore $I\succeq X^TX\succeq 0$ and $\det(X)^2\le1$. QED.
The above proof has the merit that a similar argument (with $S$ being a skew-symmetric square root of $-I_{2n}$) can be used to show that

$\det(AB^T\color{red}{-}BA^T)\le \det(AA^T+BB^T)$.

