Proof of Convexity Is the function $Trace(AX^TBX)$ a convex function in $X$ or not ? Here, $X$ is a rectangular matrix and $A,B$ are square, symmetric, p.s.d matrices. The entries in $X,A,B$ are real valued.
 A: One easy way to show is $trace(AX^{T}BX)=vec(X)^{T}(A\otimes B)vec(X)$. This is a convex quadratic function, since $A$ and $B$ are psd and hence their kronecker product also will be. 
A: $\def\Mat{\operatorname{Mat}}\def\tr{\operatorname{trace}}$Let $f\colon \Mat_n(\mathbb R) \to \mathbb R$ the function in question, that is $f(X) = \tr(AX^tBX)$. We have for $X, H \in \Mat_n(\mathbb R)$: 
\begin{align*}
  f(X+H) - f(X) &= \tr(AH^tBX) + \tr(AX^tBH) + \tr(AH^tBH)\\
     &= 2\tr(AH^tBX) + \tr(AH^tBH)
\end{align*}
So $f'(X)H = 2\tr(AH^tBX)$ and $f''(X)[H,K] = 2\tr(AH^tBK)$. Now let $E_{ij}$ denote the matrix which has exactly one non zero entry, a 1 at $(i,j)$, then
\begin{align*}
  f''(X)[E_{ij},E_{kl}] &= 2\tr(AE_{ji}BE_{kl})\\
          &= 2\tr(Ab_{ik}E_{jl})\\
          &= 2b_{ik}\tr(AE_{jl})\\
          &= 2b_{ik}a_{jl}
\end{align*}
So, the representing matrix of $f''(X)$ in the standard basis of $\Mat_n(\mathbb R)$ is $2(A \otimes B)$ (Kronecker product). As this is positive definite, $f$ is convex.
A: Let $X_0, \, X_1$ be suitable rectangular matrices. Consider the function $t \mapsto trace(A(X_0^T + tX_1)B(X_0 + tX_1)) = \alpha t^2 + \beta t + \gamma$. All you need to do is compute $\alpha$ and show that it is non-negative. 
