Partial derivative of $x_1^TAB^Tx_2$ Let $A$ and $B$ be orthogonal matrices, and $x_1$, $x_2$ be vectors. What are:
$$ \dfrac{\partial}{\partial x_1}\left( x_1^TAB^Tx_2 \right)$$
$$ \dfrac{\partial}{\partial x_2}\left( x_1^TAB^Tx_2 \right)$$
Are they
$$\dfrac{\partial}{\partial x_1}\left( x_1^TAB^Tx_2 \right) = AB^Tx_2 $$
$$\dfrac{\partial}{\partial x_2}\left( x_1^TAB^Tx_2 \right) = B A^Tx_1 $$
 A: Let
$$f ({\rm x}, {\rm y}) := {\rm x}^\top {\rm A} \, {\rm B}^\top {\rm y} = \langle {\rm x} , {\rm A} \,{\rm B}^\top {\rm y} \rangle = \langle {\rm B} \,{\rm A}^\top {\rm x}, {\rm y} \rangle$$
Hence,
$$\begin{aligned} \nabla_{{\rm x}} f ({\rm x}, {\rm y}) &= \color{blue}{{\rm A} \,{\rm B}^\top {\rm y}} \\ \nabla_{{\rm y}} f ({\rm x}, {\rm y}) &= \color{blue}{{\rm B} \,{\rm A}^\top {\rm x}}  \end{aligned}$$
A: Yes, with the convention that all results are column vectors, then what you have written is true. In fact, for any constant matrix $A$ and constant column vector $b$,
$$ \frac{\partial}{\partial x} x^T A b = Ab $$
Thus, we have
$$ \frac{\partial}{\partial x_1} x_1^T AB^T x_2 = \frac{\partial}{\partial x_1} x_1^T (AB^T) x_2 = AB^T x_2 $$
Since $ x_1^T AB^T x_2 $ is a scalar,
$$ x_1^T AB^T x_2 = (x_1^T AB^T x_2)^T = x_2^T BA^T x_1 $$
and so
$$ \frac{\partial}{\partial x_2} x_1^T AB^T x_2 = \frac{\partial}{\partial x_2} x_2^T BA^T x_1 = BA^T x_1 $$
Whether $A, B$ are orthogonal is irrelevant.
