how to prove $\frac{\partial (Wx)^T A (Wx)}{\partial W} = 2AWxx^T$ I'm trying to evaluate $\frac{\partial (Wx)^T A (Wx)}{\partial W}$, where

*

*W is a $m \times n$ matrix

*x is a $n \times 1$ vector

*A is a $m \times m$ symmetric positive semi-definite matrix (e.g. a covariance matrix)

I did some calculations with simple concrete examples and it looks the answer can be $2AWxx^T$, but I am not sure how to prove it in general.
I am familiar with $\frac{\partial y^TAy}{\partial y} = 2Ay$ (e.g. $y=Wx$), but I am confused what $\frac{\partial Wx}{\partial W}$ would be like and how the chain rule would work with it.
 A: $
\def\o{{\tt1}}\def\p{\partial}
\def\L{\left}\def\R{\right}\def\LR#1{\L(#1\R)}
\def\trace#1{\operatorname{Tr}\LR{#1}}
\def\grad#1#2{\frac{\p #1}{\p #2}}
$Starting with the known gradient, use it to write the differential, then change the independent variable from $y\to W$, and recover the new gradient
$$\eqalign{
\grad{\LR{y^TAy}}{y} &= 2Ay \\
d\LR{y^TAy} &= 2Ay:dy \\
  &= 2A\LR{Wx}:\LR{dW\,x} \\
  &= 2AWxx^T:dW \\
\grad{\LR{y^TAy}}{W} &= 2AWxx^T \\
}$$
where a colon denotes the matrix inner product, i.e.
$$\eqalign{
A:B &= \sum_{i=1}^m\sum_{j=1}^n A_{ij}B_{ij} \;=\; \trace{AB^T} \\
A:A &= \big\|A\big\|^2_F \\
}$$
A: As you said, the derivative of the map $$f(y)= y^T A y$$ is the linear form
$$f^\prime(y)(h) = (2Ay)^Th$$ which is summarized using gradient notation as
$$\frac{\partial f}{\partial y} = \nabla_y f = 2Ay.$$
The map
$$\begin{array}{l|rcl}
g : & \mathcal M_{m,n}(\mathbb R) & \longrightarrow & M_{m,1}(\mathbb R)  \\
    & W & \longmapsto & Wx \end{array}$$ is linear. Therefore, its derivative at any point is itself. That is
$$g^\prime(W)(H) = Hx.$$
You are looking for the derivative of the map $h=f \circ g$. Using the chain rule, you get
$$\begin{aligned}h^\prime(W)(H) &= \left(f^\prime(g(W)) \cdot g^\prime(W)\right)(H)\\
&=(2AWx)^THx
\end{aligned}$$
Now, you have to make a decision. Either you stay with this result. That would be sufficient for numerical analysis. Or you want to get the result in terms of matrix calculus.
For each $W \in M_{m,n}(\mathbb R) $, $h^\prime(W)$ is a linear form on $M_{m,n}(\mathbb R) $ that can be defined as
$$\begin{aligned}
h^\prime(W)(H) &= \operatorname{tr}\left(\left(\frac{\partial h}{\partial W}\right)^T H\right)\\
&= \operatorname{tr}((2AWx)^THx)\\
&= \operatorname{tr}(x(2AWx)^TH)\\
&= \operatorname{tr}(2xx^TW^TAH)\\
&= \operatorname{tr}((2AWxx^T)^TH)
\end{aligned}$$
using the properties of the transpose and the cyclic property of the trace. That gives exactly the result you were looking for from the definition in matrix calculus of $\frac{\partial h}{\partial W}$ where $h$ takes real values.
Note: on my side, I don't use matrix calculus conventions. Those are difficult to remember, not always implemented in the same way in software packages (numerator vs. denominator conventions), and obfuscate the natural properties of the derivative.
