Derivative of $f(A) = \|A x\|^2$ with respect to the Matrix Suppose I have $A \in \mathbb{R}^{n^2}$ and $x \in \mathbb{R}^n$ where $A$ is interpreted as a matrix. We can define $f(A) = ||A x||^2$ for some constant $x$.
What is the derivative of $f$, written as matrix? 
I have tried flattening out the matrix and using usual vector calculus and I got $f'(A)_{i j} = 2 x_i e_j^T A x$ where $e_j$ is the $j$th standard basis vector.
Is this correct? Is there a faster way to do this? Can the answer be written as a matrix expression?
 A: Let $g(x) = \|x\|^2$, then by looking at the linear terms of $h$ in $g(x+h)-g(x)$, we see that $Dg(x)(h) = 2 x^T h$.
If we let $L(A) = Ax$, we see that since $L$ is linear that we have
$DL(A)(H) = Hx$.
Using the composition rule, we have
$Df(A)(H)=D(g \circ L)(A)(H) = Dg(L(A))DL(A)(H)$, and expanding this gives
$Df(A)(H) = 2 (Ax)^T Hx = 2 x^T A^THx$.
Just to clarify, that is the map $H \mapsto 2 x^T A^THx$.
To check, the partial derivative with respect to $[A]_{ij}$ is
$Df(A)(e_i e_j^T) = 2x^T A^Te_i e_j^Tx = 2 x_j e_i^T Ax$.
A: I found an easier way to derive this after looking at related stack exchange questions.
$$f(A) = x^T A^T A x$$
$$f(A+\Delta) = f(A) + x^T A^T \Delta x + (x^T A^T \Delta x)^T + o(||\Delta||^2)$$
Therefore the derivative is the linear term
$$\Delta \mapsto x^T A^T \Delta x + (x^T A^T \Delta x)^T = 2 x^T A^T \Delta x$$  (since the result is one dimensional, the term and its transpose are equal)
Setting $\Delta = \delta_{i,j}$ we can put the result in component form
$$ 2 x^T A^T \delta_{i,j} x = 2 x^T A^T x_i e_j = 2(e_j^T Ax)^Tx_i$$
$$ = (2 A x x^T)_{i,j} $$
So in matrix form the derivative is $f'(A) = 2 A x x^T$
A: Note that
$$ f'(A)(H)=\lim_{t\to0}\frac{f(A+tH)-f(A)}{t}. $$
Easy calculation shows that
$$ f'(A)(H)=2x^TA^THx $$
from which you can get $f'(A)$.
