Vector Differentiation and Matrix Multiplication For a homework, we're supposed to solve an optimization problem, and in this context, I have a term of the form $$f\left( x\right):= x^{T}A^{T}Ax,$$
where $A\in\mathbb R^{m\times n}$, $x\in \mathbb R^{n}$, $b\in \mathbb R^{m}$.
I now want to differentiate the vector-valued function $f$ with respect to $x$. So:
$$\nabla_{x}f\left(x\right) = \nabla_{x}\left( x^{T}A^{T}Ax \right) = \left(\nabla_{x}x^{T}\right)A^{T}Ax + x^{T}A^{T}A\left(\nabla_{x} x\right)$$
Now, in my understanding, $\nabla_{x}x^{T}$ and $\nabla_{x}x$ are just the identity matrix, and so I obtain:
$$\nabla_{x}f\left(x\right) = \nabla_{x}\left( x^{T}A^{T}Ax \right) = A^{T}Ax + x^{T}A^{T}A$$
This is troubling me, because $A^{T}Ax\in\mathbb R^{n\times 1}$ and $x^{T}A^{T}A\in\mathbb R^{1\times n}$, which would imply that the last Eq. is ill-defined. Does anybody see where the mistake lies?
 A: If you want $\nabla_x f$ to be a column vector, then as a consequence you get that $\nabla_x x$ is not the identity matrix: instead, it is the transpose operator for vectors on its left. In other words, you have $v(\nabla_xX)=v^T$ for any row vector $v$.
Why is this?
Well, if $a$ and $x$ are column vectors, where $a$ is constant, then you expect that $\nabla_x(a^Tx)=a$. But since $a^T$ is constant, you can take out the $a^T$ and obtain $a^T(\nabla_xx)=a$.
In your case, note that $x^TA^TA$ is a row vector, so you have $(x^TA^TA)(\nabla_xx)=(x^TA^TA)^T=A^TAx$.

Other possible aproach I've seen used by greg (I hoped to see him around, but he hasn't showed up yet) is to use differential notation keeping in mind that
$$\nabla_xf=y
\hspace{10mm}\text{if and only if}\hspace{10mm}
df=dx^Ty$$
if you want the gradient to be a column vector.
Then your calculation is as follows
\begin{aligned}
  f &= x^TA^TAx \\
  df
  &= dx^TA^TAx + x^TA^TAdx \\
  &= dx^TA^TAx + (x^TA^TAdx)^T
    & \text{since $x^TA^TAdx$ is a scalar} \\
  &= dx^TA^TAx + dx^TA^TAx \\
  &= 2dx^TA^TAx.
\end{aligned}
Thus $\nabla_xf=2A^TAx$.
