# Matrix differentiation with vector variables on either side of matrix

Is there a trick to matrix differentiation for some function $$\nabla_xf(x)$$ where $$f(x)=x^TAx$$ for example and $$x$$ is just a variable of vectors $$x=[x_1 \;x_2,...,\:x_n]^{T}\in \mathbb{R}^n$$ and $$A \in \mathbb{R}^{n\times n}$$ ? I know that for $$Ax$$ it's just $$A$$ but I'm wondering how to account for the fact that we have $$x$$ on both sides? Does the product rule apply? How would it even work here?

We can use this property:

$$\begin{equation} \begin{split} f & = x^TAy = y^TA^Tx \\ df &= dx^T(Ay) + (x^TA)dy = dy^T(A^Tx) + (y^TA^T)dx\\ & = (x^TA)dy + (y^TA^T)dx \\ \end{split} \end{equation}$$

Now for your expression, we just put $$x=y$$ and we differentiate

$$\begin{equation} \begin{split} g & = x^TAx \\ dg &= (x^TA)dx + (x^TA^T)dx\\ & = x^T(A + A^T)dx \\ \frac{dg}{dx} &= x^T(A + A^T) \end{split} \end{equation}$$

Depending on your preferred Layout convention, the derivative could be either

$$\frac{d(x^TAx)}{dx} = x^T(A + A^T)$$

or

$$\frac{d(x^TAx)}{dx} = (A + A^T)x$$

Yes. Write $$g(x,y) = x^TAy$$, note that $$f(x)=g(x,x)$$, and compute $$Df(x) = D_xg(x,x) + D_yg(x,x).$$You know that $$D_xg(x,y)(v) = v^TAy\quad\mbox{and}\quad D_yg(x,y)(v)=x^TAv.$$So $$Df(x)(v) = v^TAx + x^TAv = x^TA^Tv + x^TAv = x^T(A+A^T)v = ((A+A^T)x)^Tv.$$This means that $$\nabla f(x) = (A+A^T)x,$$in view of the characterization $$Df(x)(v) = (\nabla f(x))^Tv$$.

To differentiate expressions involving matrices, you can also go back to the definition of the derivative and compute $$f(x+h)$$. \begin{align*} f(x+h) &= (x+h)^T A (x+h) \\ &= x^T A x + x^T A h + h^T A x + h^T A h \\ &= f(x) + x^T A h + ( h^T A x )^T + O(\|h\|^2) \\ &= f(x) + x^T A h + x^T A^T h + O(\|h\|^2) \\ &= f(x) + x^T (A + A^T) h + O(\|h\|^2) \end{align*} The derivative of $$f$$ is the linear map $$h \mapsto x^T (A + A^T) h$$ or, equivalently, the row vector $$x^T (A + A^T)$$, or the column vector $$(A^T + A) x$$.