0
$\begingroup$

Is there a trick to matrix differentiation for some function $\nabla_xf(x)$ where $f(x)=x^TAx$ for example and $x$ is just a variable of vectors $x=[x_1 \;x_2,...,\:x_n]^{T}\in \mathbb{R}^n$ and $A \in \mathbb{R}^{n\times n}$ ? I know that for $Ax$ it's just $A$ but I'm wondering how to account for the fact that we have $x$ on both sides? Does the product rule apply? How would it even work here?

$\endgroup$
1
$\begingroup$

We can use this property:

\begin{equation} \begin{split} f & = x^TAy = y^TA^Tx \\ df &= dx^T(Ay) + (x^TA)dy = dy^T(A^Tx) + (y^TA^T)dx\\ & = (x^TA)dy + (y^TA^T)dx \\ \end{split} \end{equation}

Now for your expression, we just put $x=y$ and we differentiate

\begin{equation} \begin{split} g & = x^TAx \\ dg &= (x^TA)dx + (x^TA^T)dx\\ & = x^T(A + A^T)dx \\ \frac{dg}{dx} &= x^T(A + A^T) \end{split} \end{equation}

Depending on your preferred Layout convention, the derivative could be either

$$\frac{d(x^TAx)}{dx} = x^T(A + A^T)$$

or

$$\frac{d(x^TAx)}{dx} = (A + A^T)x$$

$\endgroup$
0
$\begingroup$

Yes. Write $g(x,y) = x^TAy$, note that $f(x)=g(x,x)$, and compute $$Df(x) = D_xg(x,x) + D_yg(x,x).$$You know that $$D_xg(x,y)(v) = v^TAy\quad\mbox{and}\quad D_yg(x,y)(v)=x^TAv.$$So $$Df(x)(v) = v^TAx + x^TAv = x^TA^Tv + x^TAv = x^T(A+A^T)v = ((A+A^T)x)^Tv.$$This means that $$\nabla f(x) = (A+A^T)x,$$in view of the characterization $Df(x)(v) = (\nabla f(x))^Tv$.

$\endgroup$
0
$\begingroup$

To differentiate expressions involving matrices, you can also go back to the definition of the derivative and compute $f(x+h)$. \begin{align*} f(x+h) &= (x+h)^T A (x+h) \\ &= x^T A x + x^T A h + h^T A x + h^T A h \\ &= f(x) + x^T A h + ( h^T A x )^T + O(\|h\|^2) \\ &= f(x) + x^T A h + x^T A^T h + O(\|h\|^2) \\ &= f(x) + x^T (A + A^T) h + O(\|h\|^2) \end{align*} The derivative of $f$ is the linear map $h \mapsto x^T (A + A^T) h$ or, equivalently, the row vector $x^T (A + A^T)$, or the column vector $(A^T + A) x$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.