1
$\begingroup$

I was trying to take the gradient of $x^TAx$ i.e. $\nabla_xx^TAx$.

I did have one idea of how to do this which was expression $x^TAx$ as a double summation and then take the partial derivatives wrt to each $x_i$. However, it seemed a little ugly to me, and therefore, was wondering if there was a different maybe more cleaver/cleaner way to derive it.

The kind of idea I was thinking to apply was maybe the product rule of gradients:

$$\nabla_x(uF) = \nabla_xuF + u\nabla_xF$$

Where u is a scalar function and F is a vector field. (I got the above from the following OCW video).

However, I was not 100% sure how to apply it. The issue I had was that if I let $u = x^T$ and $Ax = F$, then F does correspond to a vector field because to start with, it is a valid vector. However, $u = x^T$ does not correspond correctly to a scalar function because its a vector. I guess maybe I am not using the most general version of the product rule? Is there a more general version of it such that it yields a nicer derivation for $\nabla_xx^TAx$?

$\endgroup$
3
  • $\begingroup$ Found a related question: math.stackexchange.com/questions/482742/… $\endgroup$ Nov 23, 2014 at 2:06
  • $\begingroup$ Please feel free to leave different derivations, if you think you have a clean alternative answer (even if the question already has a accepted answer and you think you can still provide a nice/clean derivation for it!) :) $\endgroup$ Nov 23, 2014 at 3:08
  • $\begingroup$ Check out the resources on matrix differentials listed at the bottom of this question: math.stackexchange.com/questions/4105372/… $\endgroup$
    – kanso37
    Apr 18, 2021 at 3:31

1 Answer 1

7
$\begingroup$

$$J(x) = x^TAx = \sum_{i,j} A_{ij}x_ix_j \implies \dfrac{dJ(x)}{dx_k} = \sum_{i,j} A_{ij} \dfrac{d(x_ix_j)}{dx_k}$$ We have $$\dfrac{d(x_ix_j)}{dx_k} = \delta_{ik}x_j + \delta_{jk}x_i$$ Hence, \begin{align} \dfrac{dJ(x)}{dx_k} & = \sum_{i,j} A_{ij} \left(\delta_{ik}x_j + \delta_{jk}x_i\right) = \sum_{j}A_{kj}x_j + \sum_{i}A_{ik}x_i = \left((Ax)_k + (A^Tx)_k\right) \end{align} Hence, $$\dfrac{dJ}{dx} = \left(A+A^T\right)x$$

$\endgroup$
4
  • $\begingroup$ Very nicely done. $\endgroup$
    – GDumphart
    Nov 23, 2014 at 1:44
  • $\begingroup$ Do you mind explaining a little more you second line with the deltas? What do the deltas exactly mean? Is it possible for a little more detail there? $\endgroup$ Nov 23, 2014 at 2:30
  • 1
    $\begingroup$ @Pinocchio $\delta_{ij}$ is the Kronecker delta, i.e., $\delta_{ij} = 0$ if $i \neq j$ and $\delta_{ij}=1$ if $i=j$. $\endgroup$
    – Adhvaitha
    Nov 23, 2014 at 2:32
  • $\begingroup$ For the last line when you use the subscript k to "index" from a vector, is that standard notation? Or should you explicitly say a small comment instead of just saying "Hence". Personally, it took me a little while to see why the step was correct, not sure if it was notational and wanted to make it more useful for everyone else too just to avoid confusion. Btw, thanks a lot and a very clean way of deriving it btw! :) $\endgroup$ Nov 23, 2014 at 3:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.