If I have a function $f = v^TAv$, I can compute its derivative w.r.t. $v$ as follows

$$ \begin{align} f &= \sum_i \sum_j A_{ij}v_iv_j \\ \frac{\delta f}{\delta v_k} &= \sum_j A_{kj}v_j + \sum_i A_{ik}v_i \\ &= A_{k,:}v + vA_{:,k} \\ \frac{\delta f}{\delta v} &= (A + A^T)v. \end{align}$$

This is how I learnt how to compute the derivative wrt a vector. How can I compute derivative wrt a matrix ?

More specifically, what is $\dfrac{\delta f}{\delta A_{nm}}$? And how do I convert the result to a matrix?

  • 1
    $\begingroup$ A matrix is also a vector; an NXM matrix is a vector with NM coordinates. $\endgroup$ – Kavi Rama Murthy Jan 13 '18 at 11:40
  • $\begingroup$ What ha the function whose derivative you want to compute? $\endgroup$ – Mathemagical Jan 13 '18 at 11:53
  • $\begingroup$ @Mathemagical I want to compute $\frac{\delta f}{\delta A}$ $\endgroup$ – Kong Jan 13 '18 at 12:11
  • $\begingroup$ Use the trace/Frobenius product, i.e. $A:B={\rm tr}(A^TB)$, to write the function in a form such that finding the differential and gradient is trivial $$\eqalign{f&=vv^T:A\cr df&=vv^T:dA\cr\frac{\partial f}{\partial A}&=vv^T}$$ $\endgroup$ – greg Jan 13 '18 at 15:59

Differentiation is the finding of a linear approximation to the function at a particular point, denoted by $Df$. (See page 16 of Spivak, Calculus on Manifolds, for example). When $f$ is already linear in the variable, $Df=f$.

Now, $f$ is non-linear in $v$ and you computed the linear function $D_vf$ that best approximates $f(v)$.

But as a function of $A$, $f$ is already linear. (Since $f(aA+bB)=af(A)+bf(B)$). So the derivative $D_Af$ is $f$ itself.

However, you cannot write down an explicit expression for $\frac{\partial f}{\partial A}$, (a different thing from $D_Af$)

Note: the way you have written $\frac{\partial f}{\partial v}$ is not strictly correct. It should be $v^T(A+A^T)$. That is, it should be a row vector if $v$ is a column vector.

  • $\begingroup$ Not possible to write explicitly unless you represent the derivative itself using a non-standard meaning of matrix multiplication, that is. (Frobenius) $\endgroup$ – Mathemagical Jan 14 '18 at 10:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.