I'd like to find the gradient of $\frac{1}{2} \big \| X A^T \big \|_F^2$ with respect to $X_{ij}$.

Going by the chain rule in the Matrix Cookbook (eqn 126), it's something like

$\partial \, \Big[\frac{1}{2} \big\| X A \big\|_F^2 \Big] /\, \partial X_{ij}= \text{Tr} \Big[(XA^T)^T (J^{jk} A^T) \Big]$, where $J$ has same dimensions as $X$ and has zeros everywhere except for entry $(j,k)$.

I'm not so sure about the $J^{jk} A^T$ bit (cookbook eqn 66 applies here?).



Recall that if $A,B \in \mathbb{R}^{m \times n}$ then \begin{equation} \langle A, B \rangle = \text{Tr}(A^T B) \end{equation} and \begin{align*} \|A\|_F^2 &= \langle A,A \rangle \\ &= \text{Tr}(A^T A) \\ &= \text{Tr}(A A^T). \end{align*}

Let $f:\mathbb{R}^{m \times n} \to \mathbb{R}$ such that \begin{align*} f(X) &= \frac12 \| X A^T \|_F^2 \\ &= \frac12 \text{Tr}(X A^T A X^T). \end{align*} Let $J$ be the $m \times n$ matrix whose entries are all $0$ except $J_{ij}$ which is equal to $1$. Let $\Delta X = \epsilon J$, where $\epsilon > 0$ is tiny.


\begin{align*} f(X + \Delta X) &= \frac12 \text{Tr}((X + \Delta X)A^T A (X + \Delta X)^T) \\ &= \frac12 \text{Tr}(X A^T A X^T) + \frac12 \text{Tr}(\Delta X A^T A X^T) + \frac12 \text{Tr}(X A^T A \Delta X^T) \\ & \qquad + \frac12 \text{Tr}(\Delta X A^T A \Delta X^T) \\ &\approx \frac12 \text{Tr}(X A^T A X^T) + \frac12 \text{Tr}(\Delta X A^T A X^T) + \frac12 \text{Tr}(X A^T A \Delta X^T) \\ &= \frac12 \text{Tr}(X A^T A X^T) + \text{Tr}(X A^T A \Delta X^T) \\ &= f(X) + \left\langle X A^T A,\Delta X \right\rangle \\ &= f(X) + \epsilon \left \langle X A^T A,J \right\rangle. \end{align*}

Comparing this result with the equation \begin{equation} f(X + \epsilon J) \approx f(X) + \epsilon \frac{\partial f(X)}{\partial X_{ij}} \end{equation} we see that \begin{equation} \frac{\partial f(X)}{\partial X_{ij}} = \left \langle X A^T A,J \right\rangle. \end{equation}

  • 1
    $\begingroup$ thanks, I think there's a mistake in the second line of f(X + delta X), where the first term should be 1/2 Tr(X A^T A X^T) to match your definition of the Frobenius norm in terms of trace earlier. $\endgroup$ – purple51 Nov 11 '12 at 23:08
  • $\begingroup$ Thanks. I think I fixed it, but let me know if you see any other errors. $\endgroup$ – littleO Nov 12 '12 at 6:35
  • $\begingroup$ Why do you say that all entries of J are zero and then say that all entries (J_ij) are 1? Or did you mean to say that the diagonal entries i.e. J_ii are 1? $\endgroup$ – Stephen Tierney Oct 17 '13 at 23:29
  • $\begingroup$ @StephenTierney $J$ has only one nonzero entry -- namely, $J_{ij}$. I'm viewing $i$ and $j$ as given, fixed indices, and we want to compute $\frac{\partial f(X)}{\partial X_{ij}}$. $\endgroup$ – littleO Oct 18 '13 at 1:59

Let $M=XA^T$, then taking the differential leads directly to the derivative $$\eqalign{ f &= \frac{1}{2}\,M:M \cr df &= M:dM \cr &= M:dX\,A^T \cr &= MA:dX \cr &= XA^TA:dX \cr \frac{\partial f}{\partial X} &= XA^TA \cr }$$ Your question asks for the {$i,j$}-th component of this derivative, which is obtained by taking its Frobenius product with $J_{ij}$ $$\eqalign{ \frac{\partial f}{\partial X_{ij}} &= XA^TA:J_{ij} \cr }$$ If you are unfamiliar with the Frobenius product, you can express the result in terms of the trace function instead, since $\,X\!:\!Y={\rm tr}(X^TY)$.

This yields the same result as you found using the Cookbook -- except you messed up your indices between the LHS {$ij$} and RHS {$jk$} for some inexplicable reason.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.