# compute gradient of $\dfrac{1}{2} \left\lVert A - XY^{T} \right\rVert _{F}^{2}$ via chain rule

Let $$A \in \mathbb{R}^{n \times n}$$ and $$X, Y \in \mathbb{R}^{n \times r}$$. Consider the function $$$$H \left( X , Y \right) := \dfrac{1}{2} \left\lVert A - XY^{T} \right\rVert _{F}^{2} ,$$$$ where $$\left\lVert \cdot \right\rVert _{F}$$ denotes the Frobinus norm.

The first one seems to be easy. I used the chain rule to get $$$$\nabla_{X} H \left( X , Y \right) = \left( A - XY^{T} \right) \nabla_{X} \left( - XY^{T} \right) = - \left( A - XY^{T} \right) Y^{T} .$$$$

For the second one, as $$A - XY^{T} = A - \left( X^{T}Y \right) ^{T}$$, we have \begin{align} \nabla_{Y} H \left( X , Y \right) = \left( \left( A - XY^{T} \right) \nabla_{Y} \left( - \left( X^{T}Y \right) ^{T} \right) \right) ^{T} & = \left( - \left( A - XY^{T} \right) X^{T} \right) ^{T} \\ & = - X \left( A - XY^{T} \right) ^{T} . \end{align}

Is my $$\nabla_{Y} H \left( X , Y \right)$$ formula correct?

And is there other approaches to compute the gradient. I guess we can compute $$H \left( X + \delta X , Y \right)$$ then deduce the gradient from the difference $$H \left( X + \delta X , Y \right) - H \left( X , Y \right)$$.

## 1 Answer

Let $$B = (XY^T-A) \implies dB = (dX\,Y^T+X\,dY^T)$$ Write the function in terms of $$B$$, then find its differential and gradients. \eqalign{ H &= \tfrac{1}{2}B:B \cr dH &= B:dB \cr &= B:dX\,Y^T &+ \,\,B:X\,dY^T \cr &= B:dX\,Y^T &+ \,\,B^T:dY\,X^T \cr &= BY:dX &+ \,\,B^TX:dY \cr \frac{\partial H}{\partial X} &= BY, \quad \frac{\partial H}{\partial Y} &= B^TX \cr } where the colon product is a convenient notation for the trace, i.e. \eqalign{ M:N &= {\rm Tr}(M^TN) } Depending on your preferred layout convention, these gradients might need to be transposed.

Your first solution has a problem (in red) \eqalign{ \frac{\partial H}{\partial X} &= BY^T = X\color{red}{Y^TY^T} - AY^T \cr } In matrix calculus, terms involving the transpose are invariably of the form $$Y^TY$$ or $$YY^T$$

Your second solution is $$XB^T$$ but it should be $$B^TX$$.

To get a better handle on these transpose issues, I recommend that you work through the problem in which all of the matrices are rectangular, i.e. $$A \in {\mathbb R}^{m\times n}, \quad X \in {\mathbb R}^{m\times r}, \quad Y \in {\mathbb R}^{n\times r}$$ Then the solution to the current problem can be recovered by setting $$m=n$$.

• thanks for your recommendation. now it makes more sense since, at some point when I consider $H(Z,Z)$ and try to compute its gradients, my solution cause some trouble but your solution gives $\left( B + B^{T} \right) X$ which I think should be correct now – mortal Apr 4 at 8:28