# Derivative with respect to matrix, of a multi-dimensional function, used for gradient descent optimization.

Consider the loss function $$\min_{A}\sum_{k=1}^{3} (A^kx_1-x_{k+1})^2,$$ which is optimized using a gradient descent method, i.e. $$A_{2\times2} = A_{2\times2} - \alpha (\text{ gradient w.r.t. A})_{2\times2}.$$ The parameter matrix $A$ is a $2\times2$ matrix, say $$A = \begin{pmatrix}a_1 & a_2 \\ a_3 & a_4\end{pmatrix}.$$ However, the loss function is two dimensional ($2\times1$ vector), so what would that gradient in this case be, in order for the parameters in $A$ to be updated correctly. If I try to compute said gradient, let's say for the term $$A^2x_1,$$ I end up with $$\frac{\partial A^2_{x_1}}{\partial A}= \frac{\partial \begin{pmatrix}(a_1^2+a_2a_3)x_1+(a_1a_2+a_2a_4)x_2 \\ (a_1a_3+a_1a_4)x_1+(a_2a_3+a_4^2)x_2\end{pmatrix}}{\partial A}=\begin{pmatrix}2a_1x_1+a_2x_2 & a_3x_1+a_2x_2 & a_2x_1 & a_2x_2 \\ a_3x_1 & a_3x_2 & a_1x_1+a_2x_2 & a_1x_1 + 2a_4x_2 \end{pmatrix},$$

which seems to be a $2\times4$ matrix. This is meaningless for my algorithm! I still don't know how to update my parameters, since I've got too many elements in my gradient. I would expect a gradient of same dimension as my parameter matrix $A$, so the arithmetic operations remain valid.

What is the correct way to compute a gradient with respect to a (square) matrix, when your loss function is multi-dimensional? Or is my method/interpretation of gradient update wrong in the first place? If so, what do I do instead?

• It does not quite make sense for the objective function to be two-dimensional (unless you are looking at multi-objective optimization, but that is a different story). Maybe you want $\| A^k x_1- x_{k+1}\|^2$? (To see, why it doesn't quite make sense, in the ordinary sense, note that the two components of your loss could be minimized at different points.) – passerby51 Sep 9 '17 at 15:24
• Why not if I may ask? The problem concerns a system identification objective, where we have multiple states. Also my advisor (full professor) came up with this function and my other supervisor (postdoc) approves of this idea. – BlueRine S Sep 9 '17 at 15:28
• First have a look at multi-objective optimization, and see if this is what you want to do. Why not? Because of what I mentioned. You have to think about what it means for a function $f: \mathbb R^4 \to \mathbb R^2$ to be minimized. If it clear what it means for scalar-valued functions. Not so clear otherwise. – passerby51 Sep 9 '17 at 15:45

passerby51 is right. I forgot that my professor drew a function of the form $(\cdot)^T(\cdot)$, which would be equivalent to $(\cdot)^2$ in the scalar case. And that is equivalent to passerby51's suggestion, that we use the 2-norm squared, which is a scalar function.