Derivative with respect to matrix, of a multi-dimensional function, used for gradient descent optimization.

Consider the loss function $$\min_{A}\sum_{k=1}^{3} (A^kx_1-x_{k+1})^2,$$ which is optimized using a gradient descent method, i.e. $$A_{2\times2} = A_{2\times2} - \alpha (\text{ gradient w.r.t. A})_{2\times2}.$$ The parameter matrix $A$ is a $2\times2$ matrix, say $$A = \begin{pmatrix}a_1 & a_2 \\ a_3 & a_4\end{pmatrix}.$$ However, the loss function is two dimensional ($2\times1$ vector), so what would that gradient in this case be, in order for the parameters in $A$ to be updated correctly. If I try to compute said gradient, let's say for the term $$A^2x_1,$$ I end up with $$\frac{\partial A^2_{x_1}}{\partial A}= \frac{\partial \begin{pmatrix}(a_1^2+a_2a_3)x_1+(a_1a_2+a_2a_4)x_2 \\ (a_1a_3+a_1a_4)x_1+(a_2a_3+a_4^2)x_2\end{pmatrix}}{\partial A}=\begin{pmatrix}2a_1x_1+a_2x_2 & a_3x_1+a_2x_2 & a_2x_1 & a_2x_2 \\ a_3x_1 & a_3x_2 & a_1x_1+a_2x_2 & a_1x_1 + 2a_4x_2 \end{pmatrix},$$

which seems to be a $2\times4$ matrix. This is meaningless for my algorithm! I still don't know how to update my parameters, since I've got too many elements in my gradient. I would expect a gradient of same dimension as my parameter matrix $A$, so the arithmetic operations remain valid.

What is the correct way to compute a gradient with respect to a (square) matrix, when your loss function is multi-dimensional? Or is my method/interpretation of gradient update wrong in the first place? If so, what do I do instead?

• It does not quite make sense for the objective function to be two-dimensional (unless you are looking at multi-objective optimization, but that is a different story). Maybe you want $\| A^k x_1- x_{k+1}\|^2$? (To see, why it doesn't quite make sense, in the ordinary sense, note that the two components of your loss could be minimized at different points.) Sep 9, 2017 at 15:24
• Why not if I may ask? The problem concerns a system identification objective, where we have multiple states. Also my advisor (full professor) came up with this function and my other supervisor (postdoc) approves of this idea. Sep 9, 2017 at 15:28
• First have a look at multi-objective optimization, and see if this is what you want to do. Why not? Because of what I mentioned. You have to think about what it means for a function $f: \mathbb R^4 \to \mathbb R^2$ to be minimized. If it clear what it means for scalar-valued functions. Not so clear otherwise. Sep 9, 2017 at 15:45

passerby51 is right. I forgot that my professor drew a function of the form $(\cdot)^T(\cdot)$, which would be equivalent to $(\cdot)^2$ in the scalar case. And that is equivalent to passerby51's suggestion, that we use the 2-norm squared, which is a scalar function.