Consider the loss function $$ \min_{A}\sum_{k=1}^{3} (A^kx_1-x_{k+1})^2, $$ which is optimized using a gradient descent method, i.e. $$ A_{2\times2} = A_{2\times2} - \alpha (\text{ gradient w.r.t. A})_{2\times2}. $$ The parameter matrix $A$ is a $2\times2$ matrix, say $$ A = \begin{pmatrix}a_1 & a_2 \\ a_3 & a_4\end{pmatrix}. $$ However, the loss function is two dimensional ($2\times1$ vector), so what would that gradient in this case be, in order for the parameters in $A$ to be updated correctly. If I try to compute said gradient, let's say for the term $$ A^2x_1, $$ I end up with $$ \frac{\partial A^2_{x_1}}{\partial A}= \frac{\partial \begin{pmatrix}(a_1^2+a_2a_3)x_1+(a_1a_2+a_2a_4)x_2 \\ (a_1a_3+a_1a_4)x_1+(a_2a_3+a_4^2)x_2\end{pmatrix}}{\partial A}=\begin{pmatrix}2a_1x_1+a_2x_2 & a_3x_1+a_2x_2 & a_2x_1 & a_2x_2 \\ a_3x_1 & a_3x_2 & a_1x_1+a_2x_2 & a_1x_1 + 2a_4x_2 \end{pmatrix}, $$

which seems to be a $2\times4$ matrix. This is meaningless for my algorithm! I still don't know how to update my parameters, since I've got too many elements in my gradient. I would expect a gradient of same dimension as my parameter matrix $A$, so the arithmetic operations remain valid.

What is the correct way to compute a gradient with respect to a (square) matrix, when your loss function is multi-dimensional? Or is my method/interpretation of gradient update wrong in the first place? If so, what do I do instead?

  • 1
    $\begingroup$ It does not quite make sense for the objective function to be two-dimensional (unless you are looking at multi-objective optimization, but that is a different story). Maybe you want $\| A^k x_1- x_{k+1}\|^2$? (To see, why it doesn't quite make sense, in the ordinary sense, note that the two components of your loss could be minimized at different points.) $\endgroup$
    – passerby51
    Sep 9, 2017 at 15:24
  • $\begingroup$ Why not if I may ask? The problem concerns a system identification objective, where we have multiple states. Also my advisor (full professor) came up with this function and my other supervisor (postdoc) approves of this idea. $\endgroup$
    – WiseDev
    Sep 9, 2017 at 15:28
  • $\begingroup$ First have a look at multi-objective optimization, and see if this is what you want to do. Why not? Because of what I mentioned. You have to think about what it means for a function $f: \mathbb R^4 \to \mathbb R^2$ to be minimized. If it clear what it means for scalar-valued functions. Not so clear otherwise. $\endgroup$
    – passerby51
    Sep 9, 2017 at 15:45

1 Answer 1


passerby51 is right. I forgot that my professor drew a function of the form $(\cdot)^T(\cdot)$, which would be equivalent to $(\cdot)^2$ in the scalar case. And that is equivalent to passerby51's suggestion, that we use the 2-norm squared, which is a scalar function.

Don't mind my mistake guys... and stay away from drugs.


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .