Consider the loss function $$ \min_{A}\sum_{k=1}^{3} (A^kx_1-x_{k+1})^2, $$ which is optimized using a gradient descent method, i.e. $$ A_{2\times2} = A_{2\times2} - \alpha (\text{ gradient w.r.t. A})_{2\times2}. $$ The parameter matrix $A$ is a $2\times2$ matrix, say $$ A = \begin{pmatrix}a_1 & a_2 \\ a_3 & a_4\end{pmatrix}. $$ However, the loss function is two dimensional ($2\times1$ vector), so what would that gradient in this case be, in order for the parameters in $A$ to be updated correctly. If I try to compute said gradient, let's say for the term $$ A^2x_1, $$ I end up with $$ \frac{\partial A^2_{x_1}}{\partial A}= \frac{\partial \begin{pmatrix}(a_1^2+a_2a_3)x_1+(a_1a_2+a_2a_4)x_2 \\ (a_1a_3+a_1a_4)x_1+(a_2a_3+a_4^2)x_2\end{pmatrix}}{\partial A}=\begin{pmatrix}2a_1x_1+a_2x_2 & a_3x_1+a_2x_2 & a_2x_1 & a_2x_2 \\ a_3x_1 & a_3x_2 & a_1x_1+a_2x_2 & a_1x_1 + 2a_4x_2 \end{pmatrix}, $$

which seems to be a $2\times4$ matrix. This is meaningless for my algorithm! I still don't know how to update my parameters, since I've got too many elements in my gradient. I would expect a gradient of same dimension as my parameter matrix $A$, so the arithmetic operations remain valid.

What is the correct way to compute a gradient with respect to a (square) matrix, when your loss function is multi-dimensional? Or is my method/interpretation of gradient update wrong in the first place? If so, what do I do instead?

  • $\begingroup$ It does not quite make sense for the objective function to be two-dimensional (unless you are looking at multi-objective optimization, but that is a different story). Maybe you want $\| A^k x_1- x_{k+1}\|^2$? (To see, why it doesn't quite make sense, in the ordinary sense, note that the two components of your loss could be minimized at different points.) $\endgroup$ – passerby51 Sep 9 '17 at 15:24
  • $\begingroup$ Why not if I may ask? The problem concerns a system identification objective, where we have multiple states. Also my advisor (full professor) came up with this function and my other supervisor (postdoc) approves of this idea. $\endgroup$ – BlueRine S Sep 9 '17 at 15:28
  • $\begingroup$ First have a look at multi-objective optimization, and see if this is what you want to do. Why not? Because of what I mentioned. You have to think about what it means for a function $f: \mathbb R^4 \to \mathbb R^2$ to be minimized. If it clear what it means for scalar-valued functions. Not so clear otherwise. $\endgroup$ – passerby51 Sep 9 '17 at 15:45

passerby51 is right. I forgot that my professor drew a function of the form $(\cdot)^T(\cdot)$, which would be equivalent to $(\cdot)^2$ in the scalar case. And that is equivalent to passerby51's suggestion, that we use the 2-norm squared, which is a scalar function.

Don't mind my mistake guys... and stay away from drugs.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.