I want to optimize a function $E = \sum_i(Y_i-f(Z_i))^2$ w.r.t. $T$ where $f(Z)=ZTV^t$, and all are matrices (not necessarily square). Here $T$ acts as weights as in regression for example. $Y \in R^{n \times p}$, $Z \in R^{n \times k}$, $T \in R^{k \times k}$, and $V \in R^{p \times k}$. $p,n,k$ are arbitrary integers ( $>1$).

How can I do this?

From a gradient based algorithm point of view, do you need to compute the $\partial E/\partial T_{a,b}$ where $T_{a,b}$ is an element of $T$? If so, how can you do that and after computing, do you just simultaneously perform $T_{a,b} = T_{a,b} - \alpha * \partial E/\partial T_{a,b} $ for all $a,b$ and learning rate $\alpha$ as in the usual gradient descent?

($A^t$ denotes the transpose of matrix $A$)

  • $\begingroup$ You can represent matrix multiplication with Kronecker products. Then it becomes a normal least squares problem in the vectorization you choose. $\endgroup$ Apr 1, 2018 at 7:09

1 Answer 1


Let's use a colon to denote the trace/Frobenius product $$A:B={\rm tr}(A^TB)$$ and let's define the additional matrix variables $$\eqalign{ F &= ZTV^T,\,\,\,M = (F-Y) \cr }$$ Write the function in terms of these new variables, then find its differential and gradient $$\eqalign{ {\mathcal E} &= M:M \cr\cr d{\mathcal E} &= 2M:dM = 2M:dM \cr&= 2M:Z\,dT\,V^T \cr &= 2Z^TMV:dT \cr\cr \frac{\partial {\mathcal E}}{\partial T} &= 2Z^TMV \cr&= 2Z^T(F-Y)V \cr }$$ Set the gradient to zero and solve $$\eqalign{ Z^TYV &= Z^TFV = Z^T(ZTV^T)V \cr\cr T &= (Z^TZ)^{-1}Z^TYV(V^TV)^{-1} \cr&= Z^+Y(V^+)^T \cr }$$ where $M^+$ denotes the pseudoinverse of $M$.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.