I am trying to optimize this cost function by using Gauss-Newton method. $$f = \sum_{i = 1}^n Tr{(Z^TZ)}$$ where $Z$ is a $4\times4$ matrix and it is a function of real vector $\vec{a}\in\mathbb{R}^5$. And because I am using Gauss-Newton approach, I define $g \in\mathbb{R}^{n}$ to be $\sqrt{Tr(Z^T_iZ_i)} = \sqrt{p_i}$, where $i = 1,2,...,n$

This is my attempt to find the gradient and its Hessian $H$. Gradient of the cost function is $2J^Tg$ and Hessian is $2(J^TJ + \sum(g_iH_i))$, where $J$ is Jacobian matrix.

$$\frac{\partial g}{\partial\epsilon}|_{\epsilon = 0} = \frac{\partial g}{\partial p}\frac{\partial p}{\partial Z}\frac{\partial Z}{\partial\epsilon}|_{\epsilon = 0}$$ $$\frac{\partial g}{\partial\epsilon}|_{\epsilon = 0} = \frac{1}{2\sqrt{p_i}}Z(\epsilon\vec{a})\frac{\partial Z(\epsilon\vec{a})}{\partial\epsilon}|_{\epsilon = 0}$$

Apparently, the expression above is going to be $4\times4$ matrix. How to proceed find the gradient of the cost function because we cannot multiply $J$ and $g$ right away? I have tried with Hessian matrix and it is even worse. I am not sure whether I am on the right path or not so I did not show my attempt on Hessian here.

  • $\begingroup$ You need the matrix cookbook! $\endgroup$ – Michael Grant Feb 19 '15 at 3:57
  • $\begingroup$ Are you aware that $\mathop{\textrm{Tr}}(Z^TZ)$ is simply $\sum_{i,j} Z_{ij}^2$? Yes, the sum of the squares of the elements of $Z$. That should make the calculus rather simple. $\endgroup$ – Michael Grant Feb 19 '15 at 3:59
  • $\begingroup$ Thank you, I will take a look at the cookbook. According to your comment, my $g_i$ will be $\mathbb{R}^{16}$, and Jacobian will be of dimension $5\times16$, correct? $\endgroup$ – aofkrittin Feb 19 '15 at 4:10
  • $\begingroup$ I believe so but I am rusty on my Gauss Newton. I see no reason not to convert Z to a 16-element vector though! $\endgroup$ – Michael Grant Feb 19 '15 at 13:19
  • $\begingroup$ Come to think of it, my $g_i$ will still be $\mathbb{R}^{n}$. If we change from the cost function into sum of square, the cost will be $\sum_{i=1}^{n}\sum_{j,k}Z_{jk}^2$ and the problem will still be the same. $\endgroup$ – aofkrittin Feb 19 '15 at 16:03

Your latest formulation of the cost function can be greatly simplified $$ \eqalign { f &= \sum_k {\rm tr}\bigg((A_kX)^T(A_kX)\bigg) \cr &= \sum_k (A_k^TA_k):(XX^T) \cr &= M:(XX^T) \cr } $$ by summing the $A_k$ once and for all, and removing them from the problem. I also replaced the trace function with a Frobenius product for algebraic convenience.

Next, instead of jumping immediately to the chain rule, let's find the differential of $f$. $$ \eqalign { df &= M:d(XX^T) \cr &= M:(dXX^T + XdX^T) \cr &= (M+M^T):dXX^T \cr &= (M+M^T)X:dX \cr &= 2\,MX:dX \cr } $$ You haven't provided enough information about $X=X(a)$ for me to evaluate the derivative, but it will look like this $$ \eqalign { \frac {\partial f} {\partial a^T} &= 2\,MX : \bigg(\frac {\partial X} {\partial a^T}\bigg) \cr &= 2\,{\rm vec}(MX)^T\,\,\frac {\partial {\rm vec}(X)} {\partial a^T} \cr } $$


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.