Derivative of vector valued function with Kronecker products involved

Question

I would like to calulate the derivative of the following function $f:R^m \rightarrow R$ with: $ f(\mathbf{x})=f(x_1,\dots,x_m )= \| A - G B\|_F^2 $, where $A \in C^{n\times k},$ $B \in C^{m\times k},$ and $G \in C^{n\times m}$ with $G:=[\mathbf{g}(x_1),\dots,\mathbf{g}(x_m)]$, where $\mathbf{g}:R\rightarrow C^n $ is a differentiable function with derivative $\mathbf{g}'(x)$.

If we express the inner product in vector form and define $W=A - G B$ we have:

$ f(\mathbf{x})=\mathrm{vec}(W)^H\mathrm{vec}(W)$. Hence, its differential is:

$df = 2\mathrm{Re}\{ \mathbf{w^*}:d \mathbf{w}\}, $ where $\mathbf{w}=\mathrm{vec}(W).$

Next, follows: $d \mathbf{w}=-(B^T \otimes I_n)d \mathbf{g}$, where $\mathbf{g} = \mathrm{vec}(G).$ Now for $d \mathbf{g}$ I came to the following expression:

$d \mathbf{g}= d \begin{bmatrix}\mathbf{g}(x_1)\\ \vdots\\ \mathbf{g}(x_m)\end{bmatrix}= \begin{bmatrix}\mathbf{g}'(x_1)dx_1\\ \vdots\\ \mathbf{g}'(x_m)dx_m \end{bmatrix}=\mathrm{vec}(H \mathrm{diag}(d\mathbf{x})) = (I_m \otimes H) \mathrm{vec}(\mathrm{diag}(d\mathbf{x}))$, where

$H=\begin{bmatrix}\mathbf{g}'(x_1), \dots, \mathbf{g}'(x_m)\end{bmatrix}$ is a $n\times m$ matrix of derivatives. Finally, in my attempt to derive the gradient I came to:

$\mathbf{w}^*:d\mathbf{w}=- (I_m \otimes H^T)(B \otimes I_n)\mathbf{w}^*:\mathrm{vec}(\mathrm{diag}(d\mathbf{x})).$

However, I am stuck and not sure I can derive the gradient from that formula. Can I express the vector on the right with a selection matrix from a higher dimension?

Any help will be greatly appreciated.

Answer 1

The vec-operator messed you up. You only need the Diag/diag operators.
First, as you have done, define the matrices $$\eqalign{ W &= GB-A,\quad X={\rm Diag}(x),\quad x={\rm diag}(X) \cr G &= g(x),\quad H = g'(x) \quad\implies\; dG = H\,dX \cr X &= X^H = X^T = X^* \cr }$$ Write the function in terms of these matrices.
Then calculate its differential and gradient. $$\eqalign{ f &= W^*:W \cr df &= W^*:dW \;+\; W:dW^* \cr &= W^*:dG\,B \;+\; W:dG^*B^* \cr &= W^*B^T:dG \;+\; WB^H:dG^* \cr &= W^*B^T:H\,dX \;+\; WB^H:H^*dX \cr &= H^TW^*B^T:dX \;+\; H^HWB^H:dX \cr &= (H^TW^*B^T + H^HWB^H):{\rm Diag}(dx) \cr &= {\rm diag}(H^TW^*B^T + H^HWB^H):dx \cr \frac{\partial f}{\partial x} &= {\rm diag}(H^TW^*B^T + H^HWB^H) \cr &= 2\,{\cal Re}\Big({\rm diag}(H^TW^*B^T)\Big) \cr &= 2\,{\cal Re}\Big({\rm diag}(H^HWB^H)\Big) \cr &= 2\,{\cal Re}\Big({\rm diag}(B^*W^TH^*)\Big) \cr &= 2\,{\cal Re}\Big({\rm diag}(BW^HH)\Big) \cr }$$ Note that a colon indicates the trace/Frobenius product, i.e. $$\eqalign{ A:B = {\rm Tr}(A^TB)}$$ Also note that my $W$ matrix is defined as the negative of yours.