0
$\begingroup$

When differentiating a matrix by a matrix it is true that dA(X)/dX is the same thing as dvec(A(X))/dvec(X) acorrording to this paper.

So, I compared kronecker product representation with vec operator representation following case.

$\mathbf{Y}=(2,3)$, $\mathbf{X}=(4,2)$

$$ \begin{aligned} \frac{\partial \mathbf{Y}}{\partial \mathbf{X}} &= \frac{\partial }{\partial \mathbf{X}} \otimes \mathbf{Y} \\[10pt] &=\left[\begin{array}{cc} \dfrac{\partial }{\partial X_{11}} & \dfrac{\partial }{\partial X_{12}} \\ \dfrac{\partial }{\partial X_{21}} & \dfrac{\partial }{\partial X_{22}} \\ \dfrac{\partial }{\partial X_{31}} & \dfrac{\partial }{\partial X_{32}} \\ \dfrac{\partial }{\partial X_{41}} & \dfrac{\partial }{\partial X_{42}} \end{array}\right] \otimes \mathbf{Y} \\[10pt] &= \left[\begin{array}{ccc|ccc} \dfrac{\partial Y_{11}}{\partial X_{11}} & \dfrac{\partial Y_{12}}{\partial X_{11}} & \dfrac{\partial Y_{13}}{\partial X_{11}} & \dfrac{\partial Y_{11}}{\partial X_{12}} & \dfrac{\partial Y_{12}}{\partial X_{12}} & \dfrac{\partial Y_{13}}{\partial X_{12}} \\ \dfrac{\partial Y_{21}}{\partial X_{11}} & \dfrac{\partial Y_{22}}{\partial X_{11}} & \dfrac{\partial Y_{23}}{\partial X_{11}} & \dfrac{\partial Y_{21}}{\partial X_{12}} & \dfrac{\partial Y_{22}}{\partial X_{12}} & \dfrac{\partial Y_{23}}{\partial X_{12}} \\[10pt] \hline \dfrac{\partial Y_{11}}{\partial X_{21}} & \dfrac{\partial Y_{12}}{\partial X_{21}} & \dfrac{\partial Y_{13}}{\partial X_{21}} & \dfrac{\partial Y_{11}}{\partial X_{22}} & \dfrac{\partial Y_{12}}{\partial X_{22}} & \dfrac{\partial Y_{13}}{\partial X_{22}} \\ \dfrac{\partial Y_{21}}{\partial X_{21}} & \dfrac{\partial Y_{22}}{\partial X_{21}} & \dfrac{\partial Y_{23}}{\partial X_{21}} & \dfrac{\partial Y_{21}}{\partial X_{22}} & \dfrac{\partial Y_{22}}{\partial X_{22}} & \dfrac{\partial Y_{23}}{\partial X_{22}} \\[10pt] \hline \dfrac{\partial Y_{11}}{\partial X_{31}} & \dfrac{\partial Y_{12}}{\partial X_{31}} & \dfrac{\partial Y_{13}}{\partial X_{31}} & \dfrac{\partial Y_{11}}{\partial X_{32}} & \dfrac{\partial Y_{12}}{\partial X_{32}} & \dfrac{\partial Y_{13}}{\partial X_{32}} \\ \dfrac{\partial Y_{21}}{\partial X_{31}} & \dfrac{\partial Y_{22}}{\partial X_{31}} & \dfrac{\partial Y_{23}}{\partial X_{31}} & \dfrac{\partial Y_{21}}{\partial X_{32}} & \dfrac{\partial Y_{22}}{\partial X_{32}} & \dfrac{\partial Y_{23}}{\partial X_{32}} \\[10pt] \hline \dfrac{\partial Y_{11}}{\partial X_{41}} & \dfrac{\partial Y_{12}}{\partial X_{41}} & \dfrac{\partial Y_{13}}{\partial X_{41}} & \dfrac{\partial Y_{11}}{\partial X_{42}} & \dfrac{\partial Y_{12}}{\partial X_{42}} & \dfrac{\partial Y_{13}}{\partial X_{42}} \\ \dfrac{\partial Y_{21}}{\partial X_{41}} & \dfrac{\partial Y_{22}}{\partial X_{41}} & \dfrac{\partial Y_{23}}{\partial X_{41}} & \dfrac{\partial Y_{21}}{\partial X_{42}} & \dfrac{\partial Y_{22}}{\partial X_{42}} & \dfrac{\partial Y_{23}}{\partial X_{42}} \end{array} \right] \end{aligned} $$

and

$ \text{vec}(\mathbf{Y})= \mathbf{y}= \begin{bmatrix} y_{11} & y_{21} & y_{12} & y_{22} & y_{13} & y_{23} \end{bmatrix}^T$

$ \text{vec}(\mathbf{X})= \mathbf{x}= \begin{bmatrix} x_{11} & x_{21} & x_{31} & x_{41} & x_{12} & x_{22} & x_{32} & x_{42} \end{bmatrix}^T$

$$ \begin{aligned} \frac{\partial \text{vec}(\mathbf{Y})}{\partial \text{vec}(\mathbf{X})} &= \frac{\partial \mathbf{y}}{\partial \mathbf{x}} \\ &= \left[\begin{array}{cccccc} \dfrac{\partial Y_{11}}{\partial X_{11}} & \dfrac{\partial Y_{21}}{\partial X_{11}} & \dfrac{\partial Y_{12}}{\partial X_{11}} & \dfrac{\partial Y_{22}}{\partial X_{11}} & \dfrac{\partial Y_{13}}{\partial X_{11}} & \dfrac{\partial Y_{23}}{\partial X_{11}} \\[5pt] \dfrac{\partial Y_{11}}{\partial X_{21}} & \dfrac{\partial Y_{21}}{\partial X_{21}} & \dfrac{\partial Y_{12}}{\partial X_{21}} & \dfrac{\partial Y_{22}}{\partial X_{21}} & \dfrac{\partial Y_{13}}{\partial X_{21}} & \dfrac{\partial Y_{23}}{\partial X_{21}} \\[5pt] \dfrac{\partial Y_{11}}{\partial X_{31}} & \dfrac{\partial Y_{21}}{\partial X_{31}} & \dfrac{\partial Y_{12}}{\partial X_{31}} & \dfrac{\partial Y_{22}}{\partial X_{31}} & \dfrac{\partial Y_{13}}{\partial X_{31}} & \dfrac{\partial Y_{23}}{\partial X_{31}} \\[5pt] \dfrac{\partial Y_{11}}{\partial X_{41}} & \dfrac{\partial Y_{21}}{\partial X_{41}} & \dfrac{\partial Y_{12}}{\partial X_{41}} & \dfrac{\partial Y_{22}}{\partial X_{41}} & \dfrac{\partial Y_{13}}{\partial X_{41}} & \dfrac{\partial Y_{23}}{\partial X_{41}} \\[5pt] \dfrac{\partial Y_{11}}{\partial X_{12}} & \dfrac{\partial Y_{21}}{\partial X_{12}} & \dfrac{\partial Y_{12}}{\partial X_{12}} & \dfrac{\partial Y_{22}}{\partial X_{12}} & \dfrac{\partial Y_{13}}{\partial X_{12}} & \dfrac{\partial Y_{23}}{\partial X_{12}} \\[5pt] \dfrac{\partial Y_{11}}{\partial X_{22}} & \dfrac{\partial Y_{21}}{\partial X_{22}} & \dfrac{\partial Y_{12}}{\partial X_{22}} & \dfrac{\partial Y_{22}}{\partial X_{22}} & \dfrac{\partial Y_{13}}{\partial X_{22}} & \dfrac{\partial Y_{23}}{\partial X_{22}} \\[5pt] \dfrac{\partial Y_{11}}{\partial X_{32}} & \dfrac{\partial Y_{21}}{\partial X_{32}} & \dfrac{\partial Y_{12}}{\partial X_{32}} & \dfrac{\partial Y_{22}}{\partial X_{32}} & \dfrac{\partial Y_{13}}{\partial X_{32}} & \dfrac{\partial Y_{23}}{\partial X_{32}} \\[5pt] \dfrac{\partial Y_{11}}{\partial X_{42}} & \dfrac{\partial Y_{21}}{\partial X_{42}} & \dfrac{\partial Y_{12}}{\partial X_{42}} & \dfrac{\partial Y_{22}}{\partial X_{42}} & \dfrac{\partial Y_{13}}{\partial X_{42}} & \dfrac{\partial Y_{23}}{\partial X_{42}} \end{array} \right] \end{aligned} $$

If these results are the same, how do I transpose $\frac{\partial }{\partial \mathbf{X}} \otimes \mathbf{Y}$ to make it equal to $\frac{\partial \text{vec}(\mathbf{Y})}{\partial \text{vec}(\mathbf{X})}$?

$\endgroup$
1
$\begingroup$

The derivative operation is a distraction, so let's omit it.

Given the matrices $$\eqalign{ X &\in {\mathbb R}^{m\times n} \cr Y &\in {\mathbb R}^{p\times q} \cr }$$ and their Kronecker products $$\eqalign{ A &= X\otimes Y &\in {\mathbb R}^{mp\times nq} \cr B &= {\rm vec}(X)\otimes {\rm vec}(Y)^T &\in {\mathbb R}^{mn\times pq}\cr }$$ you want to know how to transform $A \implies B$

The elements in these matrices are identical. The only difference is in the matrix dimensions and in the ordering of the elements.

Vectorizing the matrices yields two vectors related by a permutation. $$\eqalign{ P\,{\rm vec}(A) &= {\rm vec}(B) \cr Pa &= b \cr P_{ij}\,a_j &= b_i \cr }$$ The elements of the permuation are easy to find:
   $P_{ij}$ equals one if the $j^{th}$ element of $a$ maps to the $i^{th}$ element of $b$.

The single-entry matrix $E_{mn}$ has all elements equal to zero, except for the $(m,n)$ element which is equal to one. Replacing $(X,Y)$ by single-entry matrices leads to $(A,B)$ becoming single-entry matrices and $(a,b)$ becoming single-entry vectors.

The non-zero elements of $P$ can be found by iterating through every possible single-entry matrix for the input matrices $X \,{\rm and}\, Y$. Then setting $\,P_{ij}=1$, where the non-zero elements of the output vectors $(b,a)$ are observed at $(b_i,a_j)$.

The permutation depends only on the dimensions of the starting matrices. For your derivative expressions the dimensions are $(m,n,p,q)=(4,2,2,3)$ and the transformation is
$${\rm vec}\Bigg(\frac{\partial{\rm vec}(Y)}{\partial{\rm vec}(X)}\Bigg) = P\,\,{\rm vec}\Bigg(\frac{\partial}{\partial X}\otimes Y\Bigg) $$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.