I am trying to differentiate the Kronecker product of a vector with respect to itself: $$ \frac{\partial}{\partial \pmb{\delta}} (\pmb{\delta} \otimes \pmb{\delta} ) $$ That is part of this larger differentiation: $$ \begin{equation} \label{eq1} \tag{1} \frac{\partial}{\partial \pmb{\delta}} \left(\pmb{\delta}^\intercal \mathbf{A} (\pmb{\delta} \otimes \pmb{\delta} ) \right) \end{equation} $$ Where:

-$\pmb{\delta}_{K \times 1}$ is a vector and $\pmb{\delta}^{\intercal}_{1\times K}$ is its transpose.

-$ \mathbf{A}_{K \times K^2}$ is constant matrix that is independent from $\pmb{\delta}$ ($ \partial \mathbf{A}/ \partial \pmb{\delta}=0) $.

Using definitions that can be found in Matrix Calculus textbooks, I have simplified it as follows: $$ \begin{split} \frac{\partial}{\partial \pmb{\delta}} (\pmb{\delta} \otimes \pmb{\delta} ) &=\frac{\partial \pmb{\delta}}{\partial \pmb{\delta}} \otimes \pmb{\delta}+ [\mathbf{I}_K \otimes \mathbf{U_{K^2 \times K^2}}][\frac{\partial \pmb{\delta}}{\partial \pmb{\delta}} \otimes \pmb{\delta}][\mathbf{I}_1 \otimes \mathbf{I}_{1\times 1} ]\\ &=vec(\mathbf{I}_K) \otimes \pmb{\delta}+[\mathbf{I}_K \otimes \mathbf{U_{K^2 \times K^2}}][vec(\mathbf{I}_K) \otimes \pmb{\delta}]\\ &=[\mathbf{I}_{K^3}+\mathbf{I}_K \otimes \mathbf{U_{K^2 \times K^2}}][vec(\mathbf{I}_K) \otimes \pmb{\delta}] \end{split} $$ Where:

  • $\mathbf{I}_n$ is $n \times n$ unit matrix
  • $vec(\mathbf{B})$ is the classic vectorization operator.
  • $\mathbf{U}_{r \times s}$ is the permutation matrix as it is defined in classic matrix calculus textbooks (e.g. here is shown with $T_{m,n}$).

Can the above equation get simplified further? My goal is to collect whatever form of $ \pmb{\delta}$ that appears as a result of equation \ref{eq1} to the right hand side of the term. To be more precise the furthest I got from \ref{eq1} is here: $$ \begin{equation} \label{eq2} \tag{2} \begin{split} \frac{\partial}{\partial \pmb{\delta}} \left(\pmb{\delta}^\intercal \mathbf{A} (\pmb{\delta} \otimes \pmb{\delta} ) \right)& =\mathbf{A} (\pmb{\delta} \otimes \pmb{\delta} ) \\ &+ [\mathbf{I}_K \otimes (\pmb{\delta}^\intercal \mathbf{A})][\mathbf{I}_{K^3}+\mathbf{I}_K \otimes \mathbf{U_{K^2 \times K^2}}][vec(\mathbf{I}_K) \otimes \pmb{\delta}] \end{split} \end{equation} $$ and I am trying to collect the $\pmb{\delta}$ in the second term of the right hand side of above equation on the right hand side like the first term.


For ease of typing, let's use $\,x={\pmb\delta}\,$ as the independent variable.

We'll also use the fact that the Kronecker product of two vectors can be expanded in two ways: $$a\otimes b = (I_a\otimes b)\,a = (a\otimes I_b)\,b$$ where $I_a$ is the identity matrix whose dimensions are compatible with the $a$ vector, while $I_b$ is compatible with the $b$ vector.

The differential and Jacobian of first function can be calculated as $$\eqalign{ y &= x\otimes x \cr dy &= dx\otimes x + x\otimes dx = \big(I\otimes x + x\otimes I\big)\,dx \cr J = \frac{\partial y}{\partial x} &= I\otimes x + x\otimes I \cr }$$ Using the previous result $\big(dy=J\,dx\big),\,$ your second function can be easily dispatched $$\eqalign{ \phi &= x^TAy = y^TA^Tx \cr d\phi &= x^TA\,dy + y^TA^T\,dx \cr &= \big(x^TAJ + y^TA^T\big)\,dx \cr &= \big(Ay + J^TA^Tx\big)^T\,dx \cr \frac{\partial\phi}{\partial x} &= Ay + J^TA^Tx \cr &= A(x\otimes x) + \big(I\otimes x^T + x^T\otimes I\big)A^Tx \cr }$$ The first term is the same between you, Frank, and me.
My second term appears to be the transpose of Frank's and different from yours.


Consider a 3rd order tensor ${\mathcal K}$ which allows you to express the Kronecker product of two vectors $a\in{\mathbb R}^m$, $b\in{\mathbb R}^p$ using dot products $$\eqalign{ c = a\otimes b &= a\cdot{\mathcal K}\cdot b \cr }$$ where all of the tensor components ${\mathcal K}_{i\alpha k}$ are equal zero or one; the nonzero components occurring when $\big\{\,\alpha=p*(i-1)+k\,\big\}$ is satisfied.

Applying it to the first problem $$\eqalign{ f &= x\otimes x \cr\cr df &= x\otimes dx + dx\otimes x \cr &= (I+U)\,(x\otimes dx) \cr &= (I+U)\,(x\cdot{\mathcal K}\cdot dx) \cr\cr \frac{\partial f}{\partial x} &= (I+U)\,(x\cdot{\mathcal K}) \cr }$$ where $U$ is the permutation (aka Commutation) matrix mentioned in the question.

Applying it to the second problem $$\eqalign{ h &= x^TAf \cr &= x\cdot A(x\otimes x) \cr &= x\cdot A(x\cdot{\mathcal K}\cdot x) \cr\cr dh &= A(x\cdot{\mathcal K}\cdot x)\cdot dx + x\cdot A\,df \cr &= \Big(A(x\cdot{\mathcal K}\cdot x) + x\cdot A\,(I+U)\,(x\cdot{\mathcal K)}\Big)\cdot dx \cr \cr \frac{\partial h}{\partial x} &= A(x\cdot{\mathcal K}\cdot x) + (x\cdot A)\,(I+U)\,(x\cdot{\mathcal K)} \cr &= A(x\otimes x) + (x^TA)\,\frac{\partial f}{\partial x} \cr\cr }$$ Notice that the final result does not depend on ${\mathcal K}$ or dot products, so you should be able to derive it via another route that does not utilize these techniques.


For convenience, let the independent variable be $x$, which is easier to type than $\delta$.

Let's use a colon to denote the trace/Frobenius product, i.e. $$A:B={\rm tr}(A^TB)$$

Start by finding the SVD of the constant matrix $$A=(US)V^T = WV^T = \sum_k w_kv_k^T = w_kv_k^T$$ The last expression uses an index summation convention where repeated indices imply summation.

Let's work on the objective function $$\eqalign{ f &= x:w_kv_k^T(x\otimes x) \cr &= (w_k:x)\,(v_k:(x\otimes x)) \cr }$$ Now we need to factor the vectors $$v_k=\sum_j b_{kj}\otimes c_{kj}$$ such that each factor has the same dimensions as $x$. This factorization is a bit unusual, but search for papers by vanLoan and Pitsianis, or Pitsianis' 1997 dissertation.

With these factorizations, we can write $$\eqalign{ f &= (w_k:x)\,(b_{kj}\otimes c_{kj}):(x\otimes x)) \cr &= (w_k:x)(b_{kj}:x)(c_{kj}:x) \cr }$$ We've managed to factor the function into the product of 3 scalars, a form which is very easy to differentiate. $$\eqalign{ df &= \Big[(b_{kj}:x)(c_{kj}:x)w_k + (w_k:x)(c_{kj}:x)b_{kj} + (w_k:x)(b_{kj}:x)c_{kj}\Big]:dx \cr\cr \frac{\partial f}{\partial x} &= (b_{kj}:x)(c_{kj}:x)w_k + (w_k:x)(c_{kj}:x)b_{kj} + (w_k:x)(b_{kj}:x)c_{kj} \cr &= \sum_k \sum_j \,\Big(x^Tb_{kj}\,x^Tc_{kj}\,w_k + x^Tw_k\,x^Tc_{kj}\,b_{kj} + x^Tw_k\,x^Tb_{kj}\,c_{kj}\Big) \cr }$$


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.