Differentiating Kronecker product of a vector with respect to itself

Question

I am trying to differentiate the Kronecker product of a vector with respect to itself: $$ \frac{\partial}{\partial \pmb{\delta}} (\pmb{\delta} \otimes \pmb{\delta} ) $$ That is part of this larger differentiation: $$ \begin{equation} \label{eq1} \tag{1} \frac{\partial}{\partial \pmb{\delta}} \left(\pmb{\delta}^\intercal \mathbf{A} (\pmb{\delta} \otimes \pmb{\delta} ) \right) \end{equation} $$ Where:

-$\pmb{\delta}_{K \times 1}$ is a vector and $\pmb{\delta}^{\intercal}_{1\times K}$ is its transpose.

-$ \mathbf{A}_{K \times K^2}$ is constant matrix that is independent from $\pmb{\delta}$ ($ \partial \mathbf{A}/ \partial \pmb{\delta}=0) $.

Using definitions that can be found in Matrix Calculus textbooks, I have simplified it as follows: $$ \begin{split} \frac{\partial}{\partial \pmb{\delta}} (\pmb{\delta} \otimes \pmb{\delta} ) &=\frac{\partial \pmb{\delta}}{\partial \pmb{\delta}} \otimes \pmb{\delta}+ [\mathbf{I}_K \otimes \mathbf{U_{K^2 \times K^2}}][\frac{\partial \pmb{\delta}}{\partial \pmb{\delta}} \otimes \pmb{\delta}][\mathbf{I}_1 \otimes \mathbf{I}_{1\times 1} ]\\ &=vec(\mathbf{I}_K) \otimes \pmb{\delta}+[\mathbf{I}_K \otimes \mathbf{U_{K^2 \times K^2}}][vec(\mathbf{I}_K) \otimes \pmb{\delta}]\\ &=[\mathbf{I}_{K^3}+\mathbf{I}_K \otimes \mathbf{U_{K^2 \times K^2}}][vec(\mathbf{I}_K) \otimes \pmb{\delta}] \end{split} $$ Where:

• $\mathbf{I}_n$ is $n \times n$ unit matrix
• $vec(\mathbf{B})$ is the classic vectorization operator.
• $\mathbf{U}_{r \times s}$ is the permutation matrix as it is defined in classic matrix calculus textbooks (e.g. here is shown with $T_{m,n}$).

Can the above equation get simplified further? My goal is to collect whatever form of $ \pmb{\delta}$ that appears as a result of equation \ref{eq1} to the right hand side of the term. To be more precise the furthest I got from \ref{eq1} is here: $$ \begin{equation} \label{eq2} \tag{2} \begin{split} \frac{\partial}{\partial \pmb{\delta}} \left(\pmb{\delta}^\intercal \mathbf{A} (\pmb{\delta} \otimes \pmb{\delta} ) \right)& =\mathbf{A} (\pmb{\delta} \otimes \pmb{\delta} ) \\ &+ [\mathbf{I}_K \otimes (\pmb{\delta}^\intercal \mathbf{A})][\mathbf{I}_{K^3}+\mathbf{I}_K \otimes \mathbf{U_{K^2 \times K^2}}][vec(\mathbf{I}_K) \otimes \pmb{\delta}] \end{split} \end{equation} $$ and I am trying to collect the $\pmb{\delta}$ in the second term of the right hand side of above equation on the right hand side like the first term.

Answer 1

For ease of typing, let's use $\,x={\pmb\delta}\,$ as the independent variable.

We'll also use the fact that the Kronecker product of two vectors can be expanded in two ways: $$a\otimes b = (I_a\otimes b)\,a = (a\otimes I_b)\,b$$ where $I_a$ is the identity matrix whose dimensions are compatible with the $a$ vector, while $I_b$ is compatible with the $b$ vector.

The differential and Jacobian of first function can be calculated as $$\eqalign{ y &= x\otimes x \cr dy &= dx\otimes x + x\otimes dx = \big(I\otimes x + x\otimes I\big)\,dx \cr J = \frac{\partial y}{\partial x} &= I\otimes x + x\otimes I \cr }$$ Using the previous result $\big(dy=J\,dx\big),\,$ your second function can be easily dispatched $$\eqalign{ \phi &= x^TAy = y^TA^Tx \cr d\phi &= x^TA\,dy + y^TA^T\,dx \cr &= \big(x^TAJ + y^TA^T\big)\,dx \cr &= \big(Ay + J^TA^Tx\big)^T\,dx \cr \frac{\partial\phi}{\partial x} &= Ay + J^TA^Tx \cr &= A(x\otimes x) + \big(I\otimes x^T + x^T\otimes I\big)A^Tx \cr }$$ The first term is the same between you, Frank, and me.
My second term appears to be the transpose of Frank's and different from yours.

Answer 2

Consider a 3rd order tensor ${\mathcal K}$ which allows you to express the Kronecker product of two vectors $a\in{\mathbb R}^m$, $b\in{\mathbb R}^p$ using dot products $$\eqalign{ c = a\otimes b &= a\cdot{\mathcal K}\cdot b \cr }$$ where all of the tensor components ${\mathcal K}_{i\alpha k}$ are equal zero or one; the nonzero components occurring when $\big\{\,\alpha=p*(i-1)+k\,\big\}$ is satisfied.

Applying it to the first problem $$\eqalign{ f &= x\otimes x \cr\cr df &= x\otimes dx + dx\otimes x \cr &= (I+U)\,(x\otimes dx) \cr &= (I+U)\,(x\cdot{\mathcal K}\cdot dx) \cr\cr \frac{\partial f}{\partial x} &= (I+U)\,(x\cdot{\mathcal K}) \cr }$$ where $U$ is the permutation (aka Commutation) matrix mentioned in the question.

Applying it to the second problem $$\eqalign{ h &= x^TAf \cr &= x\cdot A(x\otimes x) \cr &= x\cdot A(x\cdot{\mathcal K}\cdot x) \cr\cr dh &= A(x\cdot{\mathcal K}\cdot x)\cdot dx + x\cdot A\,df \cr &= \Big(A(x\cdot{\mathcal K}\cdot x) + x\cdot A\,(I+U)\,(x\cdot{\mathcal K)}\Big)\cdot dx \cr \cr \frac{\partial h}{\partial x} &= A(x\cdot{\mathcal K}\cdot x) + (x\cdot A)\,(I+U)\,(x\cdot{\mathcal K)} \cr &= A(x\otimes x) + (x^TA)\,\frac{\partial f}{\partial x} \cr\cr }$$ Notice that the final result does not depend on ${\mathcal K}$ or dot products, so you should be able to derive it via another route that does not utilize these techniques.

Answer 3

For convenience, let the independent variable be $x$, which is easier to type than $\delta$.

Let's use a colon to denote the trace/Frobenius product, i.e. $$A:B={\rm tr}(A^TB)$$

Start by finding the SVD of the constant matrix $$A=(US)V^T = WV^T = \sum_k w_kv_k^T = w_kv_k^T$$ The last expression uses an index summation convention where repeated indices imply summation.

Let's work on the objective function $$\eqalign{ f &= x:w_kv_k^T(x\otimes x) \cr &= (w_k:x)\,(v_k:(x\otimes x)) \cr }$$ Now we need to factor the vectors $$v_k=\sum_j b_{kj}\otimes c_{kj}$$ such that each factor has the same dimensions as $x$. This factorization is a bit unusual, but search for papers by vanLoan and Pitsianis, or Pitsianis' 1997 dissertation.

With these factorizations, we can write $$\eqalign{ f &= (w_k:x)\,(b_{kj}\otimes c_{kj}):(x\otimes x)) \cr &= (w_k:x)(b_{kj}:x)(c_{kj}:x) \cr }$$ We've managed to factor the function into the product of 3 scalars, a form which is very easy to differentiate. $$\eqalign{ df &= \Big[(b_{kj}:x)(c_{kj}:x)w_k + (w_k:x)(c_{kj}:x)b_{kj} + (w_k:x)(b_{kj}:x)c_{kj}\Big]:dx \cr\cr \frac{\partial f}{\partial x} &= (b_{kj}:x)(c_{kj}:x)w_k + (w_k:x)(c_{kj}:x)b_{kj} + (w_k:x)(b_{kj}:x)c_{kj} \cr &= \sum_k \sum_j \,\Big(x^Tb_{kj}\,x^Tc_{kj}\,w_k + x^Tw_k\,x^Tc_{kj}\,b_{kj} + x^Tw_k\,x^Tb_{kj}\,c_{kj}\Big) \cr }$$