I want to compute the derivative of

\begin{align} f(x) = Axx^\top B^\top \label{eqn} \end{align}

with respect to $x$ where $A$ and $B$ are $n\times n$ matrices and $x$ is a (column) vector of size $n \times 1$. By this I mean the derivative of each component of $f(x)$ with respect to each component of $x$.

I can prove that if $$ g(x) = xx^\top $$

Then the derivative can be expressed as, $$ \frac{\partial g}{\partial x} = x \otimes I_n + I_n \otimes x $$ where $I_n$ is the $n\times n$ identity matrix. In here I am vectorizing $xx^\top$ and then taking the derivative with respect to each of the components of $x$.

Question: Is there a way to extend this result to $f(x)$. My gut feeling is that this should be possible. Any thoughts?. If that's not possible how do I go about computing it?

EDIT (After Rodrigo de Azevedo's comment): You are right. But I mean the derivative in the following flattened sense. I hope this makes it a bit clearer.

Let us consider the $2 \times 2$ case. Then $ Y =f(x)$ is a $2 \times 2$ matrix. If I vectorize $f(x)$ then I can view $f$ as, $$ f: \mathbb{R}^2 \to \mathbb{R}^4 $$ More precisely \begin{align} f: \begin{bmatrix} x_1 \\ x_2 \end{bmatrix} \to \begin{bmatrix} Y_{11} \\ Y_{21} \\ Y_{12} \\ Y_{22} \end{bmatrix} \end{align} Then by the symbol $\frac{\partial{f(x)}}{\partial{x}}$ I mean the following: \begin{align} \frac{\partial{f(x)}}{\partial{x}} & = \begin{bmatrix} \frac{\partial{Y_{11}}}{\partial{x_1}} & \frac{\partial{Y_{11}}}{\partial{x_2}} \\ \frac{\partial{Y_{21}}}{\partial{x_1}} & \frac{\partial{Y_{21}}}{\partial{x_2}} \\ \frac{\partial{Y_{12}}}{\partial{x_1}} & \frac{\partial{Y_{12}}}{\partial{x_2}} \\ \frac{\partial{Y_{22}}}{\partial{x_1}} & \frac{\partial{Y_{22}}}{\partial{x_2}} \end{bmatrix} \end{align}

  • $\begingroup$ So $x$ is a column vector, right? $\endgroup$ – saulspatz Mar 20 '18 at 4:43
  • $\begingroup$ Yes $x$ is indeed a column vector. $\endgroup$ – 46_and_2 Mar 20 '18 at 4:44
  • $\begingroup$ Differentiating a matrix-valued function with respect to a vector produces a 3-dimensional "matrix". $\endgroup$ – Rodrigo de Azevedo Mar 20 '18 at 7:47

Clearly, the derivative of $f$ is the linear map $Df(x):v\mapsto Avx^TB^T + Axv^TB^T$. Using the identity $\operatorname{vec}(XYZ)=(Z^T\otimes X)\operatorname{vec}(Y)$, we get \begin{align} \operatorname{vec}(Avx^TB^T + Axv^TB^T) &=\operatorname{vec}(Avx^TB^T) + \operatorname{vec}(Axv^TB^T)\\ &=[(Bx)\otimes A]\operatorname{vec}(v) + [B\otimes(Ax)]\operatorname{vec}(v^T)\\ &=[(Bx)\otimes A+B\otimes(Ax)]v. \end{align} Therefore the Jacobian matrix of $f$ is $(Bx)\otimes A + B\otimes(Ax)$.

| cite | improve this answer | |
  • $\begingroup$ Can you explain a bit more?. How do I see that the derivative of $f$ is the linear map as given in the first equation? $\endgroup$ – 46_and_2 Mar 20 '18 at 6:19
  • 1
    $\begingroup$ @minibuffer It follows from the definition of derivative. $f(x+v)-f(x)=Avx^TB^T+Axv^T+B^T$ plus a second-order term in $v$. Therefore $Df(x)(v)=Avx^TB^T+Axv^T+B^T$. $\endgroup$ – user1551 Mar 20 '18 at 14:17
  • $\begingroup$ The Jacobian can be written, without the commutation matrix, as $$(Bx)\otimes A + B\otimes(Ax)$$ $\endgroup$ – greg Apr 30 '19 at 19:56
  • $\begingroup$ You've calculated $(Ax\otimes B)$. My comment is that $K(Ax\otimes B)$ can be written without the commutator matrix as $(B\otimes Ax)$. $\endgroup$ – greg May 1 '19 at 12:55
  • $\begingroup$ @greg Oh, you are right. Thanks. $\endgroup$ – user1551 May 1 '19 at 14:49

Let matrix-valued function $\mathrm F : \mathbb R^n \to \mathbb R^{n \times n}$ be defined by

$$\mathrm F (\mathrm x) := \mathrm A \mathrm x \mathrm x^\top \mathrm B^\top$$

where $\mathrm A, \mathrm B \in \mathbb R^{n \times n}$ are given. The $(i,j)$-th entry of $\rm F$ is a scalar field given by

$$f_{ij} (\mathrm x) := \mathrm e_i^\top \mathrm A \mathrm x \mathrm x^\top \mathrm B^\top \mathrm e_j = \mathrm a_i^\top \mathrm x \mathrm x^\top \mathrm b_j = \mathrm x^\top \mathrm b_j \mathrm a_i^\top \mathrm x$$

where $\mathrm a_i^\top$ and $\mathrm b_j^\top$ are the $i$-th and $j$-th rows of matrices $\rm A$ and $\rm B$. Hence, the gradient of $f_{ij}$ is

$$\nabla f_{ij} = \color{blue}{\left( \mathrm a_i \mathrm b_j^\top + \mathrm b_j \mathrm a_i^\top \right) \mathrm x}$$

| cite | improve this answer | |

Your intuition is correct, the known derivative $$\eqalign{ G &= xx^T \\ g &= {\rm vec}(G) = x\otimes x \\ \frac{\partial g}{\partial x} &= x\otimes I + I\otimes x \\ }$$ can be used to calculate the new derivative. Just take care to distinguish between a matrix and its flattened vector form.

The calculation is straightforward. $$\eqalign{ F &= Axx^TB^T \\&= AGB^T \\ f &= {\rm vec}(F) \\&= (B\otimes A)\,{\rm vec}(G) \\&= (B\otimes A)\,g \\ \frac{\partial f}{\partial x} &= (B\otimes A)\;\frac{\partial g}{\partial x} \\ &= (B\otimes A)\;(x\otimes I + I\otimes x) \\ &= (Bx\otimes AI) + (BI\otimes Ax) \\ &= (Bx\otimes A) + (B\otimes Ax) \\ }$$

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.