function: $ f = A\cdot b$

gradient: $ \frac{\partial f}{\partial A} = b^\top \otimes \mathbb{I} $

$A$ is a matrix e.g. shape 3x3

$b$ is a vector e.g. shape 3x1

$ \otimes $ is Kronecker product

$ \mathbb{I} $ is identity matrix

My questions are:

  1. what is the shape of gradient: $ \frac{\partial f}{\partial A}$
  2. what is the definition of vector-value function with respect to matrix, since Matrix Calculus Wikipedia doesn't have this type

Thanks a lot for helping.


The gradient ${\cal G}$ is a third-order tensor, so its shape is $(3\times 3\times 3)$.
This easiest to see using index notation. $$\eqalign{ f_i &= A_{ij}b_j \\ df_i &= dA_{ij}b_j \\ \frac{\partial f_i}{\partial A_{mn}} &= \bigg(\frac{\partial A_{ij}}{\partial A_{mn}}\bigg)\,b_j \\ &= \big(\delta_{im}\delta_{jn}\big)\,b_j \\ &= \delta_{im}b_n \\ &= {\cal G}_{imn} \\ }$$ This assumes that the elements of $A$ are independent, so that $\Big(\frac{\partial A_{ij}}{\partial A_{mn}}\Big)$ equals zero
unless $(i=m)\,\&\,(j=n)\,-$ which are the same conditions enforced by the delta symbols.

In order to write this without resorting to tensor/index notation, many authors flatten the $(3\times 3)$ $A$ matrix into a $(9\times 1)$ vector using the Kronecker-vec relationship.

Their derivation goes like so. $$\eqalign{ f &= A\,b \\ df &= dA\,b \\ {\rm vec}(df) &= {\rm vec}(I\,dA\,b) \\ df &= (b^T\otimes I)\,da \\ \frac{\partial f}{\partial a} &= (b^T\otimes I) \;= G \in {\mathbb R}^{3\times 9} \\ }$$ Then they call the $G$ matrix "the gradient" $-$ but it's really a flattened representation of the ${\cal G}$ tensor.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.