0
$\begingroup$

I'm trying to derive a very simple matrix derivative:

Take the derivative of $\operatorname{Tr}(A' X)$ with respect to $X$.

However, I got two different answers by following different methods.

First Method: vec routine: $\operatorname{Tr}(A' X) = vec(A)' vec(X)$, so that $d(\operatorname{Tr}(A'X)) = vec(A)' dvec(X)$, and therefore the derivative is $vec(A)'$.

Second Method: element-wise :

$$\frac{\partial \operatorname{Tr}(A^{T}X)}{\partial Xij}% =\operatorname{Tr}(A^{T}E_{ij})=\operatorname{Tr}(e_{j}^{T}A^{T}e_{i})=A_{ij}$$

And therefore the derivative is $A$.

The first method is following the idea by Steven W. Nydick in this material.

And the Second one is introduced in 'Kronecker product and its application'.

Why these two methods give me different answers? Which one is correct?

ps: I guess I am little bit confused by how the first method is transforming the matrix derivative to vector derivative. Steven's material says that the matrix derivative of $f(X)$ w.r.t $X$ is equal to the derivative of $vec(f(X))$ w.r.t $vec(X)$.

Thanks.

P.s. Thanks to the help of user1551, I see that it is a layout transform issue. But I could not see how the transform is between $d(f(X))/d(X)$ and $d(vec(f(X))/d(vec(X))$, if $f$ returns a matrix.

$\endgroup$
2
$\begingroup$

Both are correct. They only differ in the layout of the derivative. In the first method, you group the partial derivatives into a vector; in the second one, you group them into a matrix. When people do usual multivariate calculus, they tend to use the first form; when they do matrix calculus, they tend to adopt the second form.

$\endgroup$
3
  • $\begingroup$ thanks for the explanation. But how can I transform between the two layout? Since the relation is kind of obvious but still not obvious enough to write down using permutation matrix I guess..especially when it comes to take matrix derivative w.r.t matrix $\endgroup$ – Jack2019 Mar 30 '13 at 7:06
  • $\begingroup$ @SoManyProblems Do you know what $\operatorname{vec}(A)$ means? You don't use a permutation matrix and matrix-vector multiplication to get $\operatorname{vec}(A)$ from $A$. Instead, you stack first column of $A$ on top of the second column, the second one on top of the third one and so on. $\endgroup$ – user1551 Mar 30 '13 at 7:26
  • $\begingroup$ yes I understand that. and I of course I know how to transform between vec(X) and X, but you see it is more complicated here. I can't figure out how d(f(X))/dX and d(vec(f(X)))/d(vec(X)) can be transformed back and forth easily. I understand it is just a layout issue, but I could not work it out... $\endgroup$ – Jack2019 Mar 30 '13 at 14:59
1
$\begingroup$

That's correct. The presentation basically defines the derivative of a function of a matrix w.r.t. a matrix as df(X)/dvec(X)^T, and if you take the derivative of a scalar w.r.t. a transposed vector, you get a transposed vector as the result.

The reason that the presentation (and corresponding book) does this (and why it's done in matrix algebra in general) is so that the derivative of a matrix w.r.t. a matrix is able to be organized:

da/dX (where a is scalar and X is matrix) --> matrix

dv/dX (where v is vector and X is matrix) --> ???

dY/dX (where Y is matrix and X is matrix) --> ???

By treating X as a systematically organized vector, you can simply use vector calculus theory and avoid the nuisance of figuring out how to organize the resulting derivative.

He did post the other version: da/dX --> matrix for trace operators: Here

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.