# First order differential rule for matrices

What is the difference between:

1) $\frac{\partial (\textbf{x}^{T}A)}{\partial \textbf{x}}$

and

2) $\frac{\partial (A\textbf{x})}{\partial \textbf{x}}$

where A is a nxn matrix and x is a n sized column vector.

Using the definition of a Jacobian on wikipedia (http://en.wikipedia.org/wiki/Jacobian_matrix_and_determinant)

My answers are:

1) $A^{T}$

2) $A$

However this is not correct because:

$$\textbf{x}^{T}A = (A^{T}\textbf{x})^{T}$$

performing $\frac{\partial}{\partial \textbf{x}}$ on each side results

$$A^T=A$$

which is not true. Does $\frac{\partial}{\partial \textbf{x}}$ become $\frac{\partial}{\partial \textbf{x}^{T}}$ when moved inside the transpose on the right side?

-
add comment

## 2 Answers

There is actually a hugh difference. For one, when you say $\frac{\partial (x^TA)}{\partial x}$, the numerator is a $1 \times n$ vector. Whereas in case of $\frac{\partial (Ax)}{\partial x}$, the numerator is a $n \times 1$ vector.

Look up this [wiki][1] page for details on how matrix derivatives should be done and the rules.

You may want to look up http://matrixcookbook.com/ which gives you a lot of matrix identities like these.

-
add comment

Thank for the matrixcookbook, its a useful source. I found the identity which confuses me:

$\frac{\partial \textbf{x}^{T} B \textbf{x}}{\partial \textbf{x}} = (B + B^{T})\textbf{x}$

via product rule:

$\frac{\partial \textbf{x}^{T} B \textbf{x}}{\partial \textbf{x}} = \frac{\partial \textbf{x}^{T}}{\partial \textbf{x}}B\textbf{x} + \textbf{x}^{T}\frac{\partial B \textbf{x}}{\partial \textbf{x}}$

At this point, I'm stuck because I'm not sure if $\frac{\partial \textbf{x}^{T}}{\partial \textbf{x}}$ is identity $I$.

I assume this definition of the Jacobian:

$\begin{bmatrix} \dfrac{\partial y_1}{\partial x_1} & \cdots & \dfrac{\partial y_1}{\partial x_n} \\ \vdots & \ddots & \vdots \\ \dfrac{\partial y_m}{\partial x_1} & \cdots & \dfrac{\partial y_m}{\partial x_n} \end{bmatrix}$

And if I write $\frac{\partial B \textbf{x}}{\partial \textbf{x}}$ component by component with the definition above, it becomes simply $B$.

Thus,

$\frac{\partial \textbf{x}^{T} B \textbf{x}}{\partial \textbf{x}} = B\textbf{x} + \textbf{x}^{T}B$

which is not correct, as you stated the left side is a nx1 vector and the right side is a nx1 vector. I'm not sure where my faulty assumption is.

-
Note that $x^TBx$ is just a scalar. All you need to know is if $f:\mathbb{R}^n \rightarrow \mathbb{R}$, and $x \in \mathbb{R}^n$, then $\frac{\partial f}{\partial x}$ is a column vector with the $i^{th}$ element being $\frac{\partial f}{\partial x_i}$. –  user17762 Nov 17 '10 at 5:56
The simple rules for one dimensional derivative doesn't carry forward in a straightforward way when we have vectors in the numerator of the derivative. For instance $\frac{\partial (x^Tx)}{\partial x} \neq (\frac{\partial x^T}{\partial x}) x + x^T (\frac{\partial x}{\partial x})$. –  user17762 Nov 17 '10 at 6:00
What would be the correct way of writing $\frac{\partial x^{T}x}{\partial x}$? The matrixcookbook seems to take the identity I mentioned above as assumed. I know the product rule still holds for matrix differentials, but there always seems to be a transpose I'm missing. –  Christopher Dorian Nov 17 '10 at 7:02
add comment