First order differential rule for matrices What is the difference between:
1) $\frac{\partial (\textbf{x}^{T}A)}{\partial \textbf{x}}$
and
2) $\frac{\partial (A\textbf{x})}{\partial \textbf{x}}$
where A is a nxn matrix and x is a n sized column vector.
Using the definition of a Jacobian on wikipedia (http://en.wikipedia.org/wiki/Jacobian_matrix_and_determinant)
My answers are:
1) $A^{T}$
2) $A$
However this is not correct because:
$$\textbf{x}^{T}A = (A^{T}\textbf{x})^{T}$$
performing $\frac{\partial}{\partial \textbf{x}}$ on each side results
$$A^T=A$$
which is not true.
Does $\frac{\partial}{\partial \textbf{x}}$  become $\frac{\partial}{\partial \textbf{x}^{T}}$ when moved inside the transpose on the right side?
 A: There is actually a hugh difference. For one, when you say $\frac{\partial (x^TA)}{\partial x}$, the numerator is a $1 \times n$ vector. Whereas in case of $\frac{\partial (Ax)}{\partial x}$, the numerator is a $n \times 1$ vector.
Look up this [wiki][1] page for details on how matrix derivatives should be done and the rules.
You may want to look up http://matrixcookbook.com/ which gives you a lot of matrix identities like these.
[1]: http://en.wikipedia.org/wiki/Matrix_calculus page
A: Thank for the matrixcookbook, its a useful source.  I found the identity which confuses me:
$\frac{\partial \textbf{x}^{T} B \textbf{x}}{\partial \textbf{x}} = (B + B^{T})\textbf{x}$
via product rule:
$\frac{\partial \textbf{x}^{T} B \textbf{x}}{\partial \textbf{x}} = \frac{\partial \textbf{x}^{T}}{\partial \textbf{x}}B\textbf{x} + \textbf{x}^{T}\frac{\partial B \textbf{x}}{\partial \textbf{x}}$
At this point, I'm stuck because I'm not sure if $\frac{\partial \textbf{x}^{T}}{\partial \textbf{x}}$ is identity $I$.  
I assume this definition of the Jacobian:
$\begin{bmatrix} \dfrac{\partial y_1}{\partial x_1} & \cdots & \dfrac{\partial y_1}{\partial x_n} \\ \vdots & \ddots & \vdots \\ \dfrac{\partial y_m}{\partial x_1} & \cdots & \dfrac{\partial y_m}{\partial x_n}  \end{bmatrix}$
And if I write $\frac{\partial B \textbf{x}}{\partial \textbf{x}}$ component by component with the definition above, it becomes simply $B$.
Thus,
$\frac{\partial \textbf{x}^{T} B \textbf{x}}{\partial \textbf{x}} = B\textbf{x} + \textbf{x}^{T}B$
which is not correct, as you stated the left side is a nx1 vector and the right side is a nx1 vector.  I'm not sure where my faulty assumption is.
