Matrix Differentiation proof II Someone asks a different part of this question here. I wonder how can we derive from (46) to (47)? Especially how to think the way to get the transpose? 
 A: I think the confusion might be coming from the row/column conventions for matrix calculus, which are often not well-explained in courses.
The important thing to remember is that matrix-vector multiplication can be defined per-component by
$$ [Ax]_{i} = \sum_j a_{ij} x_j $$
And, for the transpose, we get $$ [A^Tx]_{i} = \sum_j a_{ji} x_j $$
So, (46) can be rewritten as:
$$ \frac{\partial \alpha}{\partial x_k} = \sum_j a_{kj}x_j + \sum_i a_{ik}x_i = [Ax]_k + [A^Tx]_k $$
Clearly, the quantity $\frac{\partial \alpha}{\partial x}$ should be a vector with components $\frac{\partial \alpha}{\partial x_k}$.
Now, we are looking at the Jacobian of $\alpha(x)=\sum_{i,j} a_{ij}x_ix_j$, so it should be a row vector.
However, we have written the above as components of column vectors. So it is better to write: $$ \frac{\partial \alpha}{\partial x_k} = \sum_j a_{kj}x_j + \sum_i a_{ik}x_i = [x^TA^T]_k + [x^TA]_k $$
Note this makes no difference to the components since a transposed vector has the same components for a given index.
However, now we correctly get a row vector when we generalize this as a matrix derivative: $$ \frac{\partial \alpha}{\partial x} = x^TA^T + x^TA = x^T(A^T+A) $$
by simply removing the component index.
Hopefully this way of looking at it helps!
