Derivative of $a^{T}Xb$ with respect to b, where $a, b$ is a $d$-dim vector and $X$ is a $d\times d$ matrix Since the derivative of $a^{T}Xb$ with respect to a is $Xb$, I was wondering how do I solve the derivative of $a^{T}Xb$ with respect to $b$?
 A: It's easy to get confused, due to  a lack of clear and uniform conventions about what "the derivative" of a multivariate function $f(b)$ should mean.
One possibility is the gradient, defined to be the vector $\nabla f$ so that, for all vectors $\delta b$ representing a change in $b$, $\langle \nabla f, \delta b\rangle$ gives the directional derivative in the $\delta b$ direction:
$$\langle \nabla f, \delta b\rangle = \lim_{t\to 0} \frac{d}{dt} f(b + t\delta b).$$
The right-hand side in your case is $a^TX\delta b$, so $\nabla f = X^Ta$.
Another possibility is the Jacobian/differential/push-forward $Jf$, which is the linear map from an infinitesimal change $\delta b$ in $b$ to an infinitesimal change in $f$. It's defined by
$$[J f]\delta b = \lim_{t\to 0} \frac{d}{dt} f(b + t\delta b)$$
for all $\delta b$; here we see that for your $f$ the Jacobian is the row vector
$$Jf = a^TX.$$
Unfortunately it's very common (especially in applied math) to conflate the gradient and Jacobian (and don't even get me started about the hodge-podge of conventions that surround differentiation with respect to matrix variables), and you have to infer from context what kind of object you need.
