Derivative of product of matrix by vector Let $\boldsymbol{\beta} := (\beta_1, \ldots, \beta_p)^T$ and $\boldsymbol{X}$ be a matrix of dimension $n \times p$. I'd like to compute the derivative
$$\nabla_{\beta} \left( X \beta \right)$$
 A: $X\beta$ is $n$-dimensional vector with components
$$
(X\beta)_i = \sum_{j=1}^{p}X_{i,j}\beta_j, \quad i=1,\dots,n.
$$
so each of its $n$ components is function of $\beta_1,\dots,\beta_p$. Partial derivative of this vector with respect to component $\beta_k$ is obtained simply by partial differentiation of its components:
$$
\left(\frac{\partial X\beta}{\partial\beta_k}\right)_i = \frac{\partial (X\beta)_i}{\partial\beta_k} = \frac{\partial}{\partial\beta_k}\sum_{j=1}^{p}X_{i,j}\beta_j = X_{i,k}, \quad k = 1,\dots,p.
$$
So partial derivatives $\frac{\partial X\beta}{\partial\beta_k}$ are $n$-dimensional vectors. These vectors are actually columns of matrix $\frac{\partial X\beta}{\partial\beta}$ you are looking for:
$$
\left(\frac{\partial X\beta}{\partial\beta}\right)_{i,k} = \left(\frac{\partial X\beta}{\partial\beta_k}\right)_i = X_{i,k},
$$
or just $\frac{\partial X\beta}{\partial\beta} = X$ which is intuitively clear. 

For example, when $n=1$ you get
$$
X\beta = \begin{pmatrix}X_1 & \dots & X_p\end{pmatrix} \begin{pmatrix}\beta_1 \\ \vdots \\ \beta_p\end{pmatrix} = X_1\beta_1+\dots+X_p\beta_p;
$$
$$
\frac{\partial X\beta}{\partial\beta} = \begin{pmatrix}\frac{\partial X\beta}{\partial\beta_1} & \dots & \frac{\partial X\beta}{\partial\beta_p}\end{pmatrix} = \begin{pmatrix}X_1 & \dots & X_p\end{pmatrix} = X.
$$
