Derivative of product of matrix by vector

Let $\boldsymbol{\beta} := (\beta_1, \ldots, \beta_p)^T$ and $\boldsymbol{X}$ be a matrix of dimension $n \times p$. I'd like to compute the derivative

$$\nabla_{\beta} \left( X \beta \right)$$

• This is a linear equation, so it's very straightforward \eqalign{&y=X\beta,\,\,\,&dy=X\,d\beta,\,\,\,\,&\frac{\partial y}{\partial\beta}=X}
– greg
Jun 6, 2018 at 20:43

$X\beta$ is $n$-dimensional vector with components $$(X\beta)_i = \sum_{j=1}^{p}X_{i,j}\beta_j, \quad i=1,\dots,n.$$ so each of its $n$ components is function of $\beta_1,\dots,\beta_p$. Partial derivative of this vector with respect to component $\beta_k$ is obtained simply by partial differentiation of its components: $$\left(\frac{\partial X\beta}{\partial\beta_k}\right)_i = \frac{\partial (X\beta)_i}{\partial\beta_k} = \frac{\partial}{\partial\beta_k}\sum_{j=1}^{p}X_{i,j}\beta_j = X_{i,k}, \quad k = 1,\dots,p.$$ So partial derivatives $\frac{\partial X\beta}{\partial\beta_k}$ are $n$-dimensional vectors. These vectors are actually columns of matrix $\frac{\partial X\beta}{\partial\beta}$ you are looking for: $$\left(\frac{\partial X\beta}{\partial\beta}\right)_{i,k} = \left(\frac{\partial X\beta}{\partial\beta_k}\right)_i = X_{i,k},$$ or just $\frac{\partial X\beta}{\partial\beta} = X$ which is intuitively clear.
For example, when $n=1$ you get $$X\beta = \begin{pmatrix}X_1 & \dots & X_p\end{pmatrix} \begin{pmatrix}\beta_1 \\ \vdots \\ \beta_p\end{pmatrix} = X_1\beta_1+\dots+X_p\beta_p;$$ $$\frac{\partial X\beta}{\partial\beta} = \begin{pmatrix}\frac{\partial X\beta}{\partial\beta_1} & \dots & \frac{\partial X\beta}{\partial\beta_p}\end{pmatrix} = \begin{pmatrix}X_1 & \dots & X_p\end{pmatrix} = X.$$