Derivative of product of matrix by vector

Question

Let $\boldsymbol{\beta} := (\beta_1, \ldots, \beta_p)^T$ and $\boldsymbol{X}$ be a matrix of dimension $n \times p$. I'd like to compute the derivative

$$\nabla_{\beta} \left( X \beta \right)$$

Answer 1

$X\beta$ is $n$-dimensional vector with components $$ (X\beta)_i = \sum_{j=1}^{p}X_{i,j}\beta_j, \quad i=1,\dots,n. $$ so each of its $n$ components is function of $\beta_1,\dots,\beta_p$. Partial derivative of this vector with respect to component $\beta_k$ is obtained simply by partial differentiation of its components: $$ \left(\frac{\partial X\beta}{\partial\beta_k}\right)_i = \frac{\partial (X\beta)_i}{\partial\beta_k} = \frac{\partial}{\partial\beta_k}\sum_{j=1}^{p}X_{i,j}\beta_j = X_{i,k}, \quad k = 1,\dots,p. $$ So partial derivatives $\frac{\partial X\beta}{\partial\beta_k}$ are $n$-dimensional vectors. These vectors are actually columns of matrix $\frac{\partial X\beta}{\partial\beta}$ you are looking for: $$ \left(\frac{\partial X\beta}{\partial\beta}\right)_{i,k} = \left(\frac{\partial X\beta}{\partial\beta_k}\right)_i = X_{i,k}, $$ or just $\frac{\partial X\beta}{\partial\beta} = X$ which is intuitively clear.

---

For example, when $n=1$ you get $$ X\beta = \begin{pmatrix}X_1 & \dots & X_p\end{pmatrix} \begin{pmatrix}\beta_1 \\ \vdots \\ \beta_p\end{pmatrix} = X_1\beta_1+\dots+X_p\beta_p; $$ $$ \frac{\partial X\beta}{\partial\beta} = \begin{pmatrix}\frac{\partial X\beta}{\partial\beta_1} & \dots & \frac{\partial X\beta}{\partial\beta_p}\end{pmatrix} = \begin{pmatrix}X_1 & \dots & X_p\end{pmatrix} = X. $$