Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I was reading this to review the derivation of the ordinary least squares estimator but I'm having trouble differentiating (4). Can someone please help explain why

$ \dfrac{\partial (\hat{\beta}'X'X\hat{\beta})}{\partial \hat{\beta}} = 2X'X\hat{\beta} $

I understand all the other steps.


share|cite|improve this question
up vote 0 down vote accepted

Notice first that you might guess this formula because if $X$ and $\hat{\beta}$ are $1 \times 1$ then it reduces to the formula for the derivative of $x^2$ from calculus.

Let's derive a multivariable product rule that will help us here. Suppose $f:\mathbb{R}^n \to \mathbb{R}$ and suppose $$f(x) = \langle g(x), h(x) \rangle$$ for some functions $g:\mathbb{R}^n \to \mathbb{R}^m$ and $h:\mathbb{R}^n \to \mathbb{R}^m$. Then if $\Delta x \in \mathbb{R}^n$ is small (and $g$ and $h$ are differentiable at $x$), we have $$\begin{align*} f(x + \Delta x) &\approx \langle g(x) + g'(x) \Delta x, h(x) + h'(x) \Delta x \rangle \\ &= \langle g(x),h(x) \rangle + \langle g(x), h'(x) \Delta x \rangle + \langle g'(x) \Delta x, h(x) \rangle + \langle g'(x) \Delta x, h'(x) \Delta x \rangle \\ &\approx \langle g(x),h(x) \rangle + \langle g(x), h'(x) \Delta x \rangle +\langle g'(x) \Delta x, h(x) \rangle \\ &= \langle g(x),h(x) \rangle + \langle h'(x)^T g(x), \Delta x \rangle + \langle g'(x)^T h(x), \Delta x \rangle \\ &= f(x) + \langle h'(x)^T g(x) + g'(x)^T h(x), \Delta x \rangle. \end{align*}$$ Comparing this result with $$f(x + \Delta x) \approx f(x) + \langle \nabla f(x), \Delta x \rangle$$ we discover that $$\nabla f(x) = h'(x)^T g(x) + g'(x)^T h(x).$$ This is our product rule. (I'm using the convention that the gradient is a column vector, which is not completely standard.)

Now let $g(x) = Ax$ for some matrix $A$. So $g'(x) = A$. What's the gradient of the function $$\begin{align*} f(x) &= \langle g(x),g(x) \rangle \\ &= \langle Ax, Ax \rangle \\ &= x^T A^T A x \quad \text{?} \end{align*}$$

By our product rule the answer is $$\begin{align*} \nabla f(x) &= 2g'(x)^T g(x) \\ &= 2 A^T A x. \end{align*}$$

This is the result that you wanted to derive.

share|cite|improve this answer

First we need to note that $X'X$ is a symmetric matrix as $(X'X)'=X'(X')'=X'X$.

Now,Using basic matrix differentiation,we know that for symmetric matrix $A$,$\dfrac{\partial (x'Ax)}{\partial x} = 2Ax = 2x'A$ where $x$ is a vector and dimensions are proper. hence result follows trivially.

Note:Actually we can say more,if $A$ is any $n \times n$ matrix, $x$ is $n \times 1$ vector,then,

$ \dfrac{\partial (x'Ax)}{\partial x} = x'(A'+A)$

Proof:$$ x'Ax = \sum_{j=1}^n \sum_{i=1}^n a_{ij}x_ix_j$$ Differentiating wrt to $x_k$,we get, $$\dfrac{\partial (x'Ax)}{\partial x_k}= \sum_{j=1}^n a_{kj}x_j + \sum_{i=1}^n a_{ik}x_i\ ; \forall k = 1,...,n$$ hence,$$\dfrac{\partial (x'Ax)}{\partial x} = x'A'+x'A$$

Hence result follows.

More details on Matrix differentiation can be found on .


share|cite|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.