Gradient of $M \mapsto x^T M x$

Suppose we have a linear scalar field $$f : \mathbb{R}^{n \times m} \to \mathbb{R}$$ defined by

$$f(M) := x^T M x$$

where $$x \in \mathbb{R}^n$$. What is the gradient of $$f(M)$$ with respect to $$M$$? I think it is $$xx^T$$, but why?

• Since $f$ is linear, its differential is equal to $f$... Commented Nov 1, 2018 at 0:35
• Can you write down the calculation for me?
– user494522
Commented Nov 1, 2018 at 0:38
• Related Commented Apr 26, 2021 at 1:48

$$f(M)=\sum_{i,j}x_iM_{i,j}x_j.$$ Therefore $$\frac{\partial f(M)}{\partial M_{i,j}} = x_ix_j = (xx^T)_{i,j}.$$

• Could you explain the last step?
– user494522
Commented Nov 1, 2018 at 0:40
• What is $xx^T$ for you? Commented Nov 1, 2018 at 0:41

The strategy is to write the expression as a scalar using index notation, take the derivative, and re-write in matrix form.

Note that to write the function as a summation of matrices we have to write just one scalar as a matrix multiplication because the function is scalar: $$f(M)= [x^TMx]_{11}= \sum_i x_{i1}[Mx]_{i1}$$ At first summation, we have $$x_{i1}$$ because we have $$x^T$$ as the first term.

Now expand $$[Mx]_{i1}$$ $$f(M)= [x^TMx]_{11}= \sum_i x_{i1}\sum_j M_{ij}x_{j1}=\sum_i \sum_j x_{i1} M_{ij}x_{j1}$$

Now take the derivative with respect to $$M_{ij}$$

$$\frac{\partial f(M)}{\partial M_{ij}}=\sum_i \sum_j x_{i1}x_{j1}$$

Looking at the indices, we can see that

$$\sum_i \sum_j x_{i1}x_{j1}=\sum_i \sum_j x_{j1}x_{i1}=[xx^T]_{ji}$$

Therefore,

$$\frac{\partial f(M)}{\partial M_{ij}}=xx^T$$