How to take the gradient of the quadratic form?

Question

It's stated that the gradient of:

$\frac{1}{2}x^TAx - B^Tx +C$

is

$\frac{1}{2}A^Tx + \frac{1}{2}Ax - b$

How do you grind out this equation? Or specifically, how do you get from $x^TAx$ to $A^Tx + Ax$?

Answer 1

The only thing you need to remember/know is that $$\dfrac{\partial (x^Ty)}{\partial x} = y$$ and the chain rule, which goes as $$\dfrac{d(f(x,y))}{d x} = \dfrac{\partial (f(x,y))}{\partial x} + \dfrac{\partial y^T}{\partial x} \dfrac{\partial (f(x,y))}{\partial y}$$ Hence, $$\dfrac{\partial (b^Tx)}{\partial x} = \dfrac{\partial (x^Tb)}{\partial x} = b$$

$$\dfrac{\partial (x^TAx)}{\partial x} = \dfrac{\partial (x^Ty)}{\partial x} + \dfrac{\partial y^T}{\partial x} \dfrac{\partial (x^Ty)}{\partial y}$$ where $y = Ax$.

$$\dfrac{\partial (x^TAx)}{\partial x} = \dfrac{\partial (x^Ty)}{\partial x} + \dfrac{\partial y^T}{\partial x} \dfrac{\partial (x^Ty)}{\partial y} = y + \dfrac{\partial (x^TA^T)}{\partial x} \dfrac{\partial (x^Ty)}{\partial y} = y + A^Tx = (A+A^T)x$$