About matrix derivative Suppose $A$ is a matrix with order n*n.
we have the following equity but I don't know why.
$f(x)=\frac{1}{2}x^TAx-b^Tx$.
then
$f'(x)=\frac{1}{2}A^Tx+\frac{1}{2}Ax-b$
Is there any rule like scalar function's derivative?
thanks.
 A: Another way to see this is the following :
By definition, $f'(x)$ satisfies :
$$ f(x+h) = \color{blue}{f(x)} + h^{\top}\color{red}{f'(x)} + o(\Vert h \Vert)$$
So, by expanding $f(x+h)$, we get :
$$
\begin{align*}
f(x+h) &= {} \frac{1}{2}(x+h)^{\top}A(x+h)-b^{\top}(x+h) \\
 &= \Big[ \frac{1}{2} x^{\top}Ax - b^{\top}x \Big] + \Big[ \frac{1}{2} h^{\top}Ax+\frac{1}{2}x^{\top}Ah - b^{\top}h \Big] + \frac{1}{2} h^{\top}Ah \\
 &= \underbrace{\Big[ \frac{1}{2} x^{\top}Ax - b^{\top}x \Big]}_{\color{blue}{f(x)}} + h^{\top} \underbrace{\Big[ \frac{1}{2} Ax + \frac{1}{2} A^{\top}x - b \Big]}_{\color{red}{f'(x)}} + \underbrace{\frac{1}{2} h^{\top}Ah}_{o(\Vert h \Vert)}
\end{align*}
$$
As a consequence :
$$ f'(x) = \frac{1}{2} \big(A+A^{\top} \big)x - b $$
A: The easiest way is to check it once really and then remember it :).
$$\begin{array}{rcl}\frac{\partial}{\partial x_i} f(x) &=& \frac{\partial}{\partial x_i}\frac{1}{2}\sum_{k = 1}^n\sum_{l = 1}^n A_{k,l}x_kx_l -\sum_{j = 1}^n b_j \\ &=& \frac{1}{2}\left(\sum_{k = 1, k \neq i}^n (A_{k,i}x_k+A_{i,k}x_k)+2A_{i,i}x_i\right)-b_i\\ &=& \frac{1}{2}\sum_{k = 1}^n(A_{k,i}+A_{i,k})x_k-b_i  \\ &=& (\frac{1}{2}(A+A^T)x-b)_i.\end{array}$$
It follows that $$\nabla f(x)= \frac{1}{2}(A+A^T)x-b.$$
