# What is the gradient of $\frac{1}{2}(Ax)\cdot x$ if $A$ is a nonzero symmetric $3\times 3$ matrix

What is the gradient of $\frac{1}{2}(Ax)\cdot x$ if $A$ is a nonzero symmetric $3\times 3$ matrix?

Apparently the answer is simply $Ax$. Does anyone know why?

-
Assume $x$ is a column vector, otherwise it doesn't make sense to multiply matrix to a row vector due to the dimensions mismatch. Then $Ax$ is also a column vector, and again multiplying column vector to a column vector is undefined. You have two options: 1) Make it $\frac 12 (Ax)^T \cdot x$ 2) or $\frac 12 (Ax) \cdot x^T$ In first case you'll get a number, so gradient of it will be a row vector. In second case, the product is a matrix, and as far as I remember its gradient is even bigger of a matrix. In any case, you don't get $Ax$. – Kaster Apr 18 '13 at 4:54

Claim:

Let $A$ be a $k x k$ square matrix, $x$ be a $k x 1$ column vector, and let $\alpha = (1/2) x ^TAx$ be the quadratic form associated with $A$ and $x$.

Find the gradient of $\displaystyle \alpha = \frac{\partial \alpha}{\partial x} \left(\frac{1}{2} \right) x ^TAx = \frac{1}{2}\left(A + A^T\right)x$.

Furthermore, if $A$ is symmetric, this derivative reduces to $\displaystyle \left(\frac{1}{2}\right) 2 A \cdot x = A \cdot x.$

Why is that last statement true (we were told something about nonzero symmetric matrices)?

-
Nice! +1 for good work and for effort! – amWhy Apr 18 '13 at 5:12

Yes, and it is not specific to the 3 by 3 case, it is any size square symmetric. So, could you please, please, please, actually write out the 1 by 1 case (where $A$ is just a number) and the 2 by 2 case, and write out the value of your expression and find the gradient in these two easy cases.

-
+1 Good to see you 'round! And I love your comments & answers (and wit!) – amWhy Apr 18 '13 at 5:13

Here is another way to see it:

Let $\phi(x) = \frac{1}{2} \langle Ax, x \rangle$.

Then \begin{eqnarray} \phi(x+\delta) &=& \frac{1}{2} \langle A(x+\delta), x+\delta \rangle \\ &=& \frac{1}{2} \left( \langle Ax,x \rangle + \langle Ax,\delta \rangle + \langle A \delta,x \rangle + \langle A\delta, \delta \rangle \right) \\ &=& \phi(x) + \frac{1}{2} \left( \langle Ax,\delta \rangle + \langle \delta,A^T x \rangle + \langle A\delta, \delta \rangle \right) \\ &=& \phi(x) + \frac{1}{2} \left( \langle Ax,\delta \rangle + \langle A^T x, \delta \rangle + \langle A\delta, \delta \rangle \right) \\ &=& \phi(x) + \langle \frac{1}{2} (A+A^T)x,\delta \rangle + \langle A\delta, \delta \rangle \end{eqnarray} It follows from this (since $|\langle A\delta, \delta \rangle| \le \|A\| \|\delta \|^2$) that $\nabla \phi(x) = \frac{1}{2} (A+A^T)x$.

-