Proof of the derivative of a quadratic form I was reading this pdf and on page 6 proposition 8 states:

I don't really understand the steps that bring from
$$\alpha = \sum_{j=1}^n\sum_{i=1}^n a_{ij} x_{i} x_{j}$$
to its derivative 
$$\frac{\partial \alpha}{\partial \bf{x}} = \sum_{j=1}^n a_{kj} x_J + \sum_{i=1}^n a_{ik}x_i$$
and then back to the final result:
$$\frac{\partial \alpha}{\partial \bf{x}} = \bf{x}^T A^T + \bf{x}^T A$$
Can someone please help me?
 A: Another way to approach this formula is to use the definition of derivatives in multivariable calculus. The function is $\alpha: \mathbb R^n \to \mathbb R$ and the Jocabian matrix $D\alpha = \frac{\partial \alpha}{\partial x}$ is thus an $n \times n$ matrix and by definition satisfies the following equation
\begin{align*}
\lim_{\|h\| \to 0} \frac{\alpha(x+h) - \alpha(x) - D\alpha(h)} {h} = 0.
\end{align*}
But note $\alpha(x+h) - \alpha(h) = (x+h)^T A (x+h)- x^T A x = h^T A x + x^T A h$. Since $h^T A x = x^T A^T h$, we have $\alpha(x+h) - \alpha(h) = x^T(A^T + A)h$. Then it follows $\frac{\partial \alpha}{\partial x} = x^T(A^T+A)$.
A: Consider a generic $1 \leq k \leq n$. We can write the following:
$$\alpha = \sum_{j=1}^n\sum_{i=1}^n a_{ij} x_{i} x_{j} = \sum_{j=1}^n\left(\sum_{i=1, i \neq k}^n a_{ij} x_{i} x_{j} + a_{kj}x_{k}x_{j}\right) = \\
= \sum_{i=1, i \neq k}^n \sum_{j=1}^na_{ij} x_{i} x_{j} + \sum_{j=1}^na_{kj}x_{k}x_{j} =\\
= \sum_{i=1, i \neq k}^n \left(\sum_{j=1, j\neq k}^na_{ij} x_{i} x_{j} + a_{ik}x_i x_k\right) + \sum_{j=1, j \neq k}^na_{kj}x_{k}x_{j} + a_{kk}x_{k}^2 =\\
= \sum_{i=1, i \neq k}^n \sum_{j=1, j\neq k}^na_{ij} x_{i} x_{j} + \sum_{i=1, i\neq k}^na_{ik}x_i x_k + \sum_{j=1, j \neq k}^na_{kj}x_{k}x_{j} + a_{kk}x_{k}^2.\\
$$
Specifically, we have separated all the contributions depending on $x_k$ and those not depending on $x_k$. It is clear now that:
$$\frac{\partial \alpha}{\partial x_k} = \sum_{i=1, i\neq k}^na_{ik}x_i + \sum_{j=1, j \neq k}^na_{kj}x_{j} + 2a_{kk}x_{k}.$$
We can further work on the last expression:
$$\frac{\partial \alpha}{\partial x_k} = \left[\sum_{i=1}^na_{ik}x_i - a_{kk}x_k\right] + \left[\sum_{j=1}^na_{kj}x_{j} - a_{kk}x_k\right] + 2a_{kk}x_{k} = \sum_{i=1}^na_{ik}x_i + \sum_{j=1}^na_{kj}x_{j}.$$
Now, we can try to obtain a vectorial representation. Let's pose:


*

*$f_k = \displaystyle\sum_{i=1}^na_{ik}x_i,$

*$g_k = \displaystyle\sum_{j=1}^na_{kj}x_{j},$

*${\bf f} = [f_1, f_2, \ldots, f_n],$

*${\bf g} = [g_1, g_2, \ldots, g_n],$
where ${\bf f}$ and ${\bf g}$ are row vectors.
It is clear that:


*

*${\bf f} = {\bf x}^\top {\bf A},$

*${\bf g} = {\bf x}^\top {\bf A}^\top,$
and hence:
$$\frac{\partial \alpha}{\partial {\bf x}} = {\bf x}^\top {\bf A} + {\bf x}^\top {\bf A}^\top.$$
A: Alternative approach(once you get used with these notations, then it will get easier)
Before we start deriving the gradient, some facts:


*

*Trace and Frobenius product relation $$\left\langle A, B C\right\rangle={\rm tr}(A^TBC) := A : B C$$ 

*Cyclic properties of Trace/Frobenius product 
\begin{align}
A : B C 
 &= BC : A \\
 &= A^T : (BC)^T \\
 &= B^T A   :  C  \\
 &= {\text{etc.}} \cr
\end{align}
Let $f := x^T A x = {\rm tr}\left(x^T A x \right) = x:Ax$. (Trace of a scalar will return the same scalar).
Now, we can obtain the differential first, and then the gradient.
\begin{align}
df = d \ {\rm tr }\left ( x^T A x \right) 
&= d\left(x : A x \right) \\
&= \left(dx : Ax\right)  + \left(x : A \ dx\right) \\
&= \left(Ax : dx\right)  + \left(A^Tx : dx\right) \\
&= \left( Ax + A^T x \right) : dx\\
&= \left( Ax + A^T x \right)^T : dx^T\\
&= \left( x^T A^T + x^T A \right) : dx^T\\
\end{align}
Thus, the gradient is
\begin{align}
\frac{\partial}{\partial x^T} \left( x^T Ax \right)= x^T A^T + x^T A.
\end{align}
