Verifying matrix product rule example Suppose $f(x) = \textbf{x}^TA\textbf{x}$. I want to verify the product rule using this. So I decided to try a set up like $\mathcal{F}(\textbf{x}) = f(\textbf{x})g(\textbf{x})$ where $f(\textbf{x}) = \textbf{x}^T$ and $g(\textbf{x}) = A\textbf{x}$. So I would want to show $\frac{\partial \mathcal{F}}{\partial x_i} = \frac{\partial f}{\partial x_i}g(\textbf{x}) + f(\textbf{x})\frac{\partial g}{\partial x_i} = 2\textbf{x}^Ta_i$ for $a_i \in \mathbb{R}^n$ i-th row of $A$. If I show this, then it is easy to get the full case for $\textbf{x}$
My attempt is below:
$$\frac{\partial f}{\partial x_i} = [0 0 ... 1 ... 0]$$
$$\frac{\partial g}{\partial x_i} = \begin{bmatrix} 
a_{1i} \\
a_{2i} \\
\dots \\
a_{ni}
\end{bmatrix}
= a_i \in \mathbb{R}^n
$$
And so we see that $$\frac{\partial f}{\partial x_i} g(x) = [0 0 ... 1 ... 0] \begin{bmatrix}
                                                                            a_{11}x_1+...+a_{1n}x_n \\
                                                                            \dots \\
                                                                            a_{i1}x_1+...+a_{ii}x_i+...+ a_{in}x_n\\
                                                                            \dots
                                                                            \end{bmatrix}   
= \sum_{j=1}^{n} a_{ij}x_j$$
And similarly, $$f(x) \frac{\partial g}{\partial x_i} = [x_1 x_2 ... x_n] \begin{bmatrix} 
                                                                        a_{1i} \\
                                                                        a_{2i} \\
                                                                        \dots \\
                                                                        a_{ni}
                                                                        \end{bmatrix}
 = \sum_{j=1}^{n}a_{ji}x_j$$
Hence, $\frac{\partial \mathcal{F}}{\partial x_i} = \frac{\partial f}{\partial x_i} g(x) + f(x) \frac{\partial g}{\partial x_i} = \sum_{j=1}^{n} a_{ij}x_j + \sum_{j=1}^{n}a_{ji}x_j$.
Now this is where I am stuck, as the answer seems to be close, but instead I have the i-th column sum with the i-th row sum, and so I get something slightly different. Where did I go wrong? Is my setup incorrect?
 A: I'm going to use the Einstein summation notation to suppress sums and make things look nicer. The idea is that a product with matching upper and lower indices will be summed over with one sum per matching index. For example we could write that $$C_k^m = A_{ijk}B^{ijm} = \sum_{i=1}^n\sum_{j=1}^n A_{ijk} B_{ijm}$$ for some indexable objects $A,B,C.$ Note that summation is only invoked over products and not over other summed terms.
In indicial notation, $\mathcal{F}$ would be written as $\mathcal{F}(\mathbf{x})=x^i a_{ij} x^j.$ Notice that $x$ and $a$ are both scalars so we can treat them as such and write $\mathcal{F}(\mathbf{x}) = a_{ij}x^ix^j.$ We can also use the scalar product rule and write
$$\frac{\partial \mathcal{F}}{\partial x^k} = a_{ij}\frac{\partial x^i}{\partial x^k} x^j + a_{ij}x^i\frac{\partial x^j}{\partial x^k}.$$ Now we must realize that $\frac{\partial x^\ell}{\partial x^k} = \delta_k^\ell$ where $\delta$ is the Kronecker Delta defined as
$$\delta_k^\ell = 
\begin{cases}
1,  & \ell=k \\[2ex]
0, & \ell\neq k
\end{cases} $$
This means that
$$\frac{\partial \mathcal{F}}{\partial x^k} = a_{ij}\delta^i_k x^j + a_{ij}x^i\delta^j_k$$ $$\text{and thus}$$
$$\frac{\partial \mathcal{F}}{\partial x^k} = a_{kj} x^j + a_{ik}x^i$$
by noticing that $\delta$ will just rename indices over summation. As an example it is clear that
$$ x^j \delta^i_j = \sum_{j=1}^n x^j \delta^i_j = x^i .$$
Because $i$ and $j$ are summation indices, we can rename them without consequence. This means we can write
$$\frac{\partial \mathcal{F}}{\partial x^k} = a_{kj} x^j + a_{jk}x^j = (a_{kj} + a_{jk})x^j.$$
Thus, your idea that $\frac{\partial \mathcal{F}}{\partial x^k} =2\textbf{x}^Ta_k$ is only true if the matrix $A$ is symmetric (i.e. $a_{kj} = a_{jk}$).
Now as for your question on the product rule we will redefine
$$\mathcal{F}(\textbf{x}) = f(\textbf{x})g(\textbf{x}).$$ Looking at $\frac{\partial f}{\partial x^k}$ we have that
$$\frac{\partial f}{\partial x^k} = \frac{\partial x^i}{\partial x^k} = \delta^i_k.$$
Now for $\frac{\partial g}{\partial x^k}$ we have that
$$\frac{\partial g}{\partial x^k} = a_{i\ell} \frac{\partial x^\ell}{\partial x^k} = a_{i\ell}\delta ^\ell_k = a_{ik}.$$
Piecing together our guess at a product rule we see that
$$\frac{\partial f}{\partial x^k}g(\textbf{x}) + f(\textbf{x})\frac{\partial g}{\partial x^k} = \delta^i_k a_{i\ell}x^\ell + x^ia_{ik} .$$
Doing the summation with the Kronecker Delta we find that
$$\frac{\partial f}{\partial x^k}g(\textbf{x}) + f(\textbf{x})\frac{\partial g}{\partial x^k} =  a_{k\ell}x^\ell + x^ia_{ik}$$
so we can rename the $\ell$ index to $i$ and see that
$$\frac{\partial f}{\partial x^k}g(\textbf{x}) + f(\textbf{x})\frac{\partial g}{\partial x^k} =  a_{ki}x^i + x^ia_{ik} = (a_{ki} + a_{ik})x^i$$
which is exactly $\frac{\partial \mathcal{F}}{\partial x^k}.$ This means that the product rule that you described will work as expected.
A: Note that $f(x)=x^T$ and $g(x) = Ax$ are both linear hence they are their own derivatives. Then the chain rule gives
$D(f \circ g)(x)h = Df(x)h f(x) + f(x)Dg(x)h = f(h)g(x)+f(x)g(h) = x^T(A+A^T) h$.
I have been a little sloppy by not distinguishing function application and matrix multiplication.
