Differentiating vector function-matrix-vector function products Consider the following scalar which is the result of a vector-matrix-vector product:
$$ h( \bf{x} ) = \bf{f}( \bf{x} ) A \bf{g}(\bf{x})^T$$
where

*

*$\bf{x} = (x_1, x_2, \ldots, x_k)$ is an input vector

*$\bf{f}(\bf{x}) = (f_1(\bf{x}), f_2(\bf{x}), \ldots, f_p(\bf{x}))$ is a $p$ vector of simple, known functions

*$\bf{g}(\bf{x}) = (g_1(\bf{x}), g_2(\bf{x}), \ldots, g_q(\bf{x}))$ is a $q$ vector of simple, known functions

*$A$ is a $p \times q$ matrix of (known) constants. Denote the $i$th row of $A$ by $\bf{a}_i$ and the $j$th column by $\bf{a}_{(j)}$.

What I require is $\frac{\partial h(\bf{x})}{\partial \bf{x}}$. I am relatively un-familiar with matrix and vector type calculus. I studied some vector-calculus some years ago but the introduction of matrices makes this fiddly for me. but this is what I tried to use the fact that the final result is just a double sum to help me:
$$\frac{\partial}{\partial \bf{x} } h(\bf{x}) = \sum_{i=1}^p \sum_{j=1}^q \frac{\partial}{\partial \bf{x}}f_i a_{ij} g_j$$ (dropping dependence on $\bf{x}$ in $f_i$ and $g_j$)
$$= \sum_{i=1}^p \sum_{j=1}^q \frac{\partial f_i}{\partial \bf{x}} a_{ij} g_j + \sum_{i=1}^p \sum_{j=1}^qf_i a_{ij} \frac{\partial g_i}{\partial \bf{x}}$$ (product rule for partial derivatives + break up the sum)
Now in principle this is enough, however, I'd like to write the final result as a matrix-vector product for (a) consiseness and (b) I'd like to use the matrix-vector product rather than the double sum for computation. Ploughing ahead gives us
$$= \sum_{i=1}^p \frac{\partial f_i}{\partial \bf{x}}\sum_{j=1}^q  a_{ij} g_j + \sum_{j=1}^q\frac{\partial g_j}{\partial \bf{x}} \sum_{i=p}^q a_{ij}f_i $$ (re-order the sum)
$$= \sum_{i=1}^p \frac{\partial f_i}{\partial \bf{x}} \bf{a}_{i} \bf{g}^T + \sum_{j=1}^q\frac{\partial g_j}{\partial \bf{x}} \bf{a}_{(j)}^T \bf{f}^T $$
(collecting the "2nd sums" into vector products).
Now here is where I am stumped. The dimension of the result so far is correct (dim = $1 \times k$) but I'm struggling to write this as a vector-matrix product. The result looks a bit like
$\frac{\partial \bf{f}}{\partial \bf{x}} A \bf{g}^T + \frac{\partial \bf{g}}{\partial \bf{x}} A^T \bf{f}^T$ but clearly the dimensions are incorrect. Dimension of $\frac{\partial{h}}{\partial \bf{x}}$ are $1 \times k$ whereas I think the dimension of my guess is $(p \times k)(p \times q)(q \times 1) + (q \times k)(q \times p)(q \times 1)$. These dimensions are not compatible in $2$ ways!
 A: The differential approach is certainly best adapted to your needs.
Let $\phi
= \mathbf{f}^T(\mathbf{x}) \mathbf{A} \mathbf{g}(\mathbf{x})$.
Note : It is much better to use column vectors so I change your notations.
From here
$$
d\phi=
d\mathbf{f}:\mathbf{A} \mathbf{g}+
\mathbf{f}:\mathbf{A} (d\mathbf{g})
$$
the colon operator is the Frobenius inner product.
Introducing the Jacobians
$d\mathbf{f}= \mathbf{J}_f d\mathbf{x},
d\mathbf{g}= \mathbf{J}_g d\mathbf{x}$,
we easily obtain the gradient in matrix form
$$
\frac{d\phi}{d\mathbf{x}}
= \mathbf{J}_f^T \mathbf{A} \mathbf{g} + 
\mathbf{J}_g^T \mathbf{A}^T \mathbf{f}
$$
A: There are two conventions to differentiate a scalar by a vector (as in your setup), one being where the output is a row vector, and the other being where the output is a column vector. Let's do both:

*

*Output is $1\times k$ row vector :

$$\bf \partial_x (f(x)Ag(x)^T)=f(x)A\underbrace{\partial_xg(x)^T}_{q\times k}+g(x)A^T\underbrace{\partial_xf(x)^T}_{p\times k}$$


*Output is $k\times 1$ column vector:

$$\bf \partial_x (f(x)Ag(x)^T)=(\underbrace{\partial_xg(x)^T}_{k\times q})A^Tf(x)^T+(\underbrace{\partial_xf(x)^T}_{k\times p})Ag(x)^T.$$
Notice how I use different dimension conventions in the two approaches for taking derivatives of $\bf g$ and $\bf f$ with respect to $\bf x$ (which are derivatives of vectors with respect to vectors...ugh). Don't let that confuse you; I just did that to be consistent within the convention I was adopting. But the dimensions should tell you exactly how those derivatives would be computed.
If you want a full reference for matrix derivative identities, you can check out wiki.
Personally, I do derivatives quick and dirty: I just pretend I am differentiating with all scalars, and then reshuffle things in my answer or throw in transposes as needed for the dimensions to work out, and this usually gets me the right answer:)
A: You can also proceed in a pure analytical approach.
Start by considering the bilinear continuous form $B(x, y) = x^\intercal A y.$ The derivative of this bilinear form is given by the usual product rule:
$$
B'(x, y) \cdot (h,k) = h^\intercal A y + x^\intercal A k.
$$
Similarly, the derivative of a vector function is just entry-wise derivative
$$
(f, g)'(x) = (f'(x), g'(x)).
$$
Then, the chain rule shows
$$
\begin{align*}
(f(x)^\intercal A g(x))' \cdot h &= B'(f(x), g(x)) \cdot (f'(x), g'(x)) \cdot h\\
&= (f'(x) \cdot h)^\intercal A g(x) + f(x)^\intercal A (g'(x) \cdot h).
\end{align*}
$$
If you want a matrix, simply let $J_f$and $J_g$ denote the Jacobian matrices of $f$ and $g$ at $x,$ the previous formula reads
$$
J_{f^\intercal A g} h = h^\intercal J_f^\intercal A g + f^\intercal A J_g h.
$$
Note that each summand is a scalar, so you can put transposes at will. Typically, the derivative of a scalar function is a column vector, in this case we should get
$$
\partial_x \varphi = J_f^\intercal A g + J_g^\intercal A f.
$$
