Vector valued function derivative with matrix

Question

Given a function $f: \mathbb{R}^n \rightarrow \mathbb{R}^n$ and a matrix $A \in \mathbb{R}^{n \times n}$. Is there a general formula for calculating the following derivative:

$$ \frac{\partial}{\partial x} f(x)^T A f(x) \tag{1} = ? $$

I know that

$$ \frac{\partial}{\partial x} x^T A x = x^T(A + A^T) \overset{A = A^T}{=} 2 x^T A \tag{2} $$

and the solution to $(1)$ will probably look similar to $(2)$, but I am stuck here since I am not sure how to apply the chain rule in the matrix case.

Edit: Regarding notation, we have

$$ \frac{\partial }{\partial x}f(x) = \begin{bmatrix} \frac{\partial}{\partial x_1} f_1(x) & \frac{\partial}{\partial x_2} f_1(x) & \cdots & \frac{\partial}{\partial x_n} f_1(x) \\ \frac{\partial}{\partial x_1} f_2(x) & \frac{\partial}{\partial x_2} f_2(x) & \cdots & \frac{\partial}{\partial x_n} f_2(x) \\ \vdots & \vdots & \ddots & \vdots \\ \frac{\partial}{\partial x_1} f_n(x) & \frac{\partial}{\partial x_2} f_n(x) & \cdots & \frac{\partial}{\partial x_n} f_n(x) \end{bmatrix} $$

and

$$ x = \begin{bmatrix} x_1 \\ x_2 \\ \vdots \\ x_n \end{bmatrix} , f(x) = \begin{bmatrix} f_1(x) \\ f_2(x) \\ \vdots \\ f_n(x) \end{bmatrix} $$

Answer 1

I find differential notation helpful here in organizing things. The total derivative is a linear operator, so we introduce its argument and apply the product rule: $$d(f(x)^TAf(x))=d(f(x)^T)\cdot Af(x)+f(x)^TA\cdot d(f(x)$$ $$d(f(x)^TAf(x))=\left(\frac{df}{dx}\cdot dx\right)^T\cdot Af(x)+f(x)^TA\cdot \left(\frac{df}{dx}\cdot dx\right)$$ Now, the transpose of a $1\times 1$ matrix is itself, so we transpose that first term: $$d(f(x)^TAf(x)) = f(x)^TA^T\cdot \left(\frac{df}{dx}\cdot dx\right)+f(x)^TA\cdot \left(\frac{df}{dx}\cdot dx\right)$$ $$d(f(x)^TAf(x)) = f(x)^T(A+A^T)\cdot \left(\frac{df}{dx}\cdot dx\right)$$ Now that's in the form we want for the derivative. The total derivative of $f(x)^TAf(x)$ is $$f(x)^T(A+A^T)\frac{df}{dx}$$ where $\frac{df}{dx}$ is the matrix of partial derivatives of $f$, written in your question as $\frac{\partial}{\partial x}f(x)$.

Answer 2

Given a differentiable vector field $\mathrm v : \mathbb R^n \to \mathbb R^n$ and a matrix $\mathrm A \in \mathbb R^{n \times n}$, let function $f : \mathbb R^n \to \mathbb R$ be defined by

$$f (\mathrm x) := \langle \mathrm v (\mathrm x), \mathrm A \mathrm v (\mathrm x) \rangle$$

whose directional derivative in the direction of $\mathrm y \in \mathbb R^n$ at $\mathrm x \in \mathbb R^n$ is

$$D_{\mathrm y} f (\mathrm x) := \lim_{h \to 0} \frac{f (\mathrm x + h \mathrm y) - f (\mathrm x)}{h} = \cdots = \langle \mathrm y, \mathrm J_{\mathrm v}^\top (\mathrm x) \, \mathrm A \, \mathrm v (\mathrm x) \rangle + \langle \mathrm J_{\mathrm v}^\top (\mathrm x) \, \mathrm A^\top \mathrm v (\mathrm x) , \mathrm y \rangle$$

where matrix $\mathrm J_{\mathrm v} (\mathrm x)$ is the Jacobian of vector field $\rm v$ at $\mathrm x \in \mathbb R^n$. Thus, the gradient of $f$ is

$$\nabla_{\mathrm x} f (\mathrm x) = \mathrm J_{\mathrm v}^\top (\mathrm x) \left( \mathrm A + \mathrm A^\top \right) \mathrm v (\mathrm x)$$

Answer 3

Well we want to differentiate $f(x)^TAf(x)$ then it is useful to break into pieces.

First we see how to differentiate $g(x,y) = x^TAy$ with $A$ constant. $$ g(x+h,y+k) = (x+h)^TA(y+k) =(x^T+h^T)A(y+k) = x^TAy + h^TAy + x^TAk + h^TAk $$ From this we see that $Dg_{(x,y)}(h,k) = h^TAy + x^TAk$.

Now we use the chain rule $$ D(f(x)^TAf(x))_x(v) = Dg_{(f(x),f(x))}(Df_x(v),Df_x(v)) = Df_x(v)^TAf(x) + f(x)^TADf_x(v) $$