# Jacobian of a Composition involving a Linear Transformation

Let $f:{\mathbb{R}^n} \to \mathbb{R}$. For each $z \in {\mathbb{R}^n}$ define $\tilde f\left( z \right) = f\left( x \right)$, where $x = Az + s$, for some $A \in {\mathbb{R}^{n \times n}}$, $s \in {\mathbb{R}^n}$.

I want to find $\nabla \tilde f\left( z \right)$ and ${\nabla ^2}\tilde f\left( z \right)$ in terms of $\nabla f\left( x \right)$ and ${\nabla ^2}f\left( x \right)$, respectively.

• In class we defined the gradient of a function $f:{\mathbb{R}^n} \to \mathbb{R}$ w.r.t. $x=(x_{1},...,x_{n})$ as follows:

$$\nabla f\left( x \right) = {\left[ {\frac{{\partial f}}{{\partial {x_1}}}\left( x \right),...,\frac{{\partial f}}{{\partial {x_n}}}\left( x \right)} \right]^T};$$

• ${\nabla ^2}f\left( x \right)$ is the Hessian of $f$;

• I'll denote the Jacobian of $f$ by ${J_f}$.

Here's what I've done:

$x = \varphi \left( z \right) = Az + s$, where $A = {\left( {{a_{ij}}} \right)_{\scriptstyle1 \le i \le n\atop\scriptstyle1 \le j \le n}}$ and $s = {\left( {{s_i}} \right)_{1 \le i \le n}}$.

$$\nabla \tilde f\left( z \right) = {J_{\tilde f}}{\left( z \right)^T} = {J_{f \circ \varphi }}{\left( z \right)^T} = {\left( {{J_f}\left( {\varphi \left( z \right)} \right){J_\varphi }\left( z \right)} \right)^T} = {J_\varphi }{\left( z \right)^T}{J_f}{\left( x \right)^T} = {J_\varphi }{\left( z \right)^T}\nabla f\left( x \right)$$

Now, as the ith component of $\varphi$ is

$${\varphi _i \left( z \right)} = \sum\limits_{j = 1}^n {{a_{ij}}z} + {s_i}$$ then,

$\frac{{\partial {\varphi _i}}}{{\partial {z_i}}}\left( z \right) = {a_{ij}}$, for all $j = 1,...,n$ and $z \in {\mathbb{R}^n}$. Consequently, ${J_\varphi }{\left( z \right)}=A$, so I can conclude that: $$\nabla \tilde f\left( z \right) = A^T \nabla f\left( x \right).$$

The problem arises when I try to find ${\nabla ^2}\tilde f\left( z \right)$.

I know that: $$\nabla \tilde f = {A^T} \circ \nabla f \circ \varphi$$ so $${\nabla ^2}\tilde f\left( z \right) = {J_{{A^T} \circ \nabla f \circ \varphi }}\left( z \right).$$ However, I don't know how to take $A^T$ out of the Jacobian. I suspect that: $${J_{{A^T} \circ \nabla f \circ \varphi }}\left( z \right) = {A^T} {J_{\nabla f \circ \varphi }}\left( z \right) = {A^T} {J_{\nabla f}}\left( {\varphi \left( z \right)} \right) {J_\varphi }\left( z \right) = {A^T} {J_{\nabla f}}\left( {\varphi \left( z \right)} \right) A = {A^T} {\nabla ^2}f\left( x \right) A$$ but I don't know to justify the first step in the last string of equalities.

Is my suspicion correct? Why?

I'd greatly appreciate any help. Thanks in advance.

$$\nabla \tilde{f} = A^T\circ \nabla f \circ \varphi$$

is, n my opinion, not the best method to view it, since $A^T$ is a different kind of beast than $\nabla f$ and $\varphi$. You multiply the result of $\nabla f\circ\varphi$ with the constant matrix $A^T$, so it's multiplying $\nabla f\circ\varphi$ with a constant, and multiplication with a constant commutes with differentiation.

Another way to look at it is to write it as

$$\nabla\tilde{f} = \mu_{A^T}\circ \nabla f\circ\varphi,$$

where $\mu_M$ is the multiplication with the matrix $M$, to emphasize the difference between the matrix and the map it induces. Then you get

$$J_{\nabla\tilde{f}}(z) = J_{\mu_{A^T}\circ (\nabla f\circ\varphi)}(z) = J_{\mu_{A^T}}(\nabla f(\varphi(z))\circ J_{\nabla f\circ\varphi}(z).$$

But $\mu_{A^T}$ is a linear map, so its derivative is constant, and equal to $\mu_{A^T}$ itself, so $J_{\mu_{A^T}} \equiv \mu_{A^T}$, and indeed

$$J_{\nabla\tilde{f}}(z) = \mu_{A^T}\circ J_{\nabla f\circ\varphi}(z).$$