Let $f:{\mathbb{R}^n} \to \mathbb{R}$. For each $z \in {\mathbb{R}^n}$ define $\tilde f\left( z \right) = f\left( x \right)$, where $x = Az + s$, for some $A \in {\mathbb{R}^{n \times n}}$, $s \in {\mathbb{R}^n}$.

I want to find $\nabla \tilde f\left( z \right)$ and ${\nabla ^2}\tilde f\left( z \right)$ in terms of $\nabla f\left( x \right)$ and ${\nabla ^2}f\left( x \right)$, respectively.

Please note the following:

  • In class we defined the gradient of a function $f:{\mathbb{R}^n} \to \mathbb{R}$ w.r.t. $x=(x_{1},...,x_{n})$ as follows:

$$\nabla f\left( x \right) = {\left[ {\frac{{\partial f}}{{\partial {x_1}}}\left( x \right),...,\frac{{\partial f}}{{\partial {x_n}}}\left( x \right)} \right]^T};$$

  • ${\nabla ^2}f\left( x \right)$ is the Hessian of $f$;

  • I'll denote the Jacobian of $f$ by ${J_f}$.

Here's what I've done:

$x = \varphi \left( z \right) = Az + s$, where $A = {\left( {{a_{ij}}} \right)_{\scriptstyle1 \le i \le n\atop\scriptstyle1 \le j \le n}} $ and $s = {\left( {{s_i}} \right)_{1 \le i \le n}} $.

$$\nabla \tilde f\left( z \right) = {J_{\tilde f}}{\left( z \right)^T} = {J_{f \circ \varphi }}{\left( z \right)^T} = {\left( {{J_f}\left( {\varphi \left( z \right)} \right){J_\varphi }\left( z \right)} \right)^T} = {J_\varphi }{\left( z \right)^T}{J_f}{\left( x \right)^T} = {J_\varphi }{\left( z \right)^T}\nabla f\left( x \right)$$

Now, as the ith component of $\varphi$ is

$${\varphi _i \left( z \right)} = \sum\limits_{j = 1}^n {{a_{ij}}z} + {s_i}$$ then,

$\frac{{\partial {\varphi _i}}}{{\partial {z_i}}}\left( z \right) = {a_{ij}}$, for all $j = 1,...,n$ and $z \in {\mathbb{R}^n}$. Consequently, ${J_\varphi }{\left( z \right)}=A$, so I can conclude that: $$\nabla \tilde f\left( z \right) = A^T \nabla f\left( x \right).$$

The problem arises when I try to find ${\nabla ^2}\tilde f\left( z \right)$.

I know that: $$\nabla \tilde f = {A^T} \circ \nabla f \circ \varphi$$ so $${\nabla ^2}\tilde f\left( z \right) = {J_{{A^T} \circ \nabla f \circ \varphi }}\left( z \right).$$ However, I don't know how to take $A^T$ out of the Jacobian. I suspect that: $${J_{{A^T} \circ \nabla f \circ \varphi }}\left( z \right) = {A^T} {J_{\nabla f \circ \varphi }}\left( z \right) = {A^T} {J_{\nabla f}}\left( {\varphi \left( z \right)} \right) {J_\varphi }\left( z \right) = {A^T} {J_{\nabla f}}\left( {\varphi \left( z \right)} \right) A = {A^T} {\nabla ^2}f\left( x \right) A$$ but I don't know to justify the first step in the last string of equalities.

Is my suspicion correct? Why?

I'd greatly appreciate any help. Thanks in advance.


Your suspicion is correct. Writing

$$\nabla \tilde{f} = A^T\circ \nabla f \circ \varphi$$

is, n my opinion, not the best method to view it, since $A^T$ is a different kind of beast than $\nabla f$ and $\varphi$. You multiply the result of $\nabla f\circ\varphi$ with the constant matrix $A^T$, so it's multiplying $\nabla f\circ\varphi$ with a constant, and multiplication with a constant commutes with differentiation.

Another way to look at it is to write it as

$$\nabla\tilde{f} = \mu_{A^T}\circ \nabla f\circ\varphi,$$

where $\mu_M$ is the multiplication with the matrix $M$, to emphasize the difference between the matrix and the map it induces. Then you get

$$J_{\nabla\tilde{f}}(z) = J_{\mu_{A^T}\circ (\nabla f\circ\varphi)}(z) = J_{\mu_{A^T}}(\nabla f(\varphi(z))\circ J_{\nabla f\circ\varphi}(z).$$

But $\mu_{A^T}$ is a linear map, so its derivative is constant, and equal to $\mu_{A^T}$ itself, so $J_{\mu_{A^T}} \equiv \mu_{A^T}$, and indeed

$$J_{\nabla\tilde{f}}(z) = \mu_{A^T}\circ J_{\nabla f\circ\varphi}(z).$$


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.