# Derivative of the derivative of a neural network w.r.t. itself

I'm trying to find the following derivative:

$$$$\frac{\partial^2 f}{\partial f \partial x}$$$$ where $$f$$ is a neural network, and $$x$$ is an input. To be more accurate, let's say $$f$$ is a 2-layer neural net with a 1-D output and 1-D input. Then, we can write $$f$$ as: $$f(x) = tanh(xW_1 + b_1)W_2 + b$$ I tried deriving the formula for $$\frac{\partial f}{\partial x}$$ and got the following: $$\frac{\partial f}{\partial x} = W_2^T\times (W_1^T\circ tanh'(xW_1 + b_1))$$ where $$\circ$$ is the element-wise product. Now, I need to take its derivative w.r.t. $$f$$ itself. Any ideas on how to proceed from here?

Edit: As @NinadMunshi pointed out, I intend to find the following derivative after a variable change: $$z = tanh(xW_1 + b_1)W_2 + b\\ g = \frac{\partial f}{\partial x} = W_2^T\times (W_1^T\circ tanh'(xW_1 + b_1))\\ \frac{\partial g}{\partial z} = ?$$

• what is the definition of derivative with respect to itself ? Commented Jul 21, 2023 at 0:30
• @dezdichado I mean taking the derivative of $\frac{\partial f}{\partial x}$ w.r.t. $f$, i.e. $\frac{\partial^2f}{\partial f \partial x}$. In other words, I need the derivatives of $\frac{\partial f}{\partial x}$ w.r.t. the outputs of the last layer of the network. I'm not sure if this is well-defined though... Commented Jul 21, 2023 at 1:46
• The question makes no sense. Commented Jul 21, 2023 at 2:54
• @copper.hat I believe the intention is a variable change. For example take $$f(x) = e^{x^2} = z$$ Then we get $$\frac{\partial f}{\partial x} = 2xe^{x^2} = 2z\sqrt{\log z} \equiv g(z)$$ and $$\frac{\partial g}{\partial z} = 2\sqrt{\log z} + \frac{1}{\sqrt{\log z}} \equiv 2x + \frac{1}{x}$$ to which we can somewhat soundly attach the abuse of notation $$\frac{\partial^2 f}{\partial f \partial x} = 2x + \frac{1}{x}$$ Commented Jul 21, 2023 at 3:02
• @copper.hat At least for functions $f:\Bbb{R}\to\Bbb{R}$, this operator reduces to the logarithmic derivative of the derivative i.e. $\frac{f''(x)}{f'(x)}$ Commented Jul 21, 2023 at 23:09

$$\textbf{Hint}$$: Consider the case where $$W_i,b_i\in\Bbb{R}$$. Then the solution is simply
$$\frac{\partial^2 f}{\partial f \partial x} = -2W_1W_2\tanh(W_1x+b_1)$$