I'm trying to find the following derivative:

\begin{equation} \frac{\partial^2 f}{\partial f \partial x} \end{equation} where $f$ is a neural network, and $x$ is an input. To be more accurate, let's say $f$ is a 2-layer neural net with a 1-D output and 1-D input. Then, we can write $f$ as: $$ f(x) = tanh(xW_1 + b_1)W_2 + b $$ I tried deriving the formula for $\frac{\partial f}{\partial x}$ and got the following: $$ \frac{\partial f}{\partial x} = W_2^T\times (W_1^T\circ tanh'(xW_1 + b_1)) $$ where $\circ$ is the element-wise product. Now, I need to take its derivative w.r.t. $f$ itself. Any ideas on how to proceed from here?

Edit: As @NinadMunshi pointed out, I intend to find the following derivative after a variable change: $$ z = tanh(xW_1 + b_1)W_2 + b\\ g = \frac{\partial f}{\partial x} = W_2^T\times (W_1^T\circ tanh'(xW_1 + b_1))\\ \frac{\partial g}{\partial z} = ? $$

  • $\begingroup$ what is the definition of derivative with respect to itself ? $\endgroup$
    – dezdichado
    Commented Jul 21, 2023 at 0:30
  • $\begingroup$ @dezdichado I mean taking the derivative of $\frac{\partial f}{\partial x}$ w.r.t. $f$, i.e. $\frac{\partial^2f}{\partial f \partial x}$. In other words, I need the derivatives of $\frac{\partial f}{\partial x}$ w.r.t. the outputs of the last layer of the network. I'm not sure if this is well-defined though... $\endgroup$
    – nimnim
    Commented Jul 21, 2023 at 1:46
  • 1
    $\begingroup$ The question makes no sense. $\endgroup$
    – copper.hat
    Commented Jul 21, 2023 at 2:54
  • 1
    $\begingroup$ @copper.hat I believe the intention is a variable change. For example take $$f(x) = e^{x^2} = z$$ Then we get $$\frac{\partial f}{\partial x} = 2xe^{x^2} = 2z\sqrt{\log z} \equiv g(z)$$ and $$\frac{\partial g}{\partial z} = 2\sqrt{\log z} + \frac{1}{\sqrt{\log z}} \equiv 2x + \frac{1}{x}$$ to which we can somewhat soundly attach the abuse of notation $$\frac{\partial^2 f}{\partial f \partial x} = 2x + \frac{1}{x}$$ $\endgroup$ Commented Jul 21, 2023 at 3:02
  • $\begingroup$ @copper.hat At least for functions $f:\Bbb{R}\to\Bbb{R}$, this operator reduces to the logarithmic derivative of the derivative i.e. $\frac{f''(x)}{f'(x)}$ $\endgroup$ Commented Jul 21, 2023 at 23:09

1 Answer 1


$\textbf{Hint}$: Consider the case where $W_i,b_i\in\Bbb{R}$. Then the solution is simply

$$\frac{\partial^2 f}{\partial f \partial x} = -2W_1W_2\tanh(W_1x+b_1)$$

Now how would you adapt this answer from scalars to vector functions that act component wise?


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .