2
$\begingroup$

I'm trying to find the following derivative:

\begin{equation} \frac{\partial^2 f}{\partial f \partial x} \end{equation} where $f$ is a neural network, and $x$ is an input. To be more accurate, let's say $f$ is a 2-layer neural net with a 1-D output and 1-D input. Then, we can write $f$ as: $$ f(x) = tanh(xW_1 + b_1)W_2 + b $$ I tried deriving the formula for $\frac{\partial f}{\partial x}$ and got the following: $$ \frac{\partial f}{\partial x} = W_2^T\times (W_1^T\circ tanh'(xW_1 + b_1)) $$ where $\circ$ is the element-wise product. Now, I need to take its derivative w.r.t. $f$ itself. Any ideas on how to proceed from here?


Edit: As @NinadMunshi pointed out, I intend to find the following derivative after a variable change: $$ z = tanh(xW_1 + b_1)W_2 + b\\ g = \frac{\partial f}{\partial x} = W_2^T\times (W_1^T\circ tanh'(xW_1 + b_1))\\ \frac{\partial g}{\partial z} = ? $$

$\endgroup$
5
  • $\begingroup$ what is the definition of derivative with respect to itself ? $\endgroup$
    – dezdichado
    Commented Jul 21, 2023 at 0:30
  • $\begingroup$ @dezdichado I mean taking the derivative of $\frac{\partial f}{\partial x}$ w.r.t. $f$, i.e. $\frac{\partial^2f}{\partial f \partial x}$. In other words, I need the derivatives of $\frac{\partial f}{\partial x}$ w.r.t. the outputs of the last layer of the network. I'm not sure if this is well-defined though... $\endgroup$
    – nimnim
    Commented Jul 21, 2023 at 1:46
  • 1
    $\begingroup$ The question makes no sense. $\endgroup$
    – copper.hat
    Commented Jul 21, 2023 at 2:54
  • 1
    $\begingroup$ @copper.hat I believe the intention is a variable change. For example take $$f(x) = e^{x^2} = z$$ Then we get $$\frac{\partial f}{\partial x} = 2xe^{x^2} = 2z\sqrt{\log z} \equiv g(z)$$ and $$\frac{\partial g}{\partial z} = 2\sqrt{\log z} + \frac{1}{\sqrt{\log z}} \equiv 2x + \frac{1}{x}$$ to which we can somewhat soundly attach the abuse of notation $$\frac{\partial^2 f}{\partial f \partial x} = 2x + \frac{1}{x}$$ $\endgroup$ Commented Jul 21, 2023 at 3:02
  • $\begingroup$ @copper.hat At least for functions $f:\Bbb{R}\to\Bbb{R}$, this operator reduces to the logarithmic derivative of the derivative i.e. $\frac{f''(x)}{f'(x)}$ $\endgroup$ Commented Jul 21, 2023 at 23:09

1 Answer 1

1
$\begingroup$

$\textbf{Hint}$: Consider the case where $W_i,b_i\in\Bbb{R}$. Then the solution is simply

$$\frac{\partial^2 f}{\partial f \partial x} = -2W_1W_2\tanh(W_1x+b_1)$$

Now how would you adapt this answer from scalars to vector functions that act component wise?

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .