How to prove this

$\dfrac{\partial F}{\partial n^m}=\left(\dfrac{\partial n^{m+1}}{\partial n^m}\right)^T\dfrac{\partial F}{\partial n^{m+1}}$ where $n$ is a vector ($\mathbf n$), $\left(\dfrac{\partial n^{m+1}}{\partial n^m}\right)$ is of size $S^{m+1}\times S^m$, and $\dfrac{\partial F}{\partial n^{m+1}}$ is of size $S^m\times1$?

Attempt: To simplify, assume $S^{m+1}=S^m=2.$ Thus

$\dfrac{\partial n^{m+1}}{\partial n^{m}}=\begin{pmatrix} \dfrac{\partial n_1^{m+1}}{\partial n_1^{m}}&\dfrac{\partial n_1^{m+1}}{\partial n_2^{m}}\\ \dfrac{\partial n_2^{m+1}}{\partial n_1^{m}} & \dfrac{\partial n_2^{m+1}}{\partial n_2^{m}} \end{pmatrix},$ its transpose is $\left(\dfrac{\partial n^{m+1}}{\partial n^{m}}\right)^T=\begin{pmatrix} \dfrac{\partial n_1^{m+1}}{\partial n_1^{m}}&\dfrac{\partial n_2^{m+1}}{\partial n_1^{m}}\\ \dfrac{\partial n_1^{m+1}}{\partial n_2^{m}} & \dfrac{\partial n_2^{m+1}}{\partial n_2^{m}} \end{pmatrix},$

and $\dfrac{\partial F}{\partial n^{m+1}}=\begin{pmatrix} \dfrac{\partial F}{\partial n_1^{m+1}}\\ \dfrac{\partial F}{\partial n_2^{m+1}} \end{pmatrix}$

Thus $\left(\dfrac{\partial n^{m+1}}{\partial n^m}\right)^T\dfrac{\partial F}{\partial n^{m+1}}=\begin{pmatrix} \dfrac{\partial F}{\partial n_1^{m}}+ \dfrac{\partial F}{\partial n_1^{m}}\\ \dfrac{\partial F}{\partial n_2^{m}}+\dfrac{\partial F}{\partial n_2^{m}} \end{pmatrix}.$

Which is not correct because $\dfrac{\partial F}{\partial n^m}=\begin{pmatrix} \dfrac{\partial F}{\partial n_1^{m}}\\ \dfrac{\partial F}{\partial n_2^{m}} \end{pmatrix}.$

Why is that?

Thank you in advance.

This is from the book Neural Network Design by Hagan, Demuth, Beale, De Jesús, and it's a step of an equality from page [368] in the pdf book.

  • $\begingroup$ How can $S^{m+1}$ and $S^m$ both be the same number? What are the $m$'s actually representing? $\endgroup$ Apr 5 '21 at 4:11
  • $\begingroup$ @NinadMunshi $m$ stands for a layer in a neural network (first, second, etc). $S^{something}$ is the number of neurons in each layer (1,2, etc). So we can assume an equality. $\endgroup$ Apr 5 '21 at 4:13
  • $\begingroup$ But I think this can be totally separated from the ANN, and it's just math. $\endgroup$ Apr 5 '21 at 4:19
  • $\begingroup$ Oh gotcha. Your simplification is wrong, you have to use the correct multivariate chain rule, which you are not. I will type up an answer below. $\endgroup$ Apr 5 '21 at 4:24

In your simplification you are assuming that

$$\frac{\partial f}{\partial y} \frac{\partial y}{\partial x} = \frac{\partial f}{\partial x}$$

as would be the case with single variable chain rule. But the problem is that partial derivatives alone cannot account for the "whole" variation. If I have a change of variables from $(x,y)$ to $(s,t)$ the correct partials by chain rule would be as follows

$$\frac{\partial f}{\partial s} = \frac{\partial f}{\partial x}\frac{\partial x}{\partial s} + \frac{\partial f}{\partial y}\frac{\partial y}{\partial s}$$

This is because when we want to account for a variation in $s$ we have to hunt down and add all of its contributions since a change of variable could also be written as $f(s,t)=f(x(s,t),y(s,t))$. In your case this means the correct chain rule would give you

$$\frac{\partial F}{\partial n_1^{m+1}}\frac{\partial n_1^{m+1}}{\partial n_1^m} + \frac{\partial F}{\partial n_2^{m+1}}\frac{\partial n_2^{m+1}}{\partial n_1^m} = \frac{\partial F}{\partial n_1^m}$$

and so on for the second term.

  • $\begingroup$ Thank you for the help. $\endgroup$ Apr 5 '21 at 18:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.