# Prove of $\frac{\partial F}{\partial n^m}=\left(\frac{\partial n^{m+1}}{\partial n^m}\right)^T\frac{\partial F}{\partial n^{m+1}}$

How to prove this

$$\dfrac{\partial F}{\partial n^m}=\left(\dfrac{\partial n^{m+1}}{\partial n^m}\right)^T\dfrac{\partial F}{\partial n^{m+1}}$$ where $$n$$ is a vector ($$\mathbf n$$), $$\left(\dfrac{\partial n^{m+1}}{\partial n^m}\right)$$ is of size $$S^{m+1}\times S^m$$, and $$\dfrac{\partial F}{\partial n^{m+1}}$$ is of size $$S^m\times1$$?

Attempt: To simplify, assume $$S^{m+1}=S^m=2.$$ Thus

$$\dfrac{\partial n^{m+1}}{\partial n^{m}}=\begin{pmatrix} \dfrac{\partial n_1^{m+1}}{\partial n_1^{m}}&\dfrac{\partial n_1^{m+1}}{\partial n_2^{m}}\\ \dfrac{\partial n_2^{m+1}}{\partial n_1^{m}} & \dfrac{\partial n_2^{m+1}}{\partial n_2^{m}} \end{pmatrix},$$ its transpose is $$\left(\dfrac{\partial n^{m+1}}{\partial n^{m}}\right)^T=\begin{pmatrix} \dfrac{\partial n_1^{m+1}}{\partial n_1^{m}}&\dfrac{\partial n_2^{m+1}}{\partial n_1^{m}}\\ \dfrac{\partial n_1^{m+1}}{\partial n_2^{m}} & \dfrac{\partial n_2^{m+1}}{\partial n_2^{m}} \end{pmatrix},$$

and $$\dfrac{\partial F}{\partial n^{m+1}}=\begin{pmatrix} \dfrac{\partial F}{\partial n_1^{m+1}}\\ \dfrac{\partial F}{\partial n_2^{m+1}} \end{pmatrix}$$

Thus $$\left(\dfrac{\partial n^{m+1}}{\partial n^m}\right)^T\dfrac{\partial F}{\partial n^{m+1}}=\begin{pmatrix} \dfrac{\partial F}{\partial n_1^{m}}+ \dfrac{\partial F}{\partial n_1^{m}}\\ \dfrac{\partial F}{\partial n_2^{m}}+\dfrac{\partial F}{\partial n_2^{m}} \end{pmatrix}.$$

Which is not correct because $$\dfrac{\partial F}{\partial n^m}=\begin{pmatrix} \dfrac{\partial F}{\partial n_1^{m}}\\ \dfrac{\partial F}{\partial n_2^{m}} \end{pmatrix}.$$

Why is that?

Thank you in advance.

This is from the book Neural Network Design by Hagan, Demuth, Beale, De Jesús, and it's a step of an equality from page [368] in the pdf book.

• How can $S^{m+1}$ and $S^m$ both be the same number? What are the $m$'s actually representing? Apr 5 '21 at 4:11
• @NinadMunshi $m$ stands for a layer in a neural network (first, second, etc). $S^{something}$ is the number of neurons in each layer (1,2, etc). So we can assume an equality. Apr 5 '21 at 4:13
• But I think this can be totally separated from the ANN, and it's just math. Apr 5 '21 at 4:19
• Oh gotcha. Your simplification is wrong, you have to use the correct multivariate chain rule, which you are not. I will type up an answer below. Apr 5 '21 at 4:24

## 1 Answer

In your simplification you are assuming that

$$\frac{\partial f}{\partial y} \frac{\partial y}{\partial x} = \frac{\partial f}{\partial x}$$

as would be the case with single variable chain rule. But the problem is that partial derivatives alone cannot account for the "whole" variation. If I have a change of variables from $$(x,y)$$ to $$(s,t)$$ the correct partials by chain rule would be as follows

$$\frac{\partial f}{\partial s} = \frac{\partial f}{\partial x}\frac{\partial x}{\partial s} + \frac{\partial f}{\partial y}\frac{\partial y}{\partial s}$$

This is because when we want to account for a variation in $$s$$ we have to hunt down and add all of its contributions since a change of variable could also be written as $$f(s,t)=f(x(s,t),y(s,t))$$. In your case this means the correct chain rule would give you

$$\frac{\partial F}{\partial n_1^{m+1}}\frac{\partial n_1^{m+1}}{\partial n_1^m} + \frac{\partial F}{\partial n_2^{m+1}}\frac{\partial n_2^{m+1}}{\partial n_1^m} = \frac{\partial F}{\partial n_1^m}$$

and so on for the second term.

• Thank you for the help. Apr 5 '21 at 18:14