3
$\begingroup$

I have encountered the following equation:
$g: \mathbb{R}^m \rightarrow \mathbb{R}$
$u: \mathbb{R}^n \rightarrow \mathbb{R}^m$
$z = g(\mathbf{y})$, $\mathbf{y} = u(\mathbf{x})$

then using numerator layout $\frac{\partial z}{\partial \mathbf{x}} = \frac{\partial z}{\partial \mathbf{y}} \frac{\partial \mathbf{y}}{\partial \mathbf{x}}$ translates to: $$ \left(\begin{array}{ccc} \frac{\partial z}{\partial x_{1}} & \cdots & \frac{\partial z}{\partial x_{n}} \end{array}\right)= \left(\begin{array}{ccc} \frac{\partial z}{\partial y_{1}} & \cdots & \frac{\partial z}{\partial y_{m}} \end{array}\right) \left(\begin{array}{ccc} \frac{\partial y_{1}}{\partial x_{1}} & \cdots & \frac{\partial y_{1}}{\partial x_{n}}\\ \vdots & \ddots & \vdots\\ \frac{\partial y_{m}}{\partial x_{1}} & \cdots & \frac{\partial y_{m}}{\partial x_{n}} \end{array}\right) $$

(From https://pytorch.org/tutorials/beginner/blitz/autograd_tutorial.html#sphx-glr-beginner-blitz-autograd-tutorial-py presented in denominator notation there)

When calculating the matrix multiplication on paper I came to think that the equation must be wrong because for any element in the result vector: $ \frac{\partial z}{\partial x_i} = \left(\begin{array}{ccc} \frac{\partial z}{\partial y_{1}} & \cdots & \frac{\partial z}{\partial y_{m}} \end{array}\right) \left(\begin{array}{c} \frac{\partial y_{1}}{\partial x_i}\\ \vdots \\ \frac{\partial y_{m}}{\partial x_i} \end{array}\right)= \sum_{j=1}^m \frac{\partial z}{\partial y_j} \frac{\partial y_j}{\partial x_i} \overset{chain \\rule}{=} \sum_{j=1}^m \frac{\partial z}{\partial x_i} = m \frac{\partial z}{\partial x_i}$

But maybe I'm wrong? After all I've never seen the chain rule with $\partial$ but always with $d$. But even after research they seem to be 2 ways of expressing the same thing: $df/dx$ is the derivative of a term e.g. $2xy$ wrt. $x$ and $\partial f/\partial x$ the derivative of a function e.g. $f(x,y) = 2xy$.

Maybe this can be explained by the law of total derivatives.

However this law is sometimes presented using only $\partial$ s, sometimes using a mix like this $ \frac{dz}{dx_i} = \sum_{j=1}^m \frac{\partial z}{\partial y_j} \frac{dy_j}{dx_i} $, further implying that $d$, $\partial$ is really just a style choice.

But if it is, what's keeping me from replacing all $\partial$ with $d$ and applying the chain rule like this:

$$ \frac{dz}{dx_i} = \sum_{j=1}^m \frac{dz}{dy_j} \frac{dy_j}{dx_i} = \sum_{j=1}^m \frac{dz}{dx_i} = m \frac{dz}{dx_i} $$

Questions:
Is the Equation $\frac{\partial z}{\partial \mathbf{x}} = \frac{\partial z}{\partial \mathbf{y}} \frac{\partial \mathbf{y}}{\partial \mathbf{x}}$ correct?
If so, is $\frac{\partial y}{\partial x}$ equivalent to $\frac{dy}{dx}$?
If so, what am I doing wrong when applying the chain rule?

$\endgroup$
  • $\begingroup$ What do you mean by $dz/dx_i$? $\endgroup$ – Alex Provost Jan 21 at 3:11
  • $\begingroup$ $dz/dx_i$ is an ordinary differential, a term with variables $y_i$ that are dependent on $x_i$s. Opposed to the partial differential, a function of parameters $y_i$, dependent on $x_i$. As I understand only ordinary differentials can be reduced via chain rule. $\endgroup$ – user2740 Jan 21 at 11:04
  • $\begingroup$ I don't know what you mean by "ordinary differential" in this context. What do you mean precisely by $dz/dx_i$ if not $\partial z/\partial x_i$? $\endgroup$ – Alex Provost Jan 21 at 12:59
  • $\begingroup$ Initially I thought so too, I've edited the question for clarity. $\endgroup$ – user2740 Jan 21 at 14:46
2
$\begingroup$

What is the difference between an "ordinary" derivative and a partial derivative?

When a function depends on more than one variable, we use partial derivatives, but why? In fact, the definition of the partial derivative looks just like that of the normal derivative. So why the distinction?

The reason is that the partial derivative depends on more than the two variables shown. For example, suppose we define the function $$f : \Bbb R^2 \to \Bbb R : (x,y) \mapsto xy$$ Now this expression for $f$ uses cartesian coordinates. But what if we use a different coordinate system instead, by replacing $y$ with $z = x + y$? Then $$f(x,z) = x(z - x)$$

What is $\frac{\partial f}{\partial x}$? By the $(x,y)$ expression, $$\frac{\partial f}{\partial x} = y$$ By the $(x,z)$ expression, $$\frac{\partial f}{\partial x} = z - 2x$$ But $z - 2x = y - x \ne y$, so the these two partial derivative expressions are different, even though they use the exact same symbols. This is the difference between $d$ and $\partial$. The ordinary $d$ derivative depends only on what you see - the variables in the top and the bottom. But the partial derivative symbol is there to remind you that there is additional context that is not indicated by the expression which is needed to interpret it. When I see $\frac{\partial f}{\partial x}$, before I can know what it is, I need to know what other variables will be held constant while I am differentiating with respect to $x$.

Now, there is the notation $\dfrac{d\mathbf y}{d\mathbf x}$, used where $\mathbf x$ in particular is a variable of multiple dimensions. This is justifiable, as $\mathbf x$ is the entire set of variables on which $\mathbf y$ depends. So there are no unspecified variables involved that need to be known to take the derivative.

But the same cannot be said about the expression you are referring to. $\dfrac {dz}{dx_i}$ is not defined. Period. If you have seen anyone using that expression, they are abusing notation to do so. What they mean by it is $\dfrac {\partial z}{\partial x_i}$. And similarly, for your calculation. The reduction in it does not work because it is not true that $$\frac{dz}{dy_j} \frac{dy_j}{dx_i} = \frac{dz}{dx_i}$$ since this actually means $$\frac{\partial z}{\partial y_j} \frac{\partial y_j}{\partial x_i} = \frac{\partial z}{\partial x_i}$$ and that is not how the multi-variate chain rule works.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.