0
$\begingroup$

I'm reading this paper: https://explained.ai/matrix-calculus/index.html#sec4.3 , under the section- Derivatives Involving Scalar Expansion the following is written, and at the end of this section is the equation I've written here:

"$\vec{f}(\vec{x})=\vec{x}$ and $\vec{g}(z)= \vec{1} z$. (The notation represents a vector of ones of appropriate length.) $z$ is any scalar that doesn't depend on $x$, which is useful because then $\dfrac{\partial{z}}{\partial{x_i}}=0$ for any $x_i$ and that will simplify our partial derivative computations. (It's okay to think of variable z as a constant for our discussion here)."

$$\dfrac{\partial(f_i(x_i)\circ g_i(z))}{\partial z}=x_i\dfrac{\partial z}{\partial z}+z\dfrac{\partial x_i}{\partial z}=x_i+?$$

What's the $?$, I think it's undefined as derivative with respect to a constant is undefined, but the paper says it's $0$, how so?

Any help is highly appreciated.

$\endgroup$

1 Answer 1

0
$\begingroup$

Well, if you consider $x_i\frac{\partial z}{\partial z}+z\frac{\partial x_i}{\partial z}$, we can immediately see that $\frac{\partial z}{\partial z}=1$, since intuitively, z changes at the same rate as itself. Symbolically, we could write that $\frac{\partial x_k}{\partial x_l}=\delta^k_l$, where $\delta^k_l=1$ when $k=l$ and $0$ otherwise.

Now, it was explicitly stated that $z$ does not depend on $x$, therefore $\frac{\partial z}{\partial x_i}=0$. As $z$ does not depend on $x_i$, the converse is also true; namely, $x_i$ does not depend on $z$, since else you could express $z$ in terms of $x_i$, which would contradict the initial assumption. Hence, $\frac{\partial x_i}{\partial z}=0$. This implies that regardless of the value of $z$, the second term is zero.

$\endgroup$
2
  • $\begingroup$ I had the same direction of thinking, intuitively that seems correct as well, but z is a constant and $\dfrac{\partial{x}}{\partial{z}}$ means changing a variable with respect to a constant, shouldn't that be undefined? $\dfrac{\partial{x}}{\partial{y}}=\dfrac{1}{\dfrac{\partial{y}}{\partial{x}}}$ therefore, since $\dfrac{\partial z}{\partial x_i}=0 \implies \dfrac{\partial{x_i}}{\partial{z}}= \dfrac{1}{\dfrac{\partial{z}}{\partial{x_i}}}$ is undefined, this step may be wrong so I would really appreciate if you correct me here if I'm wrong. $\endgroup$ Jun 9, 2019 at 8:33
  • $\begingroup$ It is stated in the lines before the equation that $z$ is a scalar parameter, though, which changes matters quite a bit. I agree that taking the derivative of something with respect to a scalar is ill-defined, but taking a derivative with respect to a scalar parameter is perfectly well-defined. Think about the regular derivative in elementary calculus $\frac{df}{dx}$. In this case, $x$ can be regarded as a scalar parameter, and $\frac{df}{dx}$ is well-defined. Therefore, you are not taking the derivative with respect to a "constant" as you stated. $\endgroup$
    – bluestool
    Jun 9, 2019 at 23:27

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .