I’m currently reading Jeremy Howard’s paper “The Matrix Calculus You Need For Deep Learning” and came across a bit I don’t understand.

In section 4.4 (Vector Sum Reduction), they are calculating the derivative of $y = \operatorname{sum}(\mathbf{f}(\mathbf{x}))$, like so: \begin{align*} \frac{\partial y}{\partial \mathbf{x}} &= \left[ \frac{\partial y}{\partial x_1}, \frac{\partial y}{\partial x_2}, \dotsc, \frac{\partial y}{\partial x_n} \right] \\ &= \left[ \frac{\partial}{\partial x_1} \sum_i f_i(\mathbf{x}) , \frac{\partial}{\partial x_2} \sum_i f_i(\mathbf{x}), \dotsc, \frac{\partial}{\partial x_n} \sum_i f_i(\mathbf{x}) \right] \\ &= \left[ \sum_i \frac{\partial f_i(\mathbf{x})}{\partial x_1}, \sum_i \frac{\partial f_i(\mathbf{x})}{\partial x_2}, \dotsc, \sum_i \frac{\partial f_i(\mathbf{x})}{\partial x_n} \right] \\ \end{align*} (Original image here.)

Here, the following is noted:

Notice we were careful here to leave the parameter as a vector $\mathbf{x}$ because each function $f_i$ could use all values in the vector, not just $x_i$.

What does that mean? I thought it meant that we can’t reduce $f_i(\mathbf{x})$ to $f_i(x_i)$ to $x_i$, but right afterwards, that is precisely what they do:

Let’s look at the gradient of the simple $y = \operatorname{sum}(\mathbf{x})$. The function inside the summation is just $f_i(\mathbf{x}) = x_i$ and the gradient is then: \begin{align*} \nabla y &= \left[ \sum_i \frac{\partial f_i(\mathbf{x})}{\partial x_1}, \sum_i \frac{\partial f_i(\mathbf{x})}{\partial x_2}, \dotsc, \sum_i \frac{\partial f_i(\mathbf{x})}{\partial x_n} \right] \\ &= \left[ \sum_i \frac{\partial x_i}{\partial x_1}, \sum_i \frac{\partial x_i}{\partial x_2}, \dotsc, \sum_i \frac{\partial x_i}{\partial x_n} \right]. \end{align*}

(Original image here.)

Does it have something to do with the summation being taken out? Or am I reading this completely wrong?

  • 2
    $\begingroup$ It's reminding you that in general $f_i(\textbf x)=f_i(x_1,x_2,\ldots,x_n)$ is a function of all the variables $x_j$. Then they do a particular example where $f_i(x_1,x_2,\ldots,x_n)=x_i$ just depends on the one variable. $\endgroup$ – Lord Shark the Unknown Aug 24 '18 at 5:03
  • $\begingroup$ Ohh got it! Thank you! $\endgroup$ – General Thalion Aug 25 '18 at 8:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.