Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I've started taking an online machine learning class, and the first learning algorithm that we are going to be using is a form of linear regression using gradient descent. I don't have much of a background in high level math, but here is what I understand so far.

Given $m$ number of items in our learning set, with $x$ and $y$ values, we must find the best fit line $h_\theta(x) = \theta_0+\theta_1x$ . The cost function for any guess of $\theta_0,\theta_1$ can be computed as:

$J(\theta_0,\theta_1) = \frac{1}{2m}\displaystyle\sum_{i=1}^m(h_\theta(x^{(i)}) - y^{(i)})^2$

where $x^{(i)}$ and $y^{(i)}$ are the $x$ and $y$ values for the $i^{th}$ component in the learning set. If we substitute for $h_\theta(x)$,

$J(\theta_0,\theta_1) = \frac{1}{2m}\displaystyle\sum_{i=1}^m(\theta_0 + \theta_1x^{(i)} - y^{(i)})^2$

Then, the goal of gradient descent can be expressed as

$\displaystyle\min_{\theta_0, \theta_1}\;J(\theta_0, \theta_1)$

Finally, each step in the gradient descent can be described as:

$\theta_j := \theta_j - \alpha\frac{\partial}{\partial\theta_j} J(\theta_0,\theta_1)$

for $j = 0$ and $j = 1$ with $\alpha$ being a constant representing the rate of step.

I have no idea how to do the partial derivative. I have never taken calculus, but conceptually I understand what a derivative represents. The instructor gives us the partial derivatives for both $\theta_0$ and $\theta_1$ and says not to worry if we don't know how it was derived. (I suppose, technically, it is a computer class, not a mathematics class) However, I would very much like to understand this if possible. Could someone show how the partial derivative could be taken, or link to some resource that I could use to learn more? I apologize if I haven't used the correct terminology in my question; I'm very new to this subject.

share|cite|improve this question
What do you know? Do you know derivatives at all? – Phira Oct 7 '11 at 23:43
No, I do not, currently. – voithos Oct 8 '11 at 3:26
up vote 94 down vote accepted

The answer above is a good one, but I thought I'd add in some more "layman's" terms that helped me better understand concepts of partial derivatives. The answers I've seen here and in the Coursera forums leave out talking about the chain rule, which is important to know if you're going to get what this is doing...

It's helpful for me to think of partial derivatives this way: the variable you're focusing on is treated as a variable, the other terms just numbers. Other key concepts that are helpful:

  • For "regular derivatives" of a simple form like $F(x) = cx^n$ , the derivative is simply $F'(x) = cn \times x^{n-1}$
  • The derivative of a constant (a number) is 0.
  • Summations are just passed on in derivatives; they don't affect the derivative. Just copy them down in place as you derive.

Also, it should be mentioned that the chain rule is being used. The chain rule says that (in clunky laymans terms), for $g(f(x))$, you take the derivative of $g(f(x))$, treating $f(x)$ as the variable, and then multiply by the derivative of $f(x)$. For our cost function, think of it this way:

$$ g(\theta_0, \theta_1) = \frac{1}{2m} \sum_{i=1}^m \left(f(\theta_0, \theta_1)^{(i)}\right)^2 \tag{1}$$

$$ f(\theta_0, \theta_1)^{(i)} = \theta_0 + \theta_{1}x^{(i)} - y^{(i)} \tag{2}$$

To show I'm not pulling funny business, sub in the definition of $f(\theta_0, \theta_1)^{(i)}$ into the definition of $g(\theta_0, \theta_1)$ and you get:

$$ g(f(\theta_0, \theta_1)^{(i)}) = \frac{1}{2m} \sum_{i=1}^m \left(\theta_0 + \theta_{1}x^{(i)} - y^{(i)}\right)^2 \tag{3}$$

This is, indeed, our entire cost function.

Thus, the partial derivatives work like this:

$$ \frac{\partial}{\partial \theta_0} g(\theta_0, \theta_1) = \frac{\partial}{\partial \theta_0} \frac{1}{2m} \sum_{i=1}^m \left(f(\theta_0, \theta_1)^{(i)}\right)^2 = 2 \times \frac{1}{2m} \sum_{i=1}^m f(\theta_0, \theta_1)^{2-1} \frac{\partial}{\partial \theta_0}\theta_0 = \tag{4}$$

$$\frac{1}{m} \sum_{i=1}^m f(\theta_0, \theta_1)^{(i)}$$

In other words, just treat $f(\theta_0, \theta_1)^{(i)}$ like a variable and you have a simple derivative of $\frac{1}{2m} x^2 = \frac{1}{m}x$

$$ \frac{\partial}{\partial \theta_0} f(\theta_0, \theta_1)^{(i)} = \frac{\partial}{\partial \theta_0} (\theta_0 + \theta_{1}x^{(i)} - y^{(i)}) \tag{5}$$

And $\theta_1, x$, and $y$ are just "a number" since we're taking the derivative with respect to $\theta_0$, so the partial of $g(\theta_0, \theta_1)$ becomes:

$$ \frac{\partial}{\partial \theta_0} f(\theta_0, \theta_1) = \frac{\partial}{\partial \theta_0} (\theta_0 + [a \ number][a \ number]^{(i)} - [a \ number]^{(i)}) = \frac{\partial}{\partial \theta_0} \theta_0 = 1 \tag{6}$$

So, using the chain rule, we have:

$$ \frac{\partial}{\partial \theta_0} g(f(\theta_0, \theta_1)^{(i)}) = \frac{\partial}{\partial \theta_0} g(\theta_0, \theta_1) \frac{\partial}{\partial \theta_0}f(\theta_0, \theta_1)^{(i)} \tag{7}$$

And subbing in the partials of $g(\theta_0, \theta_1)$ and $f(\theta_0, \theta_1)^{(i)}$ from above, we have:

$$ \frac{1}{m} \sum_{i=1}^m f(\theta_0, \theta_1)^{(i)} \frac{\partial}{\partial \theta_0}f(\theta_0, \theta_1)^{(i)} = \frac{1}{m} \sum_{i=1}^m \left(\theta_0 + \theta_{1}x^{(i)} - y^{(i)}\right) \times 1 = \tag{8}$$

$$ \frac{1}{m} \sum_{i=1}^m \left(\theta_0 + \theta_{1}x^{(i)} - y^{(i)}\right)$$

What about the derivative with respect to $\theta_1$?

Our term $g(\theta_0, \theta_1)$ is identical, so we just need to take the derivative of $f(\theta_0, \theta_1)^{(i)}$, this time treating $\theta_1$ as the variable and the other terms as "just a number." That goes like this:

$$ \frac{\partial}{\partial \theta_1} f(\theta_0, \theta_1)^{(i)} = \frac{\partial}{\partial \theta_1} (\theta_0 + \theta_{1}x^{(i)} - y^{(i)}) \tag{9}$$

$$ \frac{\partial}{\partial \theta_1} f(\theta_0, \theta_1)^{(i)} = \frac{\partial}{\partial \theta_1} ([a \ number] + \theta_{1}[a \ number, x^{(i)}] - [a \ number]) \tag{10}$$

Note that the "just a number", $x^{(i)}$, is important in this case because the derivative of $c \times x$ (where $c$ is some number) is $\frac{d}{dx}(c \times x^1) = c \times 1 \times x^{(1-1=0)} = c \times 1 \times 1 = c$, so the number will carry through. In this case that number is $x^{(i)}$ so we need to keep it. Thus, our derivative is:

$$ \frac{\partial}{\partial \theta_1} f(\theta_0, \theta_1)^{(i)} = 0 + (\theta_{1})^1 x^{(i)} - 0 = 1 \times \theta_1^{(1-1=0)} x^{(i)} = 1 \times 1 \times x^{(i)} = x^{(i)} \tag{11}$$

Thus, the entire answer becomes:

$$ \frac{\partial}{\partial \theta_1} g(f(\theta_0, \theta_1)^{(i)}) = \frac{\partial}{\partial \theta_1} g(\theta_0, \theta_1) \frac{\partial}{\partial \theta_1} f(\theta_0, \theta_1)^{(i)} = \tag{12}$$

$$\frac{1}{m} \sum_{i=1}^m f(\theta_0, \theta_1)^{(i)} \frac{\partial}{\partial \theta_1}f(\theta_0, \theta_1)^{(i)} = \frac{1}{m} \sum_{i=1}^m \left(\theta_0 + \theta_{1}x^{(i)} - y^{(i)}\right) x^{(i)}$$

share|cite|improve this answer
Thank you for this! I must say, I appreciate it even more when I consider how long it has been since I asked this question. In reality, I have never had any formal training in any form of calculus (not even high-school level, sad to say), so, while I perhaps understood the concept, the math itself has always been a bit fuzzy. Interestingly enough, I started trying to learn basic differential (univariate) calculus around 2 weeks ago, and I think you may have given me a sneak peek. I'm glad to say that your answer was very helpful, thinking back on the course. Once more, thank you! – voithos Sep 2 '12 at 2:36
When you were explaining the derivation of $\frac{\partial}{\partial \theta_0}$, in the final form you retained the $\frac{1}{2m}$ while at the same time having $\frac{1}{m}$ as the outer term. I suspect this is a simple transcription error? – voithos Sep 2 '12 at 2:44
@voithos yup -- good catch. Just noticed that myself on the Coursera forums where I cross posted. Typing in LaTeX is tricky business! Thanks for letting me know. – Hendy Sep 2 '12 at 15:53
@voithos: also, I posted so long after because I just started the same class on it's next go-around. I have taken multivariable calculus and didn't get the derivation... so I re-taught myself. I knew the gist of taking a partial with respect to a variable, but the chain rule part was missing. After I figured it out, I added the same answer to the Coursera forum where someone had asked the same, and that linked here. The other answer is plenty fine, but it wouldn't quite have gotten things to click for me, so I thought I'd give it a whirl in my own words. – Hendy Sep 3 '12 at 1:01
I see - best wishes on your classes! 'Tis exciting times indeed. – voithos Sep 3 '12 at 1:29

conceptually I understand what a derivative represents.

So let us start from that. Consider a function $\theta\mapsto F(\theta)$ of a parameter $\theta$, defined at least on an interval $(\theta_*-\varepsilon,\theta_*+\varepsilon)$ around the point $\theta_*$. Then the derivative of $F$ at $\theta_*$, when it exists, is the number $$ F'(\theta_*)=\lim\limits_{\theta\to\theta_*}\frac{F(\theta)-F(\theta_*)}{\theta-\theta_*}. $$ Less formally, you want $F(\theta)-F(\theta_*)-F'(\theta_*)(\theta-\theta_*)$ to be small with respect to $\theta-\theta_*$ when $\theta$ is close to $\theta_*$.

One can also do this with a function of several parameters, fixing every parameter except one. The result is called a partial derivative. In your setting, $J$ depends on two parameters, hence one can fix the second one to $\theta_1$ and consider the function $F:\theta\mapsto J(\theta,\theta_1)$. If $F$ has a derivative $F'(\theta_0)$ at a point $\theta_0$, its value is denoted by $\dfrac{\partial}{\partial \theta_0}J(\theta_0,\theta_1)$.

Or, one can fix the first parameter to $\theta_0$ and consider the function $G:\theta\mapsto J(\theta_0,\theta)$. If $G$ has a derivative $G'(\theta_1)$ at a point $\theta_1$, its value is denoted by $\dfrac{\partial}{\partial \theta_1}J(\theta_0,\theta_1)$.

You consider a function $J$ linear combination of functions $K:(\theta_0,\theta_1)\mapsto(\theta_0+a\theta_1-b)^2$. Derivatives and partial derivatives being linear functionals of the function, one can consider each function $K$ separately. But, the derivative of $t\mapsto t^2$ being $t\mapsto2t$, one sees that $\dfrac{\partial}{\partial \theta_0}K(\theta_0,\theta_1)=2(\theta_0+a\theta_1-b)$ and $\dfrac{\partial}{\partial \theta_1}K(\theta_0,\theta_1)=2a(\theta_0+a\theta_1-b)$.

share|cite|improve this answer
After continuing more in the class, hitting some online reference materials, and coming back to reread your answer, I think I finally understand these constructs, to some extent. Thank you for the explanation. – voithos Nov 29 '11 at 17:50

Despite the popularity of the top answer, it has some major errors. I've submitted an edit to correct it, but who knows if it will be approved. The most fundamental problem is that $g(f(\theta_0, \theta_1)^{(i)})$ isn't even defined, much less equal to the original function. The focus on the chain rule is correct, but the actual derivation is not right at all.

So I'll give a correct derivation, followed by my own attempt to get across some intuition about what's going on with partial derivatives.

We have

$$h_\theta(x_i) = \theta_0 + \theta_1 x_i$$


$$\begin{equation} J(\theta_0, \theta_1) = \frac{1}{2m} \sum_{i=1}^m (h_\theta(x_i)-y_i)^2\end{equation}.$$

We then obtain

$$\frac{\partial}{\partial\theta_0}h_\theta(x_i)=1, \ \frac{\partial}{\partial\theta_1}h_\theta(x_i) = x_i,$$

and for $j = 0, 1,$

$$\frac{\partial}{\partial\theta_j} J(\theta_0, \theta_1) = \frac{\partial}{\partial\theta_j}\frac{1}{2m} \sum_{i=1}^m (h_\theta(x_i)-y_i)^2 = \frac{1}{2m} \sum_{i=1}^m \frac{\partial}{\partial\theta_j}(h_\theta(x_i)-y_i)^2$$

$$= \frac{1}{2m} \sum_{i=1}^m 2(h_\theta(x_i)-y_i)^{2-1}\frac{\partial}{\partial\theta_j}(h_\theta(x_i)-y_i) \ \text{(by the Chain Rule)}$$

$$=\frac{1}{m} \sum_{i=1}^m (h_\theta(x_i)-y_i)\frac{\partial}{\partial\theta_j}h_\theta(x_i)$$

Finally substituting for $\frac{\partial}{\partial\theta_j}h_\theta(x_i)$ gives us what we want.

So what's the deal with partial derivatives anyway? In one variable, we can assign a single number to a function $f(x)$ to best describe the rate at which that function is changing at a given value of $x$; this is precisely the derivative of f at that point. We would like to do something similar with functions of several variables, for example some $g(x,y)$, but we immediately run into a problem. In one variable, we can only change the independent variable in two directions, forward and backwards, and the rate of change is equal and opposite in these two cases. (For example, if f is increasing at a rate of 2 per unit change in $x$ to the right, then it's decreasing at a rate of -2 per unit change in $x$ to the left.)

With more variables we suddenly have infinitely many different directions in which we can move from a given point and we may have different rates of change depending on which direction we choose. So a single number will no longer capture how a multi-variable function is changing at a given point. However, there are certain specific directions that are easy (well, easier) and natural to work with: the ones that run parallel to the coordinate axes of our dependent variables. These resulting rates of change are called partial derivatives. (For example, $g(x,y)$ has partial derivatives $\frac{\partial g}{\partial x}$ and $\frac{\partial g}{\partial y}$ from moving parallel to the x and y axes, respectively.) Even though there are infinitely many different directions one can go in, it turns out that these partial derivatives give us enough information to compute the rate of change for any other direction.

In particular, the gradient $\nabla g = (\frac{\partial g}{\partial x}, \frac{\partial g}{\partial y})$ specifies the direction in which g increases most rapidly at a given point and $-\nabla g = (-\frac{\partial g}{\partial x}, -\frac{\partial g}{\partial y})$ gives the direction in which g decreases most rapidly; this latter direction is the one we want for gradient descent.

share|cite|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.