Partial derivative in gradient descent for two variables I've started taking an online machine learning class, and the first learning algorithm that we are going to be using is a form of linear regression using gradient descent. I don't have much of a background in high level math, but here is what I understand so far.
Given $m$ number of items in our learning set, with $x$ and $y$ values, we must find the best fit line $h_\theta(x) = \theta_0+\theta_1x$ . The cost function for any guess of $\theta_0,\theta_1$ can be computed as:
$$J(\theta_0,\theta_1) = \frac{1}{2m}\sum_{i=1}^m(h_\theta(x^{(i)}) - y^{(i)})^2$$
where $x^{(i)}$ and $y^{(i)}$ are the $x$ and $y$ values for the $i^{th}$ component in the learning set. If we substitute for $h_\theta(x)$,
$$J(\theta_0,\theta_1) = \frac{1}{2m}\sum_{i=1}^m(\theta_0 + \theta_1x^{(i)} - y^{(i)})^2$$
Then, the goal of gradient descent can be expressed as
$$\min_{\theta_0, \theta_1}\;J(\theta_0, \theta_1)$$
Finally, each step in the gradient descent can be described as:
$$\theta_j := \theta_j - \alpha\frac{\partial}{\partial\theta_j} J(\theta_0,\theta_1)$$
for $j = 0$ and $j = 1$ with $\alpha$ being a constant representing the rate of step. 
I have no idea how to do the partial derivative. I have never taken calculus, but conceptually I understand what a derivative represents. The instructor gives us the partial derivatives for both $\theta_0$ and $\theta_1$ and says not to worry if we don't know how it was derived. (I suppose, technically, it is a computer class, not a mathematics class) However, I would very much like to understand this if possible. Could someone show how the partial derivative could be taken, or link to some resource that I could use to learn more? I apologize if I haven't used the correct terminology in my question; I'm very new to this subject.
 A: Despite the popularity of the top answer, it has some major errors. The most fundamental problem is that $g(f^{(i)}(\theta_0, \theta_1))$ isn't even defined, much less equal to the original function. The focus on the chain rule as a crucial component is correct, but the actual derivation is not right at all.
So I'll give a correct derivation, followed by my own attempt to get across some intuition about what's going on with partial derivatives, and ending with a brief mention of a cleaner derivation using more sophisticated methods. That said, if you don't know some basic differential calculus already (at least through the chain rule), you realistically aren't going to be able to truly follow any derivation; go learn that first, from literally any calculus resource you can find, if you really want to know.
For completeness, the properties of the derivative that we need are that for any constant $c$ and functions $f(x)$ and $g(x)$,
$$\frac{d}{dx} c = 0, \ \frac{d}{dx} x = 1,$$
$$\frac{d}{dx} [c\cdot f(x)] = c\cdot\frac{df}{dx} \ \ \ \text{(linearity)},$$
$$\frac{d}{dx}[f(x)+g(x)] = \frac{df}{dx} + \frac{dg}{dx} \ \ \ \text{(linearity)},$$
$$\frac{d}{dx}[f(x)]^2 = 2f(x)\cdot\frac{df}{dx} \ \ \ \text{(chain rule)}.$$
Taking partial derivatives works essentially the same way, except that the notation $\frac{\partial}{\partial x}f(x,y)$ means we we take the derivative by treating $x$ as a variable and $y$ as a constant using the same rules listed above (and vice versa for $\frac{\partial}{\partial y}f(x,y)$).

Derivation
We have
$$h_\theta(x_i) = \theta_0 + \theta_1 x_i$$
and
$$\begin{equation} J(\theta_0, \theta_1) = \frac{1}{2m} \sum_{i=1}^m (h_\theta(x_i)-y_i)^2\end{equation}.$$
We first compute
$$\frac{\partial}{\partial\theta_0}h_\theta(x_i)=\frac{\partial}{\partial\theta_0}(\theta_0 + \theta_1 x_i)=\frac{\partial}{\partial\theta_0}\theta_0 + \frac{\partial}{\partial\theta_0}\theta_1 x_i =1+0=1,$$
$$\frac{\partial}{\partial\theta_1}h_\theta(x_i) =\frac{\partial}{\partial\theta_1}(\theta_0 + \theta_1 x_i)=\frac{\partial}{\partial\theta_1}\theta_0 + \frac{\partial}{\partial\theta_1}\theta_1 x_i =0+x_i=x_i,$$
which we will use later. Now we want to compute the partial derivatives of $J(\theta_0, \theta_1)$. We can actually do both at once since, for $j = 0, 1,$
$$\frac{\partial}{\partial\theta_j} J(\theta_0, \theta_1) = \frac{\partial}{\partial\theta_j}\left[\frac{1}{2m} \sum_{i=1}^m (h_\theta(x_i)-y_i)^2\right]$$
$$= \frac{1}{2m} \sum_{i=1}^m \frac{\partial}{\partial\theta_j}(h_\theta(x_i)-y_i)^2 \  \text{(by linearity of the derivative)}$$
$$= \frac{1}{2m} \sum_{i=1}^m 2(h_\theta(x_i)-y_i)\frac{\partial}{\partial\theta_j}(h_\theta(x_i)-y_i) \ \text{(by the chain rule)}$$ 
$$= \frac{1}{2m}\cdot 2\sum_{i=1}^m (h_\theta(x_i)-y_i)\left[\frac{\partial}{\partial\theta_j}h_\theta(x_i)-\frac{\partial}{\partial\theta_j}y_i\right]$$
$$= \frac{1}{m}\sum_{i=1}^m (h_\theta(x_i)-y_i)\left[\frac{\partial}{\partial\theta_j}h_\theta(x_i)-0\right]$$
$$=\frac{1}{m} \sum_{i=1}^m (h_\theta(x_i)-y_i)\frac{\partial}{\partial\theta_j}h_\theta(x_i).$$
Finally substituting for $\frac{\partial}{\partial\theta_j}h_\theta(x_i)$ gives us
$$\frac{\partial}{\partial\theta_0} J(\theta_0, \theta_1) = \frac{1}{m} \sum_{i=1}^m (h_\theta(x_i)-y_i),$$
$$\frac{\partial}{\partial\theta_1} J(\theta_0, \theta_1) = \frac{1}{m} \sum_{i=1}^m (h_\theta(x_i)-y_i)x_i.$$

Intuition for partial derivatives
So what are partial derivatives anyway? In one variable, we can assign a single number to a function $f(x)$ to best describe the rate at which that function is changing at a given value of $x$; this is precisely the derivative $\frac{df}{dx}$of $f$ at that point. We would like to do something similar with functions of several variables, say $g(x,y)$, but we immediately run into a problem. In one variable, we can only change the independent variable in two directions, forward and backwards, and the change in $f$ is equal and opposite in these two cases. (For example, if $f$ is increasing at a rate of 2 per unit increase in $x$, then it's decreasing at a rate of 2 per unit decrease in $x$.)
With more variables we suddenly have infinitely many different directions in which we can move from a given point and we may have different rates of change depending on which direction we choose. So a single number will no longer capture how a multi-variable function is changing at a given point. However, there are certain specific directions that are easy (well, easier) and natural to work with: the ones that run parallel to the coordinate axes of our independent variables. These resulting rates of change are called partial derivatives. (For example, $g(x,y)$ has partial derivatives $\frac{\partial g}{\partial x}$ and $\frac{\partial g}{\partial y}$ from moving parallel to the x and y axes, respectively.) Even though there are infinitely many different directions one can go in, it turns out that these partial derivatives give us enough information to compute the rate of change for any other direction. (Strictly speaking, this is a slight white lie. There are functions where the all the partial derivatives exist at a point, but the function is not considered differentiable at that point. This happens when the graph is not sufficiently "smooth" there.)
In particular, the gradient $\nabla g = (\frac{\partial g}{\partial x}, \frac{\partial g}{\partial y})$ specifies the direction in which g increases most rapidly at a given point and $-\nabla g = (-\frac{\partial g}{\partial x}, -\frac{\partial g}{\partial y})$ gives the direction in which g decreases most rapidly; this latter direction is the one we want for gradient descent. This makes sense for this context, because we want to decrease the cost and ideally as quickly as possible.

A higher level approach
For the interested, there is a way to view $J$ as a simple composition, namely
$$J(\mathbf{\theta}) = \frac{1}{2m} \|\mathbf{h_\theta}(\mathbf{x})-\mathbf{y}\|^2 = \frac{1}{2m} \|X\mathbf{\theta}-\mathbf{y}\|^2.$$
Note that $\mathbf{\theta}$, $\mathbf{h_\theta}(\mathbf{x})$, $\mathbf{x}$, and $\mathbf{y}$, are now vectors. Using more advanced notions of the derivative (i.e. the total derivative or Jacobian), the multivariable chain rule, and a tiny bit of linear algebra, one can actually differentiate this directly to get
$$\frac{\partial J}{\partial\mathbf{\theta}} = \frac{1}{m}(X\mathbf{\theta}-\mathbf{y})^\top X.$$
The transpose of this is the gradient $\nabla_\theta J = \frac{1}{m}X^\top (X\mathbf{\theta}-\mathbf{y})$. Setting this gradient equal to $\mathbf{0}$ and solving for $\mathbf{\theta}$ is in fact exactly how one derives the explicit formula for linear regression.
A: 
conceptually I understand what a derivative represents. 

So let us start from that. Consider a function $\theta\mapsto F(\theta)$ of a parameter $\theta$, defined at least on an interval $(\theta_*-\varepsilon,\theta_*+\varepsilon)$ around the point $\theta_*$. Then the derivative of $F$ at $\theta_*$, when it exists, is the number
$$
F'(\theta_*)=\lim\limits_{\theta\to\theta_*}\frac{F(\theta)-F(\theta_*)}{\theta-\theta_*}.
$$
Less formally, you want $F(\theta)-F(\theta_*)-F'(\theta_*)(\theta-\theta_*)$ to be small with respect to $\theta-\theta_*$ when $\theta$ is close to $\theta_*$.
One can also do this with a function of several parameters, fixing every parameter except one. The result is called a partial derivative. In your setting, $J$ depends on two parameters, hence one can fix the second one to $\theta_1$ and consider the function $F:\theta\mapsto J(\theta,\theta_1)$. If $F$ has a derivative $F'(\theta_0)$ at a point $\theta_0$, its value is denoted by $\dfrac{\partial}{\partial \theta_0}J(\theta_0,\theta_1)$. 
Or, one can fix the first parameter to $\theta_0$ and consider the function $G:\theta\mapsto J(\theta_0,\theta)$. If $G$ has a derivative $G'(\theta_1)$ at a point $\theta_1$, its value is denoted by $\dfrac{\partial}{\partial \theta_1}J(\theta_0,\theta_1)$.
You consider a function $J$ linear combination of functions $K:(\theta_0,\theta_1)\mapsto(\theta_0+a\theta_1-b)^2$. Derivatives and partial derivatives being linear functionals of the function, one can consider each function $K$ separately. But, the derivative of $t\mapsto t^2$ being $t\mapsto2t$, one sees that $\dfrac{\partial}{\partial \theta_0}K(\theta_0,\theta_1)=2(\theta_0+a\theta_1-b)$ and $\dfrac{\partial}{\partial \theta_1}K(\theta_0,\theta_1)=2a(\theta_0+a\theta_1-b)$.
