Where do the higher order terms in Taylor series come from? I can see the first order approximation for a Taylor series: if we want to approximate $f(x)$ near $x_0$, then it's close to the line with slope $f'(x_0)$ that intersects $f(x_0)$, giving 
$f(x) \approx f(x_0) + f'(x_0)(x-x_0)$
However, I don't see a good way to understand the higher order terms $f^{(n)}(x_0)/n! (x-x_0)$. I understand the proof of Taylor's theorem, but I don't have any real intuition for it. How were Taylor expansions discovered (i.e. what would lead one to guess the form of Taylor expansions)?
 A: Making several assumptions you can argue as follows.
First of all to simplify the reasoning we consider the case $x_0=0$. One way to get such coefficients is to assume that $f$ is "well approximated" by polynomials, which could be for example:
for any $n\in\mathbb{N}$ we have 
$$
f(x)=a_0+a_1x+a_2x^2+\ldots a_nx^n+R(x) \qquad (1),
$$
where $R(x)/x^n\to 0$ when $x\to 0$.  
Now supposing that the function $R$ is $n$ times differentiable at the origin we get  by computing successive derivatives of both sides of (1) at $x=0$, that $a_j$ has to be the Taylor coefficients. 
A: I'll contribute with my grain of sand.
When I studied Taylor polynomials, this was the idea under it (or what I got):
Suppose $f(x)$ is an $n$ times differentiable function, continuous in a neighborhood of $x=a$. Suppose we build up another polynomial function $p(x)$, such that $f^{(n)}(a) = p^{(n)}(a)$ for $0,\dots,n$. Then $p$ should approximate $f$ with a good degree of accuracy. Why is this so? If the functions coincide in $a$ then they are equal in $a$. If they have the same derivative, then they approach $(a,f(a))$ in a similar manner. If they have the same second derivative, then they have similar curvature and their derivative behaves similarly in that neighborhood.
The idea is that each higher order term "extends" the neighborhood in which the functions are more and more like each other, and provides a higher accuracy (this can be explained by finding a formula for the error produced by the approximation, which is very interesting theory too).
As a bonus, you can inductively check that if we build up a polynomial around $x=a$
$$p_a^n(x) = a_0 + a_1 (x-a)+a_2(x-a)^2+\cdots+a_n(x-a)^n$$
then by differentiating and equating using our condition we'll find
$$a_n = \frac{f^{(n)}(a)}{n!}$$
for ever coefficient of the polynomial.
Introducing some more theory, we can talk about Lanadu's $o$ notation. The notation is basically the following:

We say $f(x)=o(g(x))$ as $x \to a$, read "$f$ is little $o$ of $g$ when $x$ approaches $a$" if
$$\lim\limits_{x \to a} \frac{f(x)}{g(x)} \to 0$$

If you recall comparing infinitesimals, this means that $f$ is of smaller order than $g$ in $a$.
We can use this notation to write $\sin x \sim x$, as
$$\sin x = x + o(x) \text{ for } x \to 0$$
This means
$$\lim\limits_{x \to 0} \frac{\sin x -x}{x} = 0$$
$$\lim\limits_{x \to 0} \frac{\sin x}{x}-1 = 0$$
$$\lim\limits_{x \to 0} \frac{\sin x}{x}=1 $$
Which I guess isn't too new news.
So, how can we use this with Taylor's polynomial approximations?
Let $p_a^n(x)$ be the Taylor polynomial of order $n$ around $a$ of an $n$ times differentiable function. Then, what we can write is that
$$f(x)=p_a^n(x)+o((x-a)^n)$$
This means that the difference $f-p$ is of smaller order than $(x-a)^n$ for $x \to a$, or that the error is small in comparison to $(x-a)^n$.
As an example
$$\cos x = 1-\frac{x^2}{2}+o(x^2)$$
(That is why, when handling limits with polynomials of second or first or zero degree, we can substitute $\cos x$ with the given expression.$)
I hope this helps you, but remember, I'm not a professor as Arturo is. I welcome reviews or edits of this by anyone which feels can improve the ideas or wording, for example.
A: I think the most naive approximation method leads to polynomials all by itself - without explicitly trying to approximate our initial function $f$ by polynomials at all:
The most primitive approximation of a continuous function around $0$ is simply given by $f(x) \approx f(0)$. Of course we are often not very satisfied with this, so it is natural to ask what the error $f(x) - f(0)$ looks like for growing $x$, i.e. we want to compare $f(x)-f(0)$ to $x$ - that is, we want to investigate the expression
$$err_1(x) = \frac{f(x) - f(0)}{x}$$
If this error function is continuous in $0$, then we can use the same idea of an approximation as above and we obtain $err_1(x) \approx err_1(0) =  f'(0)$. Solving for $f(x)$, we obtain the approximation
$$f(x) = f(0) + err_1(x) x \approx f(0) + f'(0) x$$
If this approximation is still not good enough, then we can go on to investigate the error in the approximation of the first error, i.e.
$$err_2(x) = \frac{err_1(x) - err_1(0)}{x} = \frac{\frac{f(x)-f(0)}{x} - f'(0)}{x} = \frac{f(x) - f(0) - f'(0)x}{x^2}$$
If this is continuous in $0$, then - as above - we can make the approximation
$$err_2(x) \approx err_2(0) =  \lim_{x\to0} \frac{f(x) - f(0) - f'(0)x}{x^2} = \frac{f''(0)}{2}$$
So solving for $f(x)$ we obtain:
$$f(x) = f(0) + f'(0)x + err_1(x)x^2 \approx f(0) + f'(0) x + \frac{f''(0)}{2}x^2$$
and so on.
So all we ever do is approximate the error of the error of the error in the most crude way and hope that each step will lead to a better approximation of our initial function (which is exactly what one would do naively, I think). 
In particular, it need not even be clear that this leads to a polynomial approximation a priori. (I'm only writing this, because other answers seem to assume a certain form of the approximation and then show that this approximation is exactly the Taylor polynomial - it is however not clear to me, where the motivation for using polynomials rather than other functions comes from...)
A: I would like to add a follow-up mini-question on Arturo's answer. Your explanation is rock solid, though, how do you justify the aim for taking the limit $$\lim_{x\to 0}\frac{f(x) - (a+bx)}{x} = 0.$$ in the first instance? Why do you care about the relative error, instead of the absolute error (i.e. $$\lim_{x\to 0}(f(x) - (a+bx)) = 0.$$  Because the former one, not only asks for an approximation, but it also asks for an approximation such that the error decreases faster relative to how fast x approaches to the point at which we approximate.
Now it might sound silly, considering the absolute error we get a point approximation $$a=f(0).$$ Considering the relative error, we get a linear approximation. Is there an intuition for this (why does the extra consideration on relative error leads to a line instead of a point)? (This entire thing was a question)
A: Here is my intuition on the higher order terms in Taylor approximation (it depends on differential operator).
The first term is $$f_{x}dx+f_{y}dy$$, which is the differential of f(x,y), which approximately gives you the total change in the function if you increase x and y by a small amount. 
But this is only an approximation, since it doesnt take into account the potential variation in the rates of change when you change x and y by a "smaller then a small amount" so its off a bit. So you actually have to take into account the "change in change" when you increase x and y by a small amount, to do a better approximation. What is the "total change in change", if you increase x and y by a small amount? It is $$d(df)=d(f_{x}dx+f_{y}dy)==f_{xx}(dx)^2+2f_{xy}(dx)(dy)+f_{yy}(dy)^2$$. So adding that effect, you get a better approximation. But still, you are a bit off from the true value of change, because you didn't take into account "the change in change in change", which is the third differential etc.
A: I borrow this GIF from Wikipedia. As you can see, a higher order approximation is more accurate: 
A: Let's set
$$p(x) = \sum_{k\ge0}a_k (x-x_0)^k.
$$
Assume that for some $N$, we have $a_k=0$ for $k>N$, so that this is a polynomial of degree (at most) $N$. You will find that
$$p^{(n)}(x_0)=n!\cdot a_n.$$
Check this!
So if you want that $p^{(n)}(x_0)=f^{(n)}(x_0)$ you have to set
$$a_n = \frac{f^{(n)}}{n!}.$$
