Where does the $\mathcal{O}$-term come from in Taylor series? I understand Taylor series in general, but I've always been a bit uncertain about the $\mathcal{O}$-term, when I see it used in Taylor series. For example in one of my study materials I have the following: 

From Taylor's theorem, for small $\varepsilon$
$$f(x^*+\varepsilon X) = f(x^*)+\varepsilon
 Xf'(x^*)+\mathcal{O}(\varepsilon^2)$$

Could someone give me a bit intuition, where this $\mathcal{O}$-term is coming from, because I always get a bit scared when I see it used with Taylor series. I would like to finally get this 100% x) 
Thank you for your help!
 A: Well, if $f$ is analytic then $f$ has a Taylor series at $x^\ast$ of the form
$$f(x^\ast + \varepsilon X) = f(x^\ast) + f'(x^\ast) \varepsilon X  + \frac{1}{2} f''(x^\ast) (\varepsilon X)^2 + \frac{1}{6} f'''(x^\ast) (\varepsilon X)^3 + \cdots$$
So let's write $g(\varepsilon)$ for all the terms after the first two:
$$f(x^\ast + \varepsilon X) = f(x^\ast) + f'(x^\ast) \varepsilon X  + g(\varepsilon)$$
Now for $\varepsilon$ near zero, $\varepsilon$ is much larger than $\varepsilon^2, \varepsilon^3$, etc.  So the terms in $g(x)$ often don't matter if we're taking the limit of something related to $f$ as $\varepsilon \to 0$.  We want a systematic, formal way of dealing with the fact that the terms belonging to $g$ don't matter.
The way we do this is by noticing that 
$$\lim_{\varepsilon \to 0} \frac{g(x)}{\varepsilon^2} = C$$
for some nonzero constant $C$.  Any function $g(x)$ with this property is said to be $O(\varepsilon^2)$.
So when we do this we're forgetting exactly what function $f$ is, but remembering enough that we can calculate certain limits involving $f$.
