What does the constant mean in Big O notation? I have a big issue in understanding the real meaning of Big O notation. 
Classical definition: $f(x) = O(g(x))$ as $x\rightarrow k$ if there exist $\delta, C > 0$ such that $f(x) \leq Cg(x)$ whenever $|x-k| < \delta$.
I understand that the inequality holds when $|x-k| < \delta$ so we have an upper bound in that interval for $f(x)$ function. However, as there is a constant in the inequality I do not understand how it can help us to control the growth of the function. For example, if $C = 10^{100}$ then the bound would be quite different from $C = 1$ so I do not really know how big the bound of my function is.
The problem is related to finite difference method with Taylor series, $f\in C^2$: $$\frac{f(x+h) - f(x)}{h} - f'(x) = O(h)$$ which means that as $h\rightarrow 0$ if there exist $\delta, C > 0$ such that $\frac{f(x+h) - f(x)}{h} - f'(x) \leq Cx$ whenever $h < \delta$. So, the error between the real derivative and approximation is bounded by $Cx$ but that bound is quite different if $C = 10^{100}$ or $1$. So, how Big O notation tells me the maximum error that I will have? That error could be as big as wanted making C big enough?
Thanks a lot!
 A: Big-$O$ notation doesn't tell you anything about the maximum error you can have; it tells you about the asymptotic behavior of the function. It tells you that there is a constant $C$; for a great many purposes you don't need to have any specific value for $C$, you simply need to know it exists. Any particular $C$ that fits into the definition would put a bound on the maximum error you could have, but the big-$O$ doesn't give you any hints about what $C$'s fit.
e.g. if you want to show that, if $f(x) = x^2$, then $f'(x) = 2x$, then all you need to know is that
$$ 0 \leq \left|\frac{(x+h)^2 - x^2}{h} - 2x \right| < C h $$
for some constant $C$. It doesn't matter what value the constant has, just that some constant exists. Then you could take the limit to get
$$ \lim_{h \to 0} 0 \leq \lim_{h \to 0}\left|\frac{(x+h)^2 - x^2}{h} - 2x \right| \leq \lim_{h \to 0}  Ch $$
and thus
$$ 0 \leq \left| \left( \lim_{h \to 0} \frac{(x+h)^2 - x^2}{h} \right) - 2x \right| \leq 0 $$
and
$$ \lim_{h \to 0} \frac{(x+h)^2 - x^2}{h} = 2x $$
A: I think you are mixing two definitions - the definition of big-O and the definition of limit, expressed in $(\epsilon,\delta$) language. There a slight, but very important, difference between these definitions. For the big-O you need to find both values $\epsilon > 0$ and $\delta > 0$, satisfying the inequality, at once. For the limit you need for every $\epsilon > 0$ (which can be as small as possible) find a $\delta > 0$ (depending on this $\epsilon$), satisfying the inequality.
There is good description of the definition of limit in the Wiki.
Additionally, the big-O notation used to be used for Asymptotic Analysis (when your $x \to \infty$) - there is no sense to use it for Taylor series.
Also, please clean your question. For example, the first statement says about $x \to k$, but you never use this $k$ after that. In the derivative definition you also mix $x$ and $h$ - there is $h \to 0$, not $x$.
