Error of central difference quotient vs forward difference quotient

Most people are familiar with this difference quotient:

$$f'(x) = \lim_{n \rightarrow \infty} \frac{f(x+h) - f(x)}{h}$$

We require that $f(x)$ is twice differentiable.

The error of that forward difference quotient is, using Taylor:

$$f'(x) = \frac{f(x+h) - f(x)}{h} + + \mathcal{O}(h)$$

Now to the central difference quotient:

We establish that:

$$f(x+h) = f(x) + hf'(x) + \mathcal{O}(h^2)$$

and

$$f(x-h) = f(x) - hf'(x) + \mathcal{O}(h^2)$$

Moving on:

$$f(x+h) - f(x-h) = 2hf'(x)$$

$$\frac{f(x+h) - f(x-h)}{2h} = f'(x)$$

Where did I make a mistake ? Because it seems like the error disappeared completely. Seems like we can differentiate any function exactly with this technique ( for any $h$).

So my question is:

1. How does the error of the forward difference quotient compare to the central difference quotient ?

2. Also another more general question: What does that error exactly mean ?

I'm a little bit confused because it seems smaller $i$ for $\mathcal{O}(h^i)$ would be better, yet some sources say a bigger $i$ is better.

A short precise sentence about that error term woudl be much appreciated

• Assuming $f(x+h) = f(x) + hf'(x) + \mathcal{O}(h^2)$ and $f(x-h) = f(x) - hf'(x) + \mathcal{O}(h^2)$ we have $f(x+h)-f(x-h)=2hf'(x)+ \mathcal{O}(h^2).$ Note that $f=\mathcal{O}(h^2)$ and $g=\mathcal{O}(h^2)$ doesn't imply $f-g\equiv 0.$ – mfl Aug 14 '18 at 19:55
• The short answer would be that $\mathcal O(h)$ is not the same thing as $\mathcal O(h)$. – Saucy O'Path Aug 14 '18 at 19:58
• my bad, so the error is the same ? why do sources say, the central quotient is better than the forward quotient ? – zython Aug 14 '18 at 20:00

I'll try to be as concise as possible. The point is that $O(h^2)$ isn't a "real" algebric term that you can cancel out that simply. I hope this example clarify the big-O notation: call $D_h^Cf(x)$ the centered finite difference approximation, you can say that:

$f'(x)-D_h^Cf(x) = O(h^2)$

This means that exists $h_0$ and $C$ such that for $0<h<h_0$, $|f'(x)-D_h^Cf(x)| \leq C\cdot h^2$. Concluding, the big-O notation tells you that the difference $f'(x)-D_h^Cf(x)$ is bounded by something which is proportional to $h^2$.

This is the reason why you can't cancel out the $O(h^2)$ in your calculations.

1)The difference between the centered (C) and forward (F) finite difference lies in the fact that those two errors has two different orders, C has order 2, while F has order 1. What does this mean? It means that when $h \rightarrow 0$ (which is generally what we do, we are interested in a small $h$) the C finite difference gives an approximation which is more accurate than the F finite difference. In other terms, if you fix $h$ small (say, $10^{-2}$) the C finite difference is closer to the exact value of $f'(x)$ rather than the F finite difference. It means you can use $h$ which is not ludicrously small to get the same degree of accuracy.
The error tells you, for a given $h$, the discrepancy between the coefficient of the approximated tangent and the actual first derivative, in a given point $x$.