Finite difference method I wanted to ask something regarding the finite difference approximation. I used the finite difference to calculate the numerical derivatives of my function. The finite difference is given by the following formula:
\begin{equation}
\frac{f(x+h)-f(x)}{h}
\end{equation}
The value of $h$ is questionable. In theory we should take it as small as possible, but I am not sure if we can just pick random different values for $h$ and try to see which one works better or if there is any "rule" or constraint to pick up a good value of $h$.
With Thanks
 A: If you are using one side finite differences for evaluating the derivative, the smallest step size should be $$h=x\sqrt{\epsilon}$$ where $\epsilon$ stands for the machine accuracy ($\epsilon$ being the smallest number such that $1+\epsilon > 1$). 
This is related to the truncation error which comes from higher terms in the Taylor series expansion $$f(x+h)=f(x)+h f'(x)+\frac 12h^2f''(x)+\frac 16h^3f'''(x)$$ whence $$\begin{equation}
\frac{f(x+h)-f(x)}{h}=f'(x)+\frac 12 h f''(x)
\end{equation}$$
If you can afford two function evaluations for the derivative, it is much better to use $$f'(x)=\begin{equation}
\frac{f(x+h)-f(x-h)}{2h}
\end{equation}$$ which will be more accurate and which will give you the second derivative for almost free. In such a case, you could choose $$h=x \sqrt[3]{\epsilon }$$
A: A similar question with a method for finding the optimal values of $h$.
In this case a process method will result in $h = \sqrt \frac{4 \epsilon}{M_2}$ where $\epsilon$ is the error in measurements of the function (usually machine accuracy, unless you measure your function with noise) and $M_2$ is a bound on the second derivative on the appropriate interval.
