Alternatives For Numerical Differentiation For a long time, the following point always confused me: In High School Calculus, we are told that almost all functions we encounter are are analytically differentiable  (i.e. have derivatives) but not all functions are analytically integrable (i.e. have integrals) - if this is true, then why do we need numerical differentiation?
Then, I found out that numerical differentiation is used in instances when we do not have the actual equation of the "function", but instead only have observed data points from this "function" instead:

Thus, it seems as Numerical Differentiation is providing us with a way to "interpolate derivatives" at arbitrary points over an observed range. This leads me to my next point: it seems that there are many instances in statistics where we are faced with a very similar problem : for example, we are often given a set of data points and are sometimes interested in interpolation, or fitting a smoothed function through these points:

As far as I know, this can be done in many different ways, such as:

*

*Polynomial Regression

*(Smoothed) Splines

*(Smoothed) Kernel Density Estimation
By fitting a smoothed function to the data (as opposed to a piecewise function), this allows us to obtain an "analytical and closed form" function that passes through the data (we can also measure the quality of how well this function fits the data) - and then we should be able to differentiate (i.e. evaluate derivatives) this function at any point that we desire.
For instance, provided if we have sufficient reasons to believe that the regression models below fit the data well - could we not just evaluate derivatives (e.g. first derivative, second derivative) of these regression models and use them as surrogates for numerical differentiation?

My Question: Is fitting a statistical model a mathematically valid alternative to numerical differentiation? The only downside to this I can think of is that this approach requires you to first fit a statistical model to the observed data (a source of error and uncertainty) and then differentiate this fitted statistical model (another source of error and uncertainty) - effectively compounding the uncertainties through error propagation. I am not sure, but perhaps standard numerical differentiation techniques (e.g. Backwards Differencing, Forward Differencing ) have less of a chance of invoking such error prorogation.
Can someone please comment on this?
Thanks!
 A: You need numerical differentiation in many cases in which value of a function is given as the result of some numerical computation (for example as a result of the finite element calculation or a result of some minimization) and no derivatives are directly available, only functional values. Then, if you need derivatives, you need to determine them numerically. Example: you do numerical minimization and then, when you find a local minimum, you need to determine the Hessian matrix in order to provide some statistics. Then you need to calculate second order derivatives in the minimum with the help of functional values in the neighborhood.
Another practical reason is the case, in which analytical derivatives exist, but the analytical formula is so difficult to get, that either you have no time to derive it or you have, but you want to check, if the derived formula is correct. For this check you need numerical derivatives. Example for this case is  calculation of the stiffness matrix for any model of plasticity in the framework of finite element method (FEM), which is given as a derivative $\mathbf{B}=\frac{\text{d}\mathbf{\sigma}}{\text{d}\mathbf{\varepsilon}}$, where $\mathbf{\sigma}$ is the second order stress tensor and $\mathbf{\varepsilon}$ is the second order strain tensor and $\mathbf{B}$ is fourth-order tensor.
It is worth to notice that in numerical differentiation approximation errors as well as rounding errors can occur, which can destroy the result totally. Special treatment is needed in this case (google "complex step differential formulae" or "Ficks's hyper-dual numbers").
The answer to your last question (Is fitting a statistical model a mathematically valid alternative to numerical differentiation?): I would say yes: if you evaluate the function in sufficient number of points, which are not too far (high approximation error) and also not too close (high rounding error), around the point, where you need to know the derivative, and make polynomial fitting, then you get the same value as by the help of differential formulae. Again you need to be sure about numerical error which you get.
