Derivative for numerical models.

Question

I am working in Mechanical engineering and Computer vision, in which I use a matlab code (or codes) to represent a specific system or process. Of course such model has an input , an implimented structure , and an output. In the way of studying the sensitivity analysis of such models, I think of using local sensitivity analysis tests, in which the partial derivative of the models is computed to represent the index of sensitivity of each parameter or variable. My question here is that, If my model cannot be represented by a function to evluate partial derivatives explicitly, is it possible to use directly the numerical methods to derive the partial derivatives? For example, we may run the model for some values $x_1, x_2, ...,x_j,..., x_n$ to give $y_j$, then for the values $ x_1, x_2, ...,x_j+\Delta_j,... , x_n$ to give $y_{j+\Delta}$, and then estimate the partial derivative wrt the variable $x_j$ by $\frac{y_{j+\Delta}-y_j}{\Delta}$. I think this way is straightforward, but how to be assure that our model has a differentiable form i.e. is differentiable? Or let me say, can we speak about difference in variation without including the notion of partial derivatives and differentiability of the model?

I respect any perspective and I highly appriciate your opinions.

Thanks .

Answer 1

The answer is yes, and no, and maybe. In sensitivity analysis, we can always run a model at different parameter values and use finite differencing methods to compute the directional derivatives in our parameter space. This is, as you might imagine, of limited use without any other external information. First, it requires many runs. Second, it only gives us scraps of information. For instance, computing the change with respect to some parameter $\theta$ and separately computing the change with respect to some other parameter $\gamma$ doesn't tells us anything about whether the parameters are somehow dependent, or what happens with convex combinations of parameters, etc. If we know something about the model, then we might be able to extrapolate these data. But in a vacuum, no.

Another approach which is popular in sensitivity analysis is to perform automatic differentiation (aka algorithmic differentiation). AD methods are different than finite difference methods in that they do not compute numerical estimates of a derivative, but rather they compute the derivative of the computer code (for instance, by overloading operators). In this sense, we get a derivative for free -- running the code gives us a simulation realization and it also gives us the derivatives at the same time. AD has its limitations and I cannot enumerate them all here; some reading would be necessary.

Finally, another tool in sensitivity analysis is locally linear embedding. If we assume that the partial derivatives exist within a parameter space, then we think back to calculus and recall that a derivative is a linear approximation to a function in a neighborhood of a point. Presumably, the system we're studying isn't so volatile that any neighborhood yields an inaccurate linear approximation. So we can run a handful of simulations and use the data to embed locally-linear approximations of the result; this is enough to construct a complete approximation of the sensitivity in a neighborhood of a point in parameter space. This approximation is not always valid, however.

In short, this question is ultimately too broad, as you can find many, many papers in the field of sensitivity analysis that address many of these issues in great detail.