2
$\begingroup$

In many books that one comes across with regarding statistics, function approximation, optimization etc. it is almost certain that you run into Taylor expansion, be it multidimensional or one-dimensional :

$$f(x)=f(a)+\frac{f'(a)}{1!}(x-a)+\frac{f''(a)}{2!}(x-a)^2+\frac{f'''(a)}{3!}(x- a)^3+\cdots,$$

and almost always one can see the authors drop out the higher order terms in the Taylor approximations, i.e. terms with derivatives $\geq 3$.

I came to realize that I've never seen a well justified explanation on why the terms with derivatives $\geq 3$ are always dropped out.

My question is: What is the justification for this? Why the magic number of $3$? Why not $4$? or $900$?

UPDATE:

An example illustrating what partly motivated my question:

Let us consider a situation where we approximate a probability distribution with the normal distribution using Fisher information. If I remember correctly, in normal approximation using Fisher information we also "drop out" higher orders than 3 out. Lets also say that we are dealing with an application where we need "a very high precision", higher than we can obtain with the "2nd order" normal approximation. In order to use the normal approximation, would we now need to derive a more higher order normal approximation?

$\endgroup$
  • 1
    $\begingroup$ With the first two derivatives, one can see if the function is increasing or decreasing and also if it is convex or concave near the point $a$. These properties often suffice to prove theorems. Higher order derivatives are more difficult to see on the graph of a function. $\endgroup$ – Gribouillis Nov 2 '17 at 8:47
1
$\begingroup$

I think it's a combination of a lot of things. There is always simply the possibility that some terms cancel out, or that the higher order derivatives don't exist for the function you are trying to approximate.

Many distributions only consider second order interactions between variables (quadratic), and thus derivatives higher than the second end up evaluating as 0 anyway. You're example of the Fisher information as the covariance of the asymptotic distribution of the MLE is a good example. Suppose you were looking at the asymptotic distribution of the mean in a multivariate normal with known precision: $X_i \overset{iid}{\sim} N(\mu,\Omega)$.

If you calculate the Fisher information matrix, you end up getting rid of the exponential through the log, and you're left with quadratic interactions as a function of $\Omega$:

$\mathcal{I}(\mu)_{rs} = \frac{\partial^2}{\partial_{\mu_r}\partial_{\mu_s}} \frac{1}{2}\sum_{r,s}\mu_r\mu_s\Omega_{rs}$.

So you can see that if you consider higher order derivatives you end up having terms which cancel out anyway. This isn't explicitly a Taylor series, but the idea that most models and distributions consider second order interactions gives intuition about why second order approximations come up so much. The fact that second order interactions are so easy to work with also motivates people stopping at a second order approximation, as it's usually sufficient and the problem can rapidly get more difficult to solve if one expands into a higher order approximation.

There's also situations in which clever choices of the argument and point about which you expand your Taylor series can make it such that higher order terms cancel out. You see this a lot with people expanding their function $f(x)$ about local maxima or minima $x^*$, and having the second term $(x f'(x^*))$ fall out. It's very possible that higher order derivatives do exist in some regions of the domain (unlike in the previous Fisher information example), but if you specify $x^*$ conveniently you don't have to worry about it. Other ways this comes up are in limits; if you're interested in studying function $f(x)$ at $f(x+\epsilon)$ with an expansion around $x = 0$, then you end up with:

$f(\epsilon) = f(0) + \epsilon f'(0) + \sum_{n = 2}^\infty \frac{\epsilon^n f^{(n)}(0)}{n!}$

And you can see that if your examining the limiting behavior, with $\epsilon$ very small, then $\epsilon^n$ rapidly becomes trivial, and those higher order terms can be mostly dismissed.

As other people have also said, people definitely do consider the accuracy of the approximation as well, whether that be deterministically or probabilistically. You can see that on the Wikipedia page for the Delta Method: https://en.wikipedia.org/wiki/Delta_method. They briefly go into the order of the approximation, and most advanced textbook do this in general when using Taylor series.

In general I think a lot of it has to do with what people are interested in the problem as well. There's obviously a lot of connections between Taylor series and moments, because it turns what can be a very complicated function into a polynomial where moments are easier to calculate. You can see this used for example in these two pages:

https://en.wikipedia.org/wiki/First-order_second-moment_method

https://en.wikipedia.org/wiki/Taylor_expansions_for_the_moments_of_functions_of_random_variables

So going back to most interesting investigations focusing on second order interactions, or asymptotic normality (where you only need to get the first two moments), it makes sense that people stop at the quadratic term.

$\endgroup$
  • $\begingroup$ Appreciate it =) $\endgroup$ – jjepsuomi Nov 6 '17 at 18:07
1
$\begingroup$

It depends on what you're doing. In math nothing is droped we substitute what you call dropping with the big-O and small-o Notations.

In physics terms of higher orders than 3 are indeed usually droped. And that because of the fact that everything else is $o((x-a)^3)$ and we are usually dealing with an x close enough to a. that means that terms of order 4 and higher are really small and are considered to be negligeable. (imagine $x-a=0.1 $ then $(x-a)^3=0.001$)

For instance to obtain the familiar harmonic ossilator, terms of order three and higher are dropped in the Taylor expension of the Energy. Thath's why all small oscillations can be aproximated really well with the harmonic oscillator.

$\endgroup$
  • $\begingroup$ Thank you for your help. Let me ask you the following next: Let us consider a situation where we approximate a probability distribution with the normal distribution using Fisher information. If I remember correctly, in normal approximation using Fisher information we also "drop out" higher orders than $3$ out. Lets also say that we are dealing with an application where we need "a very high precision", higher than we can obtain with the "2nd order" normal approximation. In order to use the normal approximation, would we now need to derive a more higher order normal approximation? $\endgroup$ – jjepsuomi Nov 2 '17 at 9:05
  • $\begingroup$ My follow-up question is a bit vague but I hope you get what I mean =) $\endgroup$ – jjepsuomi Nov 2 '17 at 9:06
  • $\begingroup$ Sorry I can't help you I know nothing about the Fisher information but I can tell you any approximation has it limits and conditions where it is aplicable. You should always check to see if you have satisfied said conditions. $\endgroup$ – J. Sadek Nov 2 '17 at 9:15
  • $\begingroup$ Don't worry about it. Thank you anyway, appreciate it :) $\endgroup$ – jjepsuomi Nov 2 '17 at 9:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.