Is it always okay to drop higher order terms out in Taylor approximation? In many books that one comes across with regarding statistics, function approximation, optimization etc. it is almost certain that you run into Taylor expansion, be it multidimensional or one-dimensional : 
$$f(x)=f(a)+\frac{f'(a)}{1!}(x-a)+\frac{f''(a)}{2!}(x-a)^2+\frac{f'''(a)}{3!}(x-
a)^3+\cdots,$$
and almost always one can see the authors drop out the higher order terms in the Taylor approximations, i.e. terms with derivatives $\geq 3$. 
I came to realize that I've never seen a well justified explanation on why the terms with derivatives $\geq 3$ are always dropped out. 
My question is: What is the justification for this? Why the magic number of $3$? Why not $4$? or $900$?
UPDATE: 
An example illustrating what partly motivated my question: 
Let us consider a situation where we approximate a probability distribution with the normal distribution using Fisher information. If I remember correctly, in normal approximation using Fisher information we also "drop out" higher orders than 3 out. Lets also say that we are dealing with an application where we need "a very high precision", higher than we can obtain with the "2nd order" normal approximation. In order to use the normal approximation, would we now need to derive a more higher order normal approximation?
 A: It depends on what you're doing.
In math nothing is droped we substitute what you call dropping with the big-O and small-o Notations.
In physics terms of higher orders than 3 are indeed usually droped. And that because of the fact that everything else is $o((x-a)^3)$ and we are usually dealing with an x close enough to a. that means that terms of order 4 and higher are really small and are considered to be negligeable. (imagine $x-a=0.1 $ then $(x-a)^3=0.001$)
For instance to obtain the familiar harmonic ossilator, terms of order three and higher are dropped in the Taylor expension of the Energy. Thath's why all small oscillations can be aproximated really well with the harmonic oscillator.
A: I think it's a combination of a lot of things. There is always simply the possibility that some terms cancel out, or that the higher order derivatives don't exist for the function you are trying to approximate. 
Many distributions only consider second order interactions between variables (quadratic), and thus derivatives higher than the second end up evaluating as 0 anyway. You're example of the Fisher information as the covariance of the asymptotic distribution of the MLE is a good example. Suppose you were looking at the asymptotic distribution of the mean in a multivariate normal with known precision: $X_i \overset{iid}{\sim} N(\mu,\Omega)$.
If you calculate the Fisher information matrix, you end up getting rid of the exponential through the log, and you're left with quadratic interactions as a function of $\Omega$:
$\mathcal{I}(\mu)_{rs} = \frac{\partial^2}{\partial_{\mu_r}\partial_{\mu_s}} \frac{1}{2}\sum_{r,s}\mu_r\mu_s\Omega_{rs}$.
So you can see that if you consider higher order derivatives you end up having terms which cancel out anyway. This isn't explicitly a Taylor series, but the idea that most models and distributions consider second order interactions gives intuition about why second order approximations come up so much. The fact that second order interactions are so easy to work with also motivates people stopping at a second order approximation, as it's usually sufficient and the problem can rapidly get more difficult to solve if one expands into a higher order approximation.
There's also situations in which clever choices of the argument and point about which you expand your Taylor series can make it such that higher order terms cancel out. You see this a lot with people expanding their function $f(x)$ about local maxima or minima $x^*$, and having the second term $(x f'(x^*))$ fall out. It's very possible that higher order derivatives do exist in some regions of the domain (unlike in the previous Fisher information example), but if you specify $x^*$ conveniently you don't  have to worry about it. Other ways this comes up are in limits; if you're interested in studying function $f(x)$ at $f(x+\epsilon)$ with an expansion around $x = 0$, then you end up with:
$f(\epsilon) = f(0) + \epsilon f'(0) + \sum_{n = 2}^\infty \frac{\epsilon^n f^{(n)}(0)}{n!}$
And you can see that if your examining the limiting behavior, with $\epsilon$ very small, then $\epsilon^n$ rapidly becomes trivial, and those higher order terms can be mostly dismissed.
As other people have also said, people definitely do consider the accuracy of the approximation as well, whether that be deterministically or probabilistically.  You can see that on the Wikipedia page for the Delta Method: https://en.wikipedia.org/wiki/Delta_method. They briefly go into the order of the approximation, and most advanced textbook do this in general when using Taylor series.
In general I think a lot of it has to do with what people are interested in the problem as well. There's obviously a lot of connections between Taylor series and moments, because it turns what can be a very complicated function into a polynomial where moments are easier to calculate. You can see this used for example in these two pages:
https://en.wikipedia.org/wiki/First-order_second-moment_method
https://en.wikipedia.org/wiki/Taylor_expansions_for_the_moments_of_functions_of_random_variables
So going back to most interesting investigations focusing on second order interactions, or asymptotic normality (where you only need to get the first two moments), it makes sense that people stop at the quadratic term.
