# How safe is it to ignore low probability events?

See this question on applying probability theory principles in software design.

The question is generally the following: you design some system (say software) and rely on some well-known mathematical concept (say hash function). You know that when this concept is used without caution your system can sometimes fail, however the probability of such failure is extremely low.

You need to evaluate whether you want to alter the design or can just ignore that drawback. Consequences of a failure are usually taken into account when such evaluations are done. For example if a failure leads to a person being mildly offended then it is not that of a problem but if a failure leads to a nuclear power explosion that it is a serious problem.

Now the accepted answer goes like this (all numbers here are exaggerated for better perception): probability of Earth colliding with a space rock is 10E-50 and probability of that drawback causing a problem is 10E-100. You see - Earth colliding with a space rock is a gazillion times more likely. So relax, that design is good enough.

Is that reasoning correct? Can it be accepted at all times?

-
A crosspost to the stats site might be helpful, in case you don't get that much response here. –  Guess who it is. Oct 27 '10 at 9:15
@J.M.: Are cross-posts a good thing? Are there guidelines or best practices for that? –  sharptooth Oct 27 '10 at 9:21
It's fine I think, at least for this case. –  Guess who it is. Oct 27 '10 at 9:25
If the totalitarian principle is to be believed then everything not forbidden is compulsory –  crasic Oct 27 '10 at 10:30
Risk analysis may be appropriate here. –  You Oct 27 '10 at 11:10

Extremely small probabilities are always suspect. See The Titanic Effect. If you think the probability of a software failure is 10^-50 then you're not looking at the most probable source of failure. If you're hash algorithm, for example, has one chance in 10^50 of failing, then it's more likely that your program, operating system, or hardware has a bug.

In general it's not enough to look at probabilities of failure. You have to look at probabilities of failure and the consequences of the failures. In some contexts, a 10% probability of failure is acceptable. In other contexts, a one-in-a-million chance of failure is quite serious.

-

Maybe do it like the industry does (e.g. with hard drives) and estimate the mean time between failures. If the unsafe operation is used at most once each time interval, and the probability of a failure is $p$, you can expect a time between failures of $\frac{1}{p}$ time intervals. If this time is significantly longer than you expect your system to be used, you are of the safe side.

1e-100, is ridiculously low, by any standards. If such an error might occure once every nanosecond, you have a mean time between failures of 3e83 years. Compare that to the estimated age of the universe.... =)

-
Failure rates estimates are entirely unreliable. take for example, the Space Shuttle. Official stated MTTF is >10K launches. As both Feynman (who was on the Columbia disaster investigation committee) and reality showed, the real number is close to 50. –  crasic Oct 27 '10 at 10:24
@crasic: The law here is "Garbage in, garbage out". If the Space Shuttle team had no clue about failure probabilities, they should not have calculated MTTFs. =) I agree that estimating these probabilities may be hard. The OP linked a question dealing with SHA-256 collisions, where failures rates are much more easily accesible. –  Jens Oct 27 '10 at 10:34
my point is that in most cases, a MTTF estimate includes a huge number of assumptions about what you know about the system. In a mathematically "closed" problem like SHA-256 you can be confident in these assumptions. But in most cases, multiple failure pathways are overlooked. –  crasic Oct 27 '10 at 10:41

In addition to the probability of the failure and the possible consequences, you should also consider the cost of using an alternative method. This line of thinking is illustrated by the .jpg coding scheme. It has a high rate of failure to covey the image with perfect accuracy, the consequences are minor for most applications, but it's in common use due it's advantages in processing over more accurate schemes i.e. it's cheaper in time and space that more accurate methods.

-