Software testing - probability To test some automated diagnostic software I generated 8 random sets of input parameters which should result in a given outcome but one of my tests returns an unexpected answer.  I check this again, with the same data, and again get an unexpected result.  The developer explains his software makes a call to an external system which sometimes times out and returns no results.  He runs the program with the same input and this time gets the correct answer, thus he claims, proving that the problem lies with the external system.  (He’s a software developer so his definition of proof differs to mine.) 
My instinct is there are rather long odds on the observed events being due to the non-deterministic behaviour of the downstream application.  I calculate the odds of the testing resulting in this outcome as follows: 
The probability of failure for each individual test is 
$$\mathbf{P} = \left( \frac{2}{10} \right) = \left( \frac{1}{5} \right).  $$
The probability of success for each individual test is 
$$ \mathbf{P} = \left( \frac{4}{5}  \right). $$
The probability of the outcome of two specific failures and eight specific successful runs is the product of the individual probabilities - 
$$ \mathbf{P} =  \left(\frac{1}{5}\right)^2 \times \left(\frac{4}{5}\right)^8 
= \left( \frac{4^8}{5^{10}} \right)
= \left( \frac{1}{149}  \right)
~= 0.67\% $$
So two questions
1)  Is my probability maths sound?  (... putting aside that I might have been testing at a busy time with greater likelihood of the downstream system failing and there are only 10 samples.) 
2)  Is my software developer being straight?  
 A: 
1) Is my probability maths sound? (... putting aside that I might have been testing at a busy time with greater likelihood of the downstream system failing and there are only 10 samples.)

I think it's a bit more complicated than just multiplying P(success)$^8 \cdot$ P(failure)$^2$.
This sounds like something that requires the binomial distribution.
Why the binomial distribution?  Because it sounds like your scenario meets the 4 criteria listed here:


*

*The number of observations n is fixed.  ($n=10$ here)

*Each observation is independent. (You didn't specify but I don't think this is an unreasonable assumption.)

*Each observation represents one of two outcomes ("success" or "failure").  (This is right on the nose.)

*The probability of "success" p is the same for each outcome.  ($p = 4/5$ here)


So, using the formula from the first link, the probability of $8$ successes and $2$ failures is 
$$ {n \choose k} p^k (1-p)^{n-k},$$
where $n = 10 = $ the number of trials, $k = 8 = $ the number of successes we want, and $p = 4/5 = $ the probability of success.  Recall also that $\displaystyle {n \choose k} = \dfrac{n!}{k!(n-k)!}$.  So we have
$$
{n \choose k} p^k (1-p)^{n-k} = \frac{10!}{8! \cdot 2!} \left(\frac45\right)^8\left(\frac15\right)^2 \approx 0.30199 \approx 30.2\%
$$
So there's roughly a $30\%$ chance of getting $8$ successes out of $10$ trials.

2) Is my software developer being straight? 

Not entirely sure what you mean by this but I'm guessing you're asking if his claim that "the problem lies with the external system" is accurate.  I don't know, because all we did above is calculate the probability of $8$ successes out of $10$ trials.  Nothing we did above lets us know if the failures occur because of the external system or because of his code.  We'd have to know more about the relevant code I guess.  If I absolutely had to choose one or the other, I'd say the problem is with the external system if his code is not non-deterministic (so only the external system is non-deterministic) and if the probability of $8$ successes out of $10$ trials for the same input is only about $30\%$.  But again, I do think more info may be needed.
