Statistical significance and sample size I have a device that is said to succeed at doing some task at least 99% of attempts, and fails no more than 1% of attempts.
If I want to be 98% sure that it achieves that success rate, how many results would I need to check at minimum?
And what would be the maximum number of failures allowed in that number of results?
 A: This is how I would approach the question (Even if I made a mistake, you'll get the idea): 
At first, we should choose the model. Lets do hypothesis testing for binomial distribution. Our device has binomial distribution with some constant probability of success $p$ and probability of failure $1-p$. We are checking hypothesis $H_0: p \geq 0.99$ vs $H_1: p < 0.99$. Significance level is $\alpha = 0.02$.
Suppose we have done $n$ trials and counted the number of successful events $x$. It should have distribution $X \sim Bin(n, 0.99)$. Using binomial cumulative distribution function (or approximations) we can compute $P(X\leq x)$ and compare this value with $\alpha$. If it is bigger, then the null hypothesis is accepted (no enough evidence).
So lets imagine now that $0.985$ success rate is not satisfactory for you. To obtain $n$ we need to solve such equation:
$$P(X<0.985n)=0.02$$
where $P$ is binomial CDF for $Bin(n,0.99)$. and "trials" were taken from distribution with $p=0.985$. We will approximate binomial CDF with normal distribution to make it continuous: $F_{Bin(n,p)} \sim N(np,\sqrt{np(1-p)})$. Direct computation in R gives:
bar<-function(x){pnorm(0.985*x,mean=x*0.99,sd=sqrt(0.99*0.01*x))}

uniroot(function(x){bar(x)-0.02},lower = 1, upper = 10000000)

$n = 1670$. If you agree to consider possibility of $0.98$ success rate then $n=417$.
Note that function is decreasing and only after 1670 attempts you can notice that $p=0.985$, not $0.99$: 

And this is all for $\alpha=0.02$ confidence level, so its hard to be "sure".
