What is the minimum number of tests to achieve statistical significance? I'm dealing with a situation where in a large manufacturing facility we have approximately $2000$ plumbing fittings of the same make and model. $3$ of those fitting have failed within the last year. Each causing major property damage. We suspect that the plumbing components suffered degradation and we want to test a sample of the remaining plumbing components to see how wide spread the issue is (if at all).
What is the best way to decide how big of a sample we should choose so that we do not under test or over test the installed components.
Any idea where to begin?
 A: Take $\bar{p} = \frac{3}{2000} = 0.0015, \bar{q} = 1 - \bar{p} = 1 - 0.015 = 0.9985$. If you want the confidence level to be $95$% which is typical for hypothesis testing with the margin of error within say $5$% of the population proportion of failed fittings of all the fittings, then $z_c = 1.96$. Thus the sample size $n$ you need is $n = \dfrac{z_c^2\cdot \bar{p}\cdot \bar{q}}{E^2}= \dfrac{1.96^2\cdot 0.0015\cdot 0.9985}{0.05^2}= 2.3\implies n = 3$. Thus you should take a minimum of $3$ fittings for a test run. The number might be ridiculously small, but hey it works !
A: You did not say what you are going to measure when you look at the $n$ fittings you will sample. Also, what is the nominal (safe) value for a fitting and how much of a difference $\Delta$ below nominal do you want to detect? What is the standard deviation $\sigma$ of the quantity you want to measure? And how sure do you want to be that you will detect that difference (power of the test)?
For a test at level $\alpha = 0.05 = 5\%,$ you will need to have reasonable estimates of $\Delta$ and $\sigma$ in order to determine the necessary $n$ to ensure a given power.
For example, suppose data are approximately normal so that you will use a one-sided, one-sample t test on your $n$ measurements. Then if $\Delta = 5, \sigma = 10,$ according to the following simulation in R, $n = 35$ observations will give you the approximate power 90%.
n = 35;  del = 5;  sgm = 10
set.seed(2021) # for reproducibility
pv = replicate(10^5, t.test(rnorm(n, 100-del, sgm), 
       mu = 100, alt="less")$p.val)
mean(pv <= 0.05)  
[1] 0.89515

By contrast, if it good enough to detect a difference  $\Delta$ as large as one standard deviation $\sigma,$ then
$n = 10$ observations would suffice (provided data are very nearly normal).
n = 10; del = 10; sgm = 10
set.seed(2021) # for reproducibility
pv = replicate(10^5, t.test(rnorm(n, 100-del, sgm), 
       mu = 100, alt="less")$p.val)
mean(pv <= 0.05)  
[1] 0.89694

All else being equal, you will get the same $n$ for the same ratio $\Delta/\sigma.$ There are 'power and sample size' procedures in various kinds of statistical software and online (e.g. here).
If you specify $\Delta, \sigma,$ and desired
power, they will give the required sample size $n.$ [If you want details of the distribution theory, mathematical statistics books and advanced
books on applied statistics have formulas that use the non-central t distribution.]
Note on simulation: If the P-value of a test is below $0.05= 5\%,$ then the null hypothesis of the t test will be rejected and you will detect a decrease in the desired measurement.
