# Large sample confidence interval

Can someone please be so kind to check if my answer is correct

• These formulas relying on approximate normality do not apply when you get a proportion so close to $0$ or $1$ that only two members of the sample are in one of the two categories. – Michael Hardy Jul 27 '17 at 4:30
• $\ldots\,$and getting a negative number as the lower bound should certainly make you suspicious. – Michael Hardy Jul 27 '17 at 4:31
• @MichaelHardy So the exact Clopper Pearson is more appropriate in this case? Yeah because of the negative value that led me to posting this question – Biu Jul 27 '17 at 4:32
• Yes. $\qquad\qquad$ – Michael Hardy Jul 27 '17 at 5:22
• @MichaelHardy Thank you – Biu Jul 27 '17 at 7:00

See more details in this answer for the definition of a Clopper Pearson confidence interval, and problems with approximations in confidence intervals (the approach you have used). I've considered the same principles here.

Clopper Pearson confidence interval: with $s=2$ (defects), $n=500$ (boards), $\alpha=0.05$ and the Beta quantile function from R, that is qbeta:

$$[\text{qbeta}(\alpha/2,s,n-s+1), \text{qbeta}(1-\alpha/2,s+1,n-s)]=[0.00049,0.01435].$$

Therefore, the 95% Clopper Pearson confidence interval for the true frequency is $[0.00049,0.01435]$ in this case.

The "exact" Clopper-Pearson CI may not be so good for only 2 successes in 500 trials. The version of the Clopper-Pearson CI given in @bluemaster's answer (+1) uses a continuous beta approximation to the discrete binomial. (For some details, you can google 'Clopper-Pearson' and look at the Wikipedia article, among others.)

A Bayesian-based approach, also using beta distributions, might be be more accurate, but the answer does not differ by much for your problem.

Suppose we use the 'flat' prior distribution $\mathsf{Unif}(0,1) \equiv \mathsf{Beta}(1,1),$ which has density function $f(p) = p^{1-1}(1-p)^{1-1} = 1,$ for $0 < p < 1.$

The likelihood function corresponding to your data is $$f(x\mid p) = {500 \choose 2}p^2(1-p)^{488} \propto p^2(1-p)^{488},$$ where the symbol $\propto$ (read "proportional to") indicates that the expression on the right omits the constant of the function and shows only the 'kernel'.

The posterior distribution is $$f(p\mid x) = f(p)\times f(x\mid p) \propto p^{1-1}(1-p)^{1-1} \times p^2(1-p)^{488} \\ \propto p^2(1-p)^{488} = p^{3-1}(1-p)^{499-1},$$ where we recognize the final expression as the kernel of $\mathsf{Beta}(3, 499).$ Then a 95% Bayesian probability interval for $p$ is $(0.0012, 0.0143),$ which can be found by cutting 2.5% from each tail of the posterior distribution. The computation using R statistical software is shown below.

qbeta(c(.025, .975), 3, 499)
## 0.001236581 0.014345537


Notes: (a) Bayesian statisticians interpret 'probability intervals' differently from frequentist 'confidence intervals', but frequentists often use Bayesian interval estimates as confidence intervals. (b) The prior distribution can influence the Bayesian interval estimate, but the 'flat' prior used here provides minimal information. If prior information is available, it should be reflected in the prior distribution (c) When there are almost no successes or almost no tails, experience and simulation experiments have shown that a flat-prior Bayesian interval often comes closer to providing the promised 95% coverage than do traditional intervals based on a normal approximation.

• BruceET, when I teach confidence intervals I usually show this approach you've presented, considering a uniform prior (Beta(1,1)). For reasons I don´t know, in some fields the Clopper Pearson is a popular method. Another similar approach is the Jeffreys method, which considers a Beta(1/2,1/2) prior. My feeling is that all will produce very similar results for practical purposes, and will be better approaches to the traditional C.I. based on a Gaussian approximation. BTW, some Bayesians use "credible interval" as the name for "confidence interval". – bluemaster Jul 27 '17 at 12:01
• Jeffreys priors are unintuitive for students new to Bayes. In coverage graphs, such as in one of my earlier Answers, uniform prior seems best for use as frequentist CI. – BruceET Jul 27 '17 at 16:08
• We were taught to use the qbeta R function to compute Clopper-Pearson CI – Biu Jul 28 '17 at 5:35
• No argument with that. I do wonder about the use of the t distribution in image in your Question. The traditional CIs (Wald, Wilson, and Agresti) all use z (e.g., 1.96 for a 95% CI). For $n$ as large as 500 there is almost no difference. I have seen $t_{\alpha/2, \nu}$ advocated (mainly in psych and sociology texts), but never satisfactorily justified. // Clopper-Pearson tries to guarantee the promised coverage probability (e.g. 95%), sometimes going higher, but not by too much. It is difficult to get really 'exact' coverage because binomial is discrete while normal and beta are continuous. – BruceET Jul 28 '17 at 6:40

The problem with the Clopper-Pearson interval is that, being an exact interval, it guarantees a coverage probability of at least $100(1-\alpha)\%$, but due to the discrete nature of the binomial distribution, the actual coverage probability can be substantially higher, especially when $p$ or $1-p$ is close to $0$, or $n$ is small. The resulting interval is in some sense "too wide" or "too conservative," because not all of the $\alpha$ can be spent.

For large samples, the Wald interval (i.e. arising from a normal approximation to the binomial) is good when $p$ and $1-p$ is not close to $0$. But here, this is not the case: the number of observed events in the sample is very small.

An alternative to a Bayesian approach that remains in the frequentist regime is the Wilson score interval: $$\frac{1}{1+z_{\alpha/2}^2/n} \left( \hat p + \frac{z_{\alpha/2}^2}{2n} \pm z_{\alpha/2} \sqrt{\frac{\hat p (1 - \hat p)}{n} + \frac{z_{\alpha/2}^2}{4n^2}} \right),$$ where $\hat p = x/n$ is the sample proportion, $n$ is the sample size, and $z_{\alpha/2}$ is the upper $\alpha/2$ quantile of the standard normal distribution (so for example, when $\alpha = 0.05$, $z_{.025} \approx 1.96$). This interval has good coverage properties even for small sample sizes or extreme sample proportions. I leave it to you as an exercise to compute this interval for your question.

• @BruceET Exact intervals guarantee the nominal coverage probability, but the converse is not true: not all intervals that guarantee nominal coverage probability are exact. – heropup Jul 27 '17 at 18:38
• By that definition of "exact," I guess the CI $[0,1].$ is an exact 95% CI for $p.$ // Wilson is OK if you believe $\hat p(1-\hat p)/n$ is the actual variance instead of an estimate of it. Pretty good for large $n.$ // For 95% CI Wilson and Agresti CIs are almost indistinguishable, Agresti conflates 1.96 with 2. (Originally this Comment preceded the one above.) Definition of "exact" is misleadingly specialized. – BruceET Jul 27 '17 at 18:39