# What is a confidence interval?

What are the nature and purpose of confidence intervals?

-
I will post my own answer below. I expect probably others will as well. – Michael Hardy Sep 8 '12 at 20:49
Is this question designed to address abstract duplicates? – Alex Becker Sep 8 '12 at 20:51
I know of no duplicates, and I don't know what "abstract" duplicates are. – Michael Hardy Sep 8 '12 at 21:06
See this meta question (or many others) for a discussion of abstract duplicates. – Alex Becker Sep 8 '12 at 21:09
OK, now I can answer: No, it's not meant to address such a thing. – Michael Hardy Sep 8 '12 at 21:15

Consider the normal distribution $N(\mu,\sigma^2)$. Its probability density function is $x\mapsto\text{constant}\cdot\exp\left(\frac{-1}{2}\cdot\left(\frac{x-\mu}{\sigma}\right)^2\right)$. It has expected value $\mu$ and standard deviation $\sigma$. It puts probability about $0.95$ in the interval whose endpoints are $\mu\pm1.96\sigma$.

Suppose one cannot observe $\mu$ and must estimate based on a sample of $n_1$ independently chosen observations from a population with this distribution. One can show that the sample mean $\bar{X}=(X_1+\cdots+X_{n_1})/n_1$ has a normal distribution with mean $\mu$ and standard deviation $\sigma/\sqrt{n_1}$.

For the moment we assume, unrealistically, that $\sigma$ is known.

We have $$\Pr\left(\mu-1.96\frac{\sigma}{\sqrt{n_1}} < \bar X < \mu+1.96\frac{\sigma}{\sqrt{n_1}}\right) = 0.95$$ and hence $$\Pr\left( \bar X - 1.96\frac{\sigma}{\sqrt{n_1}} <\mu< \bar X + 1.96\frac{\sigma}{\sqrt{n_1}}\right) = 0.95.$$ The interval whose endpoints are $$\bar X \pm 1.96\frac{\sigma}{\sqrt{n_1}}\tag{1}$$ is a $95\%$ confidence interval for $\mu$.

It is tempting to say that if we observe that the two numbers $\bar X\pm 1.96\dfrac{\sigma}{\sqrt{n_1}}$ are, for example, $5$ and $9$, then $$\Pr\left(5<\mu<9\right)=0.95.\tag{DANGER!!}$$ But suppose we take a second sample, this time of size $n_2$. We can form another confidence interval, using the new sample mean in the role of $\bar X$ and $n_2$ in the role of $n_1$ in $(1)$ above. What changes when we take a new sample is the endpoints of the interval. What does not change is $\mu$. The $0.95$ probability means that $95\%$ of the time, when we take another such sample, $\mu$ will be within the interval that we get. Some people take this fact to be an objection to the statement labeled "DANGER!!" above, saying that "$\Pr\left(5<\mu<9\right)=0.95$" should be considered true only if it is the case that $95\%$ of all values of $\mu$ are between $5$ and $9$, and that is clearly false, since there is only one value of $\mu$. This, however, depends on the meaning of probability. Even if one so defines probability that this objection is not valid, nonetheless the mathematics of probability do not lead to the conclusion that the statement labeled "DANGER!!" is true. In practice, though, people often act as if one should be $95\%$ sure of the statement $5<\mu<9$, given the evidence of the sample.

To say that one is "$95%$ confident" that $\mu$ lies within the confidence interval, as a term of art in statistics, means precisely that $95\%$ of the time, when one takes a new random sample of one or more observations, the interval one gets will contain $\mu$.

It was of course unrealistic to assume we know the population S.D. $\sigma$ but we have to estimate $\mu$ based on a sample. Suppose we estimate $\sigma$ based on the sample by using the square root of $$S^2 = \frac{1}{n-1}\sum_{i=1}^n \left(X_i-\bar X\right)^2$$ as the estimate of $\sigma$. Earlier we said that $$\frac{\bar X-\mu}{\sigma/\sqrt{n}}$$ has a normal distribution with mean $0$ and variance $1$. The quantity $$\frac{\bar X-\mu}{S/\sqrt{n}}\tag{2}$$ also has a distribution that does not depend on $\mu$ or $\sigma$. This is Student's t-distribution, introduced by the pseudonymous writer "Student", who was actually William Sealey Gossett. Thus we can find $A$ such that $(2)$ is in the interval bounded by $\pm A$ with probability $0.95$. It is a bigger number than $1.96$; formerly one looked it up in a table; now one uses software (it depends not only on the "$0.95$" but also on the sample size $n$). We get $$\bar X \pm A\frac{S}{\sqrt{n}}$$ as endpoints of a confidence interval.

Similarly one can find a confidence interval for $\sigma^2$ by observing that the distribution of $$\frac{(n-1)S^2}{\sigma^2}$$ does not depend on the unobservables $\mu$ and $\sigma$; this is a chi-square distribution. From $$A<\frac{(n-1)S^2}{\sigma^2}<B$$ we get $$\frac{B}{(n-1)S^2} < \sigma^2 < \frac{A}{(n-1)S^2}$$ and that is a confidence interval.

Many other examples exist.

-
Since Michael Chernick mentions that some weird things can happen, I say a little bit more. Suppose you actually know the value of $\mu$. Then given any $95\%$ confidence interval, the (epistemic) probability that $\mu$ is in the interval contains $\mu$, given what you know, would be either $0$ or $1$. But it would still qualify as a $95\%$ confidence interval by the definition above. Now suppose that you don't know the exact value of $\mu$, but some values of $\mu$, given what you know, are more probable than others---say you have a probability distribution concentrated near.... – Michael Hardy Sep 9 '12 at 2:36
....one particular point. Then if you got a $95\%$ confidence interval far from that point, the epistemic probability, given what you knew before, plus the data on which the interval is based, that $\mu$ is in that interval, would be much smaller than $0.95$, although it would be bigger than what it was before the data showed up. In that case, you should use the conditional probability distribution of $\mu$ given the data, found via Bayes' formula. Next, we observe that some additional information about the value of $\mu$ can come from the data itself, rather than from the...... – Michael Hardy Sep 9 '12 at 2:39
.....confidence interval. For example, suppose you have a random sample from a population uniformly distributed on the interval from $0$ to $\theta>0$. Then one can readily find values of $A$ and $B$ such that the interval from $A$ times the sample mean to $B$ times the sample mean is a $95\%$ confidence interval, by the definition above. But what if that interval fails to include the largest observed value, which is to the right of the interval? In that case, obviously the data themselves tell you that $\theta$ is not in the interval! But it still satisfies the definition. Next..... – Michael Hardy Sep 9 '12 at 2:43
Next, consider a population uniformly distributed on the interval from $\theta-1/2$ to $\theta+1/2$, and one has a sample of size $2$. Then the interval bounded by the two observations is a $50\%$ confidence interval. But: to be "$50\%$ confident" that $\theta$ is in that interval, in any common-sense sense of the word "confident", if the two observations are seen to differ by $0.001$, is absurd. And to be only $50\%$ confident, by any commen-sense sense of the term, if the observations differ by $0.999$, is equally absurd. Yet the interval satisfies the definition. – Michael Hardy Sep 9 '12 at 2:49
In this last example, Fisher's technique of conditioning on an ancillary statistic, where the latter is just the distance between the two observations, gives a reasonable answer. The ancillary statistic seems to take into account information that would be disregarded by reposing $50\%$ confidence in the $50\%$ confidence interval. But "information" in this instance doesn't mean what is usually called Fisher information, nor information in the sense of Fisher's concept of sufficiency. Just what it does mean would bear examination. I don't have an answer to that one. – Michael Hardy Sep 9 '12 at 2:51

Michael Hardy: I don't think anyone could give a better description of what the definition of a confidence interval is, so I will not try to give one of my own. But what I can add to the conversation is the fact that they are not unique and they are not always exact. There are exact confidence intervals and there are asymptotic ones.

Take the success parameter for a Bernoulli trial. If we have n iid Bernoulli random variables we can construct an exact confidence interval using the Clopper-Pearson method for evaluating cumulative binomial probabilities. There are also normal approximations to the binomial and so approximate confidence intervals can be constructed using normal approximations. When the sample size is large these approximate interval will have close to the advertised coverage. Coverage is the actual probability that if you repeat the procedure the new interval will contain the parameter (putting it in Michael Hardy's terms).

Since there can be more than one confidence interval for a parameter based on a random sample, how do we determine which one to use. Efron calls confidence intervals accurate if the actual coverage is equal to or close to the advertised coverage. Accuracy is a property we want every confidence interval to have if we would want to use it. The exact and asymptotic binomial confidence intervals are accurate. A confidence interval would be correct (term due to Efron) if among all accurate intervals its expected length is the smallest.

-