1
$\begingroup$

I'm learning some basic statistics, and I have a very basic question about confidence intervals.

Here's a definition of a 95% confidence interval:

Suppose that we are estimating a proportion, and that in our sample of size $n$, we compute a statistic of $\hat p$. Call a parameter value $p$ plausible if, given the binomial distribution $B(n,p)$, the $p$-value of our observed statistic $\hat p$ is $\geq0.05$. Then the 95% confidence interval for our observed statistic is defined to be the interval consisting of the plausible parameter values.

Ok, great. Now in the textbook I'm using, I see the statement that if we were to conduct 100 trials, where each trial consists of taking sample, forming the statistic $\hat p$, and computing the 95% confidence interval, then in about 95 of those trials, the true value of the parameter would lie in our confidence interval. This is presented without justification.

My question is, is this somehow a tautological consequence of the definition? Or is there something to be proven here?

$\endgroup$
2
  • $\begingroup$ where did you get that definition of a 95% CI? $\endgroup$ Commented Sep 9, 2020 at 0:19
  • 1
    $\begingroup$ @spaceisdarkgreen it's from Tintle et al's "Introduction to Statistical Investigations", section 3.1. $\endgroup$ Commented Sep 9, 2020 at 1:56

1 Answer 1

1
$\begingroup$

Everything that can be proven in mathematics is a tautological consequence of some definitions. That's not how we decide what needs to be proven: we decide that something requires proof if it's not obvious. Clearly, this claim is not obvious to you. Therefore, it requires proof!

In general, the $p$-value of an outcome $x$, with respect to some hypothesis that specifies the probability of all outcomes, is defined to be the probability of getting an outcome as improbable as $x$, or even more improbable.

With respect to the true hypothesis (in your case, the true value of the parameter),

  • call an outcome "unlikely" if it has $p$-value less than $0.05$.
  • let $x_0$ be the "likeliest unlikely outcome": the unlikely outcome with the highest probability.
  • then, the $p$-value of $x_0$ (which is less than $0.05$) is the probability of getting any unlikely outcome, because all other unlikely outcomes are less likely than $x_0$.

In particular, we conclude that the probability of getting any unlikely outcome is less than $0.05$.

On the other hand, it should be pretty close to $0.05$. To see this, let $x_1$ be the outcome with the lowest probability that isn't unlikely. Then the $p$-value of $x_1$ (which should be at least $0.05$) is the probability of getting either $x_1$ itself, or some unlikely outcome. Also, usually, each individual outcome doesn't have too high a probability. (This is true in the case of binomial distributions: each individual outcome has a probability of at most $O(\frac1{\sqrt n})$). So the probability of getting some unlikely outcome should have been close to $0.05$, if including $x_1$ was enough to push it over the edge.

Now, flipping things around, we see that an outcome $x$ is "unlikely" (by my definition) if, when we observe $x$, the true parameter value is not "plausible" (by your textbook's definition). That's because the definitions of both are the same: the definitions of both are that "the $p$-value of $x$ is less than $0.05$ with respect to the true hypothesis".

So we get an outcome for which the true parameter value is outside the confidence interval exactly when we get an unlikely outcome, which happens with probability less than, but very close to, $0.05$.

$\endgroup$
1
  • $\begingroup$ Oook! This is very helpful. I like this answer - particularly that it doesn't depend on assuming that we're working with binomial distributions. And I think I understand it, but I'm going to sleep on it to make sure. $\endgroup$ Commented Sep 9, 2020 at 2:00

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .