3
$\begingroup$

When calculating confidence intervals for a population with standard deviation σ unknown, σ is estimated using the sample standard deviation S, which uses the Bessel correction to more closely approximate the real σ.

But suppose the population X is a Bernoulli variable. X being now a binary variable, $ \sum_{i=1}^n (x_i - \bar x)^2 = n\bar x (1 - \bar x) $ (as we can see in an answer to this question). So the formula of the sample standard deviation would be $ S = \sqrt {\frac {n} {n-1} \bar x (1 - \bar x)}$.

But in all resources I've read about confidence intervals for proportions, when the population proportion p is unknown, the standard deviation of the population is estimated by $ \sqrt{\bar p (1-\bar p)} $ . This approximation, however, does not use the Bessel correction. If it were, σ would be approximated by $ \sqrt{\frac {n} {n-1} \bar p (1-\bar p)} $.

I understand that $ \bar p (1-\bar p)$ is a consistent estimator for $p(1-p)$, but wouldn't $\frac {n} {n-1} \bar p (1-\bar p)$ be consistend and unbiased, and thus a better estimator?

$\endgroup$
4
  • $\begingroup$ Depending on the application, $n$ might be extremely large. If you are doing a statistical test and the correction becomes the deciding factor of your test's conclusion, you might be in trouble. $\endgroup$
    – Mark
    Aug 25, 2018 at 14:34
  • $\begingroup$ So the reason would be that the sample size should basically be large enough to render the correction insignificant? $\endgroup$
    – Catalin Florea
    Aug 25, 2018 at 14:44
  • $\begingroup$ I think that could be one possible explanation. Another interesting thing about Bernoulli distribution is that the standard deviation is completely determined by the mean. So if you claim that population mean is $\bar{p}$, you are not free to make a separate claim about the population standard deviation. $\endgroup$
    – Mark
    Aug 25, 2018 at 14:57
  • 1
    $\begingroup$ Bessel correction is for the sample standard deviation. Here we have that $\sigma=\sqrt{p(1-p)}$ is the standard deviation of the population. The problem is that we doesn't have the parameter $p$. But we can estimate $p$ with the unbiased estimator $\overline{x}$. We are using data for estimating $p$ and then $\sigma$. We are not using data to estimate directly $\sigma$, in that case we should use Bessel correction. $\endgroup$
    – maenju
    Apr 21, 2022 at 20:04

1 Answer 1

Reset to default
1
$\begingroup$

There may be two distinct underlying questions here:

  • for a Bernoulli random variable, $\bar{x}$ is an unbiased estimator of the population proportion $p$, while $\frac{n}{n-1}\bar{x}(1-\bar{x})$ would be an unbiased estimator of $p(1-p)$. Usually the proportion is the parameter of interest and the other properties of the distribution are based on that

  • there are a surprisingly large number of different ways of producing a confidence interval for the proportion: Wikipedia lists several, though there are more such as the Blyth-Still-Casella interval. The confidence interval method related to the Wald test is widely taught as something like $\bar{x}\pm 1.96 \sqrt{\frac{\bar{x}(1-\bar{x})}{n}}$ but is generally regarded as less satisfactory than the other methods; its results are not unreasonable when $n$ is large and $\bar{x}$ not close to $0$ or $1$, but those are not the interesting cases. In the more interesting cases, concerns such as the discreteness of the binomial distribution and the fact $p$ must be between $0$ and $1$ are more substantial issues than the bias of the variance estimate, and other methods address some of these concerns

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .