3
$\begingroup$

I have a sample of size $n=19593$ of count data

  1      2     3    4    5   6   7   8   9   10  11  12  13  14  15  16  17  18  19 20 21 22 23 24 25 26 27  
18890  3032  1542  727  599 530 409 363 298 266 261 273 231 268 129 131 145 120 119 29 62 48 0  18 0  38 10 

I want to find a confidence interval for the center of the data, specifically in the form $n \mu$. Do I have enough data to estimate $\sigma$ and use the central limit theorem. The negative binomial seems to be the best fit among known distributions, but I don't think it is the right fit because of the long tail in the data, so would bootstrapping be a better approach and using the median instead of the mean to measure the center of the data.

$\endgroup$
5
  • $\begingroup$ $n=19593$ is a quite big number to be able to consider the CLT. $\endgroup$ Commented Jun 28, 2015 at 1:10
  • $\begingroup$ @SeyhmusGüngören I was taught that having $n \gt 30$ is the requirement. Is that not correct? $\endgroup$
    – Zach466920
    Commented Jun 28, 2015 at 1:12
  • 2
    $\begingroup$ it depends on the component distributions. BUT it is known that the convergence is very fast. Sometimes even $n=10$ is enough for a good approximation. In your case you have plenty of data. $\endgroup$ Commented Jun 28, 2015 at 1:20
  • $\begingroup$ @Seyhmus I'm not the Op :) just making sure I didn't have a knowledge gap, thanks. $\endgroup$
    – Zach466920
    Commented Jun 28, 2015 at 1:31
  • $\begingroup$ I think $n = 19593$ is incorrect. Just 18890 + 3032 + 1542 + 727 is already 24191. I show my count in my Answer. Hope I didn't misunderstand your data. Different data than in Question 1341287? $\endgroup$
    – BruceET
    Commented Jun 29, 2015 at 23:45

2 Answers 2

1
$\begingroup$

Confidence intervals for mean of a highly skewed population using a large sample.

This seems to be a followup from a previous post in which you started asking about CIs for the population mean in addition to your original question (about retrieving data from confidence intervals for the frequencies of various values). Here the values $y$ and their respective frequencies $f$ are provided.

Your explicit question in the current post is about a comparison of a traditional confidence interval for the population mean $\mu$ based on the CLT and a nonparametric bootstrap. As @Bey speculates, there is good agreement.

Traditional confidence interval. First, the traditional CI. We begin by inputting your data and checking for formatting errors.

 y = c(1, 2,  3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 
       16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27) 
 f = c(18890, 3032, 1542, 727, 599, 530, 409, 363, 298, 266, 261,  
       273, 231, 268, 129, 131, 145, 120, 119, 29, 62, 48, 0, 18,
       0, 38, 10) 
 n = sum(f);  x = rep(y, times=f);  n
 ## 28538  # n
 obs.mean = sum(y * f)/n;  obs.mean;  mean(x) #check
 ## 2.740872  # x.bar
 ## 2.740872  # agreement: data correctly parsed
 obs.sd = sd(x);  pm = c(-1,1);  z.star = qnorm(.975)
 obs.mean + pm*z.star*obs.sd/sqrt(n)  # trad'l CI
 ## 2.696125 2.785619

So the standard CI of the population mean $\mu,$ assuming normality is $(2.70, 2.78).$ The distribution is indeed very seriously skewed, but the sample size is huge and the traditional interval should be OK.

Nonparametric bootstrap CI. Now to compare with results of a nonparametric bootstrap. I am bootstrapping the distribution of the variability $V = \bar X - \mu,$ of the sample mean about the population mean for (very large) samples of size $n = 28,538.$

If we knew the distribution of $V,$ we could find $v_L$ and $v_U$ cutting 2.5% of the probability from the respective tails of its distribution so that $$P(v_L \le V = \bar X - \mu \le v_U) = P(\bar X - v_U \le \mu \le \bar X - v_L) = 0.95,$$ and $(\bar X - v_U, \,\bar X - v_L)$ is a 95% CI for $\mu.$

The resulting CI turns out to be $(2.69, 2.79),$ essentially identical to the result based on the CLT. The bootstrap interval is a little longer because here we are making no assumptions about the distribution of the population (except that it has a mean). So it does not matter whether the data are Poisson, negative binomial, or from some other severely skewed distribution. (The code below assumes that the code above has been run just previously. In typical bootstrap notation, $*$'s are used for resampled quantities.)

 # Bootstrap World--Estimated Distribution of V
 B = 1000;  re.x = sample(x, B*n, repl=T)
 RDTA = matrix(re.x, nrow=B)          # B x n matrix of resamples
 re.mean = rowMeans(RDTA)             # vector of B `x-bar-star's
 re.v = re.mean - obs.mean            # vector of v-star's
 v.UL = quantile(re.v, c(.975,.025))  # estimated quantiles of V

 # Real World--Bootstrap Confidence Interval
 obs.mean - v.UL
 ##    97.5%     2.5% 
 ## 2.692651 2.785346 

Notice that no estimate of the variance is used in the bootstrap procedure. The variability of $V$ is 'deduced' from repeated sampling.

$\endgroup$
-1
$\begingroup$

You are right - your distribution is definitely not normal.

There seems to be a misconception in some of the answers - large $N$ does not imply the distribution is close to normal! The empirical distribution will converge to the actual distribution for a large $N$ - and not normal if the actual distribution isn't. (Only means converge to normal according to CLT.)


For your distribution, however, you can find confidence intervals by taking percentiles - for example the range between 2.5th percentile and 97.5th percentile is a 95% confidence interval.

The picture below illustrates this on your distribution.

enter image description here

Or in general, take $1-\alpha$ confidence interval by choosing $\hat F_{\alpha/2}$.

$\endgroup$
2
  • $\begingroup$ Nice graph of the data. However, there is a difference between saying a sample comes from a normal distribution and saying that the sum or average of observations is nearly normal. That difference is the essence of the Central Limit Theorem. Also, finding the 'middle' 95% of a sample (as in this Answer) is different from finding a 95% CI for the mean of the parent population. $\endgroup$
    – BruceET
    Commented Jul 2, 2015 at 22:52
  • $\begingroup$ Yes. I am claiming neither - in this case the data doesn't come from a normal distribution, nor does it appear to have any reason for CLT to be applied. Yes middle 95% is not best the CI - you are right. A true 95% CI for discrete distributions will be top 95% occurring values, which in this case will be 0th to 95th percentile (since probabilities are in decreasing order). $\endgroup$
    – KalEl
    Commented Jul 3, 2015 at 17:08

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .