I am interested in computing the margin-of-error for a metric computed on a random sample. The underlying distribution (finite) from which the random sampling is done is not normal (its extremely skewed; something like a zipf/power-law curve).
Planning on estimating the margin-of-error at 95% confidence by Monte Carlo Simulation:
- Repeat large number of times:
- Sample $n$ data points (without replacement) out of $N$
- Compute metric on $n$ points and store it somewhere
- Use the large number of metrics to estimate margin of error
Question1 :: I am not sure about the sampling without replacement part (should it be with replacement)?
Question2 :: How to compute the margin-of-error? (I can get the histogram, but then what?)
Edit: The metric I am concerned with is "defect rate" in %tage. The reason for sampling is to lower the cost of evaluation of item.