I have a question with a statistical nature; I think there should be some standard theory about this issue.

Suppose I have a large data set of size $N$ items, which has an amount of $K<N$ unwanted items. I am interested in finding the value of $K$. Testing all items takes too much time, so I want to determine a suitable sample size $n<N$ of randomly selected items in the data set.

Suppose I just pick a value for $n$

Then, of a randomly sample data of size $n$, I search for the unwanted items of which there are some amount of $k\leq n$. Let this amount be a test statistic $T$, i.e. I will test on the probability $P(T \geq k)$. I can now find a smallest integer value $K_\min$ such that for the estimation $K = K_\min$ we have $P(T \geq k) \geq \alpha$. That is, for any smaller integer estimation $K<K_\min$ we have $P(T \geq k) < \alpha$. If I am correct, I can now state that with a significance level $\alpha$ we have that $K \geq K_\min$. Is that true?

If this is true, the question now is: How accurate is this lower bound?

This is also my main question. Based on the amount $n$ and accuracy level $\alpha$, what can we say about the accuracy of $K_\min$. In other words, can we determine some confidence interval on $K$ in relationship to $K_\min$ and $\alpha$?

Any tips or other approaches are very much appreciated!

Best, Koen

Edit 26 November:

Another formulation of the problem as mentioned by David K is as follows:

Given some "error" tolerance $\varepsilon$, how do we choose $n$ for a given $\alpha$ such that we can guarantee that $|K_\min−K|/N\leq \varepsilon$ (or some assurance like that)?

  • $\begingroup$ At this time I suggest you do not accept my answer. (You can undo the acceptance; it will cost a few points but I'm pretty sure you can get them back by accepting an answer later.) I'm hoping that if this question stays "unanswered" for another couple of days we might see a better answer than mine; I just don't have the time at the moment to find one myself. $\endgroup$ – David K Nov 1 '17 at 14:36
  • $\begingroup$ Alright, just did so; I don't really bother too much about te points. I thought I was doing you a favor since I thought that no better answer would come by now, but let's wait and see. $\endgroup$ – Koen Nov 1 '17 at 14:50
  • $\begingroup$ You certainly weren't hurting me, so that's OK; I just want to see if we can get as good an answer for you as we can. $\endgroup$ – David K Nov 1 '17 at 18:24
  • $\begingroup$ Since merely waiting a few more days didn't elicit a better answer, I've decided to try offering a bounty. Let's see if that helps. $\endgroup$ – David K Nov 25 '17 at 15:55
  • $\begingroup$ Thanks a lot David! I am still working on this problem. So I am still interested to know and if anything comes up from my side I will post it here of course. $\endgroup$ – Koen Nov 26 '17 at 15:48

I suppose that when you compute $P(T \geq k),$ you have in mind a probability distribution of $T$ based on sampling $n$ items from $N$ of which $K$ are "unwanted". A hypergeometric distribution seems to fit the requirements.

In that light, I agree completely with your paragraph between "Suppose I just pick a value" and "If this is true".

It seems to me that then $K > K_\min$ is your confidence interval, that is, you have a one-sided confidence interval for $K$ whose lower bound is $K_\min.$

  • $\begingroup$ Hi David, indeed I had a hypergeometric distribution in mind. I also completely agree with your last paragraph. But perhaps I should clarify better what I want to find out. For example, if $n$ is very small, than chances are that $K_\min$ is a very bad lower bound. I want to have a reasonable approximation on $K$ without chosing $n$ unnecessary large. So is there a way to determine some error margin on $K_\min$? $\endgroup$ – Koen Oct 31 '17 at 18:23
  • $\begingroup$ Given some "error" tolerance $\epsilon,$ there are (somewhere) formulas for choosing $n$ for a given $\alpha$ such that you can guarantee that $|K_\min - K|/N \leq \epsilon$ (or some assurance like that). Is that the sort of thing you're looking for? $\endgroup$ – David K Oct 31 '17 at 18:35
  • $\begingroup$ Yes, that's exactly the kind of thing I am looking for! Any ideas where to find it? $\endgroup$ – Koen Oct 31 '17 at 18:41
  • $\begingroup$ There's a lot of sources about almost this question, but they usually are looking for two-sided confidence intervals. Which is annoying, because I think one-sided intervals are often more cost-effective. $\endgroup$ – David K Nov 1 '17 at 3:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.