2
$\begingroup$

I have a question with a statistical nature; I think there should be some standard theory about this issue.

Suppose I have a large data set of size $N$ items, which has an amount of $K<N$ unwanted items. I am interested in finding the value of $K$. Testing all items takes too much time, so I want to determine a suitable sample size $n<N$ of randomly selected items in the data set.

Suppose I just pick a value for $n$

Then, of a randomly sample data of size $n$, I search for the unwanted items of which there are some amount of $k\leq n$. Let this amount be a test statistic $T$, i.e. I will test on the probability $P(T \geq k)$. I can now find a smallest integer value $K_\min$ such that for the estimation $K = K_\min$ we have $P(T \geq k) \geq \alpha$. That is, for any smaller integer estimation $K<K_\min$ we have $P(T \geq k) < \alpha$. If I am correct, I can now state that with a significance level $\alpha$ we have that $K \geq K_\min$. Is that true?

If this is true, the question now is: How accurate is this lower bound?

This is also my main question. Based on the amount $n$ and accuracy level $\alpha$, what can we say about the accuracy of $K_\min$. In other words, can we determine some confidence interval on $K$ in relationship to $K_\min$ and $\alpha$?

Any tips or other approaches are very much appreciated!

Best, Koen

Edit 26 November:

Another formulation of the problem as mentioned by David K is as follows:

Given some "error" tolerance $\varepsilon$, how do we choose $n$ for a given $\alpha$ such that we can guarantee that $|K_\min−K|/N\leq \varepsilon$ (or some assurance like that)?

$\endgroup$
5
  • $\begingroup$ At this time I suggest you do not accept my answer. (You can undo the acceptance; it will cost a few points but I'm pretty sure you can get them back by accepting an answer later.) I'm hoping that if this question stays "unanswered" for another couple of days we might see a better answer than mine; I just don't have the time at the moment to find one myself. $\endgroup$
    – David K
    Commented Nov 1, 2017 at 14:36
  • $\begingroup$ Alright, just did so; I don't really bother too much about te points. I thought I was doing you a favor since I thought that no better answer would come by now, but let's wait and see. $\endgroup$
    – Koen
    Commented Nov 1, 2017 at 14:50
  • $\begingroup$ You certainly weren't hurting me, so that's OK; I just want to see if we can get as good an answer for you as we can. $\endgroup$
    – David K
    Commented Nov 1, 2017 at 18:24
  • $\begingroup$ Since merely waiting a few more days didn't elicit a better answer, I've decided to try offering a bounty. Let's see if that helps. $\endgroup$
    – David K
    Commented Nov 25, 2017 at 15:55
  • $\begingroup$ Thanks a lot David! I am still working on this problem. So I am still interested to know and if anything comes up from my side I will post it here of course. $\endgroup$
    – Koen
    Commented Nov 26, 2017 at 15:48

1 Answer 1

2
$\begingroup$

I suppose that when you compute $P(T \geq k),$ you have in mind a probability distribution of $T$ based on sampling $n$ items from $N$ of which $K$ are "unwanted". A hypergeometric distribution seems to fit the requirements.

In that light, I agree completely with your paragraph between "Suppose I just pick a value" and "If this is true".

It seems to me that then $K > K_\min$ is your confidence interval, that is, you have a one-sided confidence interval for $K$ whose lower bound is $K_\min.$

$\endgroup$
4
  • $\begingroup$ Hi David, indeed I had a hypergeometric distribution in mind. I also completely agree with your last paragraph. But perhaps I should clarify better what I want to find out. For example, if $n$ is very small, than chances are that $K_\min$ is a very bad lower bound. I want to have a reasonable approximation on $K$ without chosing $n$ unnecessary large. So is there a way to determine some error margin on $K_\min$? $\endgroup$
    – Koen
    Commented Oct 31, 2017 at 18:23
  • $\begingroup$ Given some "error" tolerance $\epsilon,$ there are (somewhere) formulas for choosing $n$ for a given $\alpha$ such that you can guarantee that $|K_\min - K|/N \leq \epsilon$ (or some assurance like that). Is that the sort of thing you're looking for? $\endgroup$
    – David K
    Commented Oct 31, 2017 at 18:35
  • $\begingroup$ Yes, that's exactly the kind of thing I am looking for! Any ideas where to find it? $\endgroup$
    – Koen
    Commented Oct 31, 2017 at 18:41
  • $\begingroup$ There's a lot of sources about almost this question, but they usually are looking for two-sided confidence intervals. Which is annoying, because I think one-sided intervals are often more cost-effective. $\endgroup$
    – David K
    Commented Nov 1, 2017 at 3:04

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .