Statistical search for two needles in random haystacks I have written a script to analyze some data at work, and for each run, it outputs a long list of integers. Each set of results is a frequency distribution (each integer appears one to many times). The result set also includes two particular integers that I have to automatically identify (called a co-pair). [P.S. The "co-pair" is just a term my supervisor uses, it is not a standard mathematical term. ]
I tested the script with smaller data sets where I already know what the co-pair should be, and realized that neither of the integers in the co-pair are always the most frequent or the least frequent in the set. Now I am at a loss what other way to statistically examine the result set and automatically find the co-pair for ANY set of data.
I have a relatively weak background in statistics so I am hoping someone can point me in the right direction.
Edit: More context.
Essentially after analyzing my data I have many sets of frequency distributions and I want to compare all of the sets at once to find answers "defining the co-pair", such as: (a) "the co-pair are ALWAYS at ogive 35 and ogive 80" or (b) "the co-pair are ALWAYS the mode and the least frequent number in each set" etc. The solution is definitely not either of the above, but what statistical methods can I use to compare many data sets to explore the relevance of particular needles in each haystack, to get an answer that works for identifying the co-pair within ALL the sets, an answer like either (a) or (b)?
Edit two:
I have thousands of data sets where I already know what the co-pairs are. I want to statistically analyze them and research what possibly defines them within each set, so I can apply the same methods to the other millions of data sets where I DO NOT know the co-pairs.
 A: The methods used to solve such problems are generally termed Machine Learning. But, for you to use these methods, you need to first postulate a model of how you think these "co-pairs" are chosen, based on your understanding of what this data actually represents.
One general method you could use is if the the data you get for each run has constant length (if not, you can always make it so using re-sampling methods). Now you have a set of $N$ vectors of length $M$ (where $N \gg M$), you want to find for each a pair of numbers $x,y$. 
You can then attempt to run a classifier method such as SVM to find, given a new vector of length $M$, what the the probabilities are to get numbers $x,y$. 
Machine learning is a complex and delicate topic, that does not lend itself very well to "plug & play" programming. You'll probably need to spend a lot of time studying it before getting good results, and if this problem is fundamental to your company, you may want to use the services of an expert in the field.
