Take the 2-minute tour ×
Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. It's 100% free, no registration required.

I have a collection of 400.000+ word-pairs. Each word-pair has an association strength, which is a measure of how related the two words are to each other (as in cow-milk). Each word-pair also has a word-frequency, which is how often the two words occur in normal conversation. There is a correlation between the two variables: as association strength increases, word-frequency increases somewhat linearly.

I want to study the effects of association strength and take word-frequency out of the picture. This means I have to correct for word-frequency, so I want to construct a subset of about 1.000 word-pairs with retains the distribution of the association strength, but has no more correlation to word-frequency. I know such a subset exists but am at a loss how to efficiently construct it.

share|improve this question
    
To do this you need to tell us how association strength and word frequency are calculated. If both are essentially linear functions of counts of how often the two words appear together in the same sentence then you may not be able achive your aim. –  Henry Jan 20 '12 at 12:44
    
I fail to see how this matters. The values have been calculated and are associated to the word-pairs. There is a correlation between association strength and word-frequency, but there is enough room to select pairs with a high association strength and relatively low word-frequency and vice versa. –  Rodin Jan 20 '12 at 13:35
    
Maybe the phrasing of the question is not very clear, any suggestions on rephrasing it are also greatly appreciated. –  Rodin Jan 20 '12 at 13:36
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.