Take the 2-minute tour ×
Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. It's 100% free, no registration required.

I'm computing cosine similarities between 2 vectors.

These vectors are information retrieval query and document representations respectively.

They have been computed using tf/idf weights.

Since my documents have different length, tf/idf weights are theoretically unbounded.

The question is: is cosine similarity still a valid measure ? Can I compare several cosine similarities for each doc ?

thanks

share|improve this question
2  
For the uninformed like me: en.wikipedia.org/wiki/Tf%2Didf –  t.b. Mar 29 '11 at 9:29
    
Try asking at metaoptimize.com/qa. It's the q&a forum for machine learning related topics, including information retrieval. And it's just a hunch but if your vectors are defined over the entire vocabulary, and elements corresponding to words that don't appear in the document are given a value of zero, then I don't see why you'd have trouble doing cosine. –  JasonMond May 28 '11 at 18:45

1 Answer 1

up vote 1 down vote accepted

If I read Wikipedia right, tf/idf is not unbounded. tf $\le 1$ (would be 1 only if the document had all words the same) and idf $\le \log N, N$ the number of documents, with equality if only one document has the term. Despite the slash in tf/idf, these are multiplied so the limit is $\log N$.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.