I am implementing an algorithm to determine a similarity score between two text documents, Document A and Document B, and I need some help with the math for computing similarity scores.
- For each document
- Break the document into sentences
- Record the hash value for each sentence into a Set
Now there is a Set of hashes for Doc. A, and a Set of Hashes for Doc. B. Take the intersection of the two Sets, and:
Similarity = # hashes in intersection __________________________ max(total # of hashes in Doc. A, total # of hashes in Doc. B
In essence, if Doc. A has 120 sentences, Doc. B has 15o sentences, and they share 75 of the same exact sentences, then the similarity score is 75/max(120,250) = 75/150 = 0.5.
Question 1: Is that a legitimate calculation for similarity? I'm especially concerned about taking the max() in the denominator.
- For each document
- Break the document into sentences (and record the hash value for each sentence)
- For each sentence, slide through the sentence with a basic sliding window algorithm
- Record the hash value for each window into a Set
Here is a quick example of a basic sliding window:
"Jack and Jill ran up the hill because Jill wanted to fetch a pail of water. Unfortunately, Jack fell down and broke his crown. Even worse, Jill came tumbling after." window 1 = Jack and Jill ran up window 2 = and Jill ran up the window 3 = Jill ran up the hill window 4 = ...
Method 2 should be more accurate in that it will pick up on potential changes within the sentences. At this point, I can compute the similarity in the same way that I did in Method 1 - the Sets will just be larger. But instead, I think it would be more accurate/more optimal to:
- Take the intersection between the sets of hashed sentences, and compute the score as described in Method 1.
- For any elements NOT in the intersection, compare the hashed windows of each sentence, rather than the hashed sentences. Compute similarity as before, with (intersection of windows / max(windows of Doc. A, windows Doc. B))
Question 2: Now that I have two partial scores, how can I combine them into one (accurate) similarity score?