13
$\begingroup$

I have words represented as vectors, and so I can compare two words using the cosine similarity of each word vector.

But, now I'd like to extrapolate that and compare two sentences, each being a set of word vectors.

What is the most robust way to go about this?

Should I compute the mean of each set of vectors and then compute the cosine-similarity of each mean vector? Should I be normalizing the vectors first?

How does that compare to my naive approach of scoring each word pair and then simply taking the mean of the scores as the similarity between the two sets?

Any insights are greatly appreciated. Thanks.

$\endgroup$
1
  • 1
    $\begingroup$ What did you end up doing? $\endgroup$
    – Lee
    Jan 10, 2016 at 10:24

3 Answers 3

3
$\begingroup$

To calculate cosine similarity between to sentences i am using this approach:

  1. Calculate cosine distance between each word vectors in both vector sets (A and B)
  2. Find pairs from A and B with maximum score
  3. Multiply or sum it to get similarity score of A and B

This approach shows much better results for me than vector averaging.

Here some python code:

import numpy as np
A = [list of word vectors]
B = [list of word vectors]

qd = np.dot(np.vstack(A), np.vstack(B).T)
rel = 1
for r in np.amax(qd, axis=1):
    rel *= r
$\endgroup$
2
$\begingroup$

Use this function n_similarity from the word2vec model in the python package gensim. http://radimrehurek.com/gensim/models/word2vec.html#gensim.models.word2vec.Word2Vec.n_similarity

$\endgroup$
1
$\begingroup$

Typically you go for the bag of words approach. Where you have a vectorizer where each index is a location of a word in a dictionary and you can count the number of occurances of that word by incrementing the count at that index. Then you have two vectors you can take the cosine similarity of. If you are using Python check out scikit learn or natural language package for vectorisers.

An example.

Take two sentences such as

"My name is chinny84 and I like MSE" and "my name is scott and I like MSE" then I can create a dictionary (vocab) with the support $$ (\text{my},\text{name},\text{is},\text{and},\text{robert},\text{scott},\text{i},\text{like},\text{MSE}) $$ And if we compute the vectors for both sentences we find $$ (1,1,1,1,1,1,0,1,1,1) $$ And $$ (1,1,1,1,1,0,1,1,1,1) $$ And take the cosine similarity. To add more occurances you increment the counter at the respective index.

Remember there are other similarity distance metrics you can use beyond cosine. Also, remember if you want to match exactly then you keep "stop" words such as "I", "and" etc to give significant terms.

$\endgroup$
2
  • $\begingroup$ Thanks. But I'm trying to incorporate the word vectors I already have. Using google's word2vec, I have a 300 dimensional vector for each word, and I quite like the result I get from cosine similarity on a word-to-word basis. I only wish to extrapolate the word-to-word scores to a bag-of-word to bag-of-word use case. So, for now, I'm taking the mean of the vectors in sentece a and the mean of b, and computing the cosine sim between those means. I'm just wondering if that's the correct approach given I wish to keep the word2vec scores in the picture. $\endgroup$ Jan 27, 2015 at 22:19
  • $\begingroup$ I have to admit I have never tried the approach you have. But my only hesitation is that word word comparison is good for determine the distance from one word such as "bag" to "bug" but I am not sure how on scale ie a sentence the distribution of letters are going to Bias your results? Ie the dominance of vowels for example may cause havoc. Similar to using the bag of words and removing stop words. But it is an interesting approach and with all machine learning its about trying and seeing if it gives the best fit. $\endgroup$
    – Chinny84
    Jan 27, 2015 at 22:24

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .