I have a list of words and their frequencies in a text corpus. So there are words like "a", "what", "some" that have really high frequencies, and other like "neurodegenerative" that are less popular.

I want to analyze sentences by assigning to each word its score and then determine if one sentence is more "technical", or more specific to a domain than others. For example:

"I have a dog and a cat." vs. "Mitochondria is the powerhouse of the cell."

I was thinking of just calculating the average of these frequencies, but sometimes I have a sentence like:

"Migraine is a serious headache.", with average 640, and

"Typical examples of continuous functions which are not holomorphic are complex conjugation and taking the real part.", with average 600, because of the many short, very common words.

Is there any better way of evaluating such sentences to give a more realistic score, or average, that would indicate how "niche" they are?

  • $\begingroup$ This is very subjective; lots of metrics are available but there's no objective notion that tells us when one is "better" than another. A simple thing to try would be to just take the max. Does that work for you? $\endgroup$
    – lulu
    Sep 14, 2021 at 13:41
  • $\begingroup$ @lulu I was thinking of something similar! Like ignoring these very common words and looking only at the ones with a frequency lower than a threshold n, for example. And maybe averaging over these. I was only thinking whether this is computationally more expensive than doing something like I mentioned in the post... $\endgroup$
    – johnnydoe
    Sep 14, 2021 at 13:43
  • $\begingroup$ I don't see a computational problem here, unless you are looking at billions of sentences. It's just a look up for each word. $\endgroup$
    – lulu
    Sep 14, 2021 at 13:45


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.