I have a list of words and their frequencies in a text corpus. So there are words like "a", "what", "some" that have really high frequencies, and other like "neurodegenerative" that are less popular.
I want to analyze sentences by assigning to each word its score and then determine if one sentence is more "technical", or more specific to a domain than others. For example:
"I have a dog and a cat." vs. "Mitochondria is the powerhouse of the cell."
I was thinking of just calculating the average of these frequencies, but sometimes I have a sentence like:
"Migraine is a serious headache.", with average 640, and
"Typical examples of continuous functions which are not holomorphic are complex conjugation and taking the real part.", with average 600, because of the many short, very common words.
Is there any better way of evaluating such sentences to give a more realistic score, or average, that would indicate how "niche" they are?