I'm doing linguistics work and I'm trying to make a measure of linguistic diversity in a text. A simple calculation seems to be

linguistic uniformity (diversity) = number of words / unique words

But this must "privilege" shorter texts like poetry for diversity, since the author is less likely to repeat themselves in so short a space. For example, Blake's poetry gets a score of 5, whereas the KJV bible scores 79 and this has a lot to do with the size of the bible compared to Blake's poetry. If we analysed more of the Blakes's corpus, his score would definitely go higher.

How can I normalize the calculation, taking into account the inevitable repetition that comes with longer texts, and the fact that shorter texts are so small that we just haven't seen an appreciable part of an author's lexicon? I know that there's a name for this, I just can't remember what it is. I've looked at the wiki article on diversity index but it hasnt helped me.

Apologise if my question seems stupid. I'm a linguist, not a mathematician. Tagged as homework even though it's not because I'm unsure what category to put it under otherwise.

  • $\begingroup$ I would add a tag "statistics" or something similar. $\endgroup$ Sep 19, 2013 at 16:00
  • 1
    $\begingroup$ This might fit better in the crossvalidated stackexchange. $\endgroup$
    – Potato
    Sep 19, 2013 at 17:49
  • $\begingroup$ Thanks Potato, I'll try asking it there. Not much activity going on here. $\endgroup$
    – Laurence
    Sep 19, 2013 at 17:53
  • 1
    $\begingroup$ I searched for "lexical diversity" and found this: "Effects of text length on lexical diversity measures", Koizumi and In'nami, System 40(4), 2012. $\endgroup$
    – user856
    Sep 19, 2013 at 18:02
  • $\begingroup$ Thanks Rahul. That paper exactly addresses my problem. $\endgroup$
    – Laurence
    Sep 19, 2013 at 18:37


You must log in to answer this question.

Browse other questions tagged .