2
$\begingroup$

Basically, there is a group of us grading a few hundred essays. We divided the number up so each of us grade 50, assigning a score 0-100 given a rubric. However, there is still subjectivity in the rubric, so we would like to normalize the grades. By this I mean if I am grading hard and someone else is grading easy, we want the final grades to be in the middle for the sake of equality.

Here was my initial thought: Each grader will assign grades to a sample of essays (n=50), and the population will be the combination of all of the samples (n=400).

The "normalized" grade would be:

Normalized grade = (original grade)+((population average) - (sample average))

Where the sample average is the average grade of whatever sample that specific essay was part of.

This seemed a little rudimentary, but that is also what we are looking for. However, is this statistically appropriate? Would using z-scores be more appropriate? If so, how would that be used in this context, provided multiple samples? My concern with z-scores is that our scale and units are the same, some folks are just grading harder than others.

Any thoughts are appreciated, thanks!

Edit: Another idea

Normalized grade = (original grade)*((population mean)/(sample mean))

This produced similar but not the same results, resulting in round differences when rounded to a whole number.

So... which method is the most appropriate for this application?

Thanks again!

$\endgroup$
0
$\begingroup$

Given that there are no essays graded by more than one grades, I would recommend a standardization procedure.

The mean $\bar X_i$ and standard deviation $S_i$ is found for the $i$th group. Scores in that group are temporarily standardized according to the formula $Z = (Original\; score) - \bar X_i)/S_i)$ The effect is that within each group the average Z-score is now 0 and the group standard deviation of group scores is now 1.

Then final scores can be determined by some method such as $(Final\; score) = 10Z + 50,$ which might make the final scores range from very low to about 100. Obvious adjustments can be made so that all scores lie within any desired interval.

Of course, it is possible that by chance, some readers get better papers than others. Imagine a perfect scoring system in which 'true' grades are distributed according to $Norm(50, 10)$. Also suppose all graders adhered perfectly to give 'true' grades. Then by simulation, I got the following four sets of scores for four groups 50 randomly chosen papers:

 x1 = rnorm(50, 50, 10);  mean(x1);  sd(x1)
 ## 48.5177
 ## 10.51593
 x2 = rnorm(50, 50, 10);  mean(x2);  sd(x2)
 ## 49.50449
 ## 10.33484
 x3 = rnorm(50, 50, 10);  mean(x3);  sd(x3)
 ## 49.86804
 ## 9.024822
 x4 = rnorm(50, 50, 10);  mean(x4);  sd(x4)
 ## 48.62404
 ## 10.33020

My guess is that differences due to chance draws of papers of this size would not be catastrophic, but you need to know that they can exist. Some sort of cross-validation scheme in which some papers are read by more than one person could minimize such differences due to sampling papers.

$\endgroup$
  • 1
    $\begingroup$ thank you for the quick reply. I had a feeling that implementing z-scores would be ideal, I just didn't know what to do once I had the ith samples transformed. That said, your answer is best for the given scenario, but perhaps I should clarify further. The "grading" is for sake of ranking. No one will see the grade/score, just their rank. Where you have Final = 10Z + 50, that seems to be just for visual purposes, right? Technically if I sorted from low to high the Z scores and did the same for the Final score with that equation, the list order would be the same? $\endgroup$ – K.C. Grimes Mar 25 '16 at 2:30
  • $\begingroup$ Right. Changing from Z to final scores as I have shown would not change the ranks. $\endgroup$ – BruceET Mar 25 '16 at 2:41
  • $\begingroup$ I'll pass this along to the group. Probably need to teach them z-scores first haha. Thanks for the help, voted. $\endgroup$ – K.C. Grimes Mar 25 '16 at 2:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.