Basically, there is a group of us grading a few hundred essays. We divided the number up so each of us grade 50, assigning a score 0-100 given a rubric. However, there is still subjectivity in the rubric, so we would like to normalize the grades. By this I mean if I am grading hard and someone else is grading easy, we want the final grades to be in the middle for the sake of equality.
Here was my initial thought: Each grader will assign grades to a sample of essays (n=50), and the population will be the combination of all of the samples (n=400).
The "normalized" grade would be:
Normalized grade = (original grade)+((population average) - (sample average))
Where the sample average is the average grade of whatever sample that specific essay was part of.
This seemed a little rudimentary, but that is also what we are looking for. However, is this statistically appropriate? Would using z-scores be more appropriate? If so, how would that be used in this context, provided multiple samples? My concern with z-scores is that our scale and units are the same, some folks are just grading harder than others.
Any thoughts are appreciated, thanks!
Edit: Another idea
Normalized grade = (original grade)*((population mean)/(sample mean))
This produced similar but not the same results, resulting in round differences when rounded to a whole number.
So... which method is the most appropriate for this application?