I am decades out of school and need help determining the correct statistic / concept to use for my use case. My agency (human services) uses a specific functional test to determine how well clients are functioning in their daily lives. We are looking for ways to automatically test how well staff are scoring the test. The test has 20 items, each of which can be rated from 1 to 7. One of the measures of test quality is that each individual administration should have some variability in the item level scores (it would be less probably for all items to be the same score OR for all items to be mainly different).
- What might be the best statistic to determine the variability of item level scores within a given administration of a test?
- What might be the best statistic to determine the variability of item level scores within a sample of these tests?
I have looked at variance and standard deviation both. However i worry about variance due to many stats text books indicating this concept is nearly useless. I also worry about using standard deviation because I don't know or have an accurate population standard deviation to compare it to and I don't want to use my samples to determine this (one of the things we are trying to do is generally verify how correctly staff are scoring the tests).