I love the studies carried out over 110 years ago by Starch & Elliott (1912, 1913a, 1913b). In short, they tested the reliability of English teachers grading papers (1912), and got disastrous results showing an absurd amount of variation in scores across many teachers. Then, they did a second study with geometry teachers (1913), got even greater variation of scores, and finally did a third study with history teachers, essentially replicating the results from the other two.

I often cite these when talking shop, saying something like “we’ve known for 100 years that grading can be incredibly unreliable,” but recently I revisited these foundational studies, and now have an even greater appreciation for their design and findings. In this blog post, I’ll dig into these groundbreaking studies, starting with the 1912 edition…