Did Bill Gates waste $1 billion because he didn’t understand standard deviation? Marginal Revolution links to Howard Wainer’s Picturing the Uncertain World, which argues that small schools look better than they really are.
The problem is that because small school don’t have a lot of students, scores are much more variable. If for random reasons a few geniuses happen to enroll one year in a small school scores jump up and if a few extra dullards enroll the next year scores fall.
Thus, for purely random reasons we would expect small schools to be among the best performing schools in any givenyear. Of course we would also expect small schools to be among the worst performing schools in any given year! And in fact, once we look at all the data this is exactly what we see.
. . . States like North Carolina which reward schools for big performance gains without correcting for size end up rewarding small schools for random reasons. Worst yet, the focus on small schools may actually be counter-productive because large schools do have important advantages such as being able to offer more advanced classes and better facilities.
The Gates Foundation noticed that breaking large schools into smaller schools didn’t produce the hoped-for results and switched its focus to improving teaching. Yet many people think small schools (with good teachers and well-designed curricula) can reach students who’d drift in a big, impersonal high school.