Value-added evaluation's 'fatal' flaws

Evaluating teachers’ performance by how much they raise students’ test scores — is “fatally flawed,” writes Dan Willingham on Britannica Blog. Among his objections to value-added analyses:

Suppose Teacher A has a class of high-achievers, and Teacher B has a class of low-achievers. The fact that we’re looking at change scores is supposed mean that if each class improves, say, 10 points on a reading scale, we infer that the teachers are equally effective. But who says it’s equally hard or easy to move high-achievers and low-achievers 10 points on the reading scale?

It’s OK to use value-added analysis for research, he writes, but not to determine who’s a good or bad teacher.

Using an unreliable measure to make important personnel decisions is a certain way to engender mistrust and lower morale.

Eduwonkette lists a series of problems with value-added performance measures. Among them are questions about how schools and colleagues affect teachers’ effectiveness and whether teachers who promote short-term score gains are equally “effective in promoting longer-term academic growth.”

About Joanne


  1. I’m surprised that a study hasn’t come to light that proves that attempting to measure teaching skill results in global warming.

  2. Roger Sweeny says:

    The fact that no measure of teacher merit is perfect can be used to argue that we should pay all teachers the same.

    Yet just about every school system in America has a system which pays some teachers more and some less. Teachers who have taken many education courses can get considerably more than teachers who haven’t, even though this is an extremely imperfect indicator of teacher effectiveness.

    I wonder why there aren’t more research papers arguing against this system. Oh, wait …

  3. As a former 5th grade teacher, I agree with Mr. Willingham. there are so many variables to take into consideration. When I was teaching, and what made me run for the exits after two years, was the notion that in my large district, the wealthier schools consistently pulled away the most experience teachers that were not incompetent. This left schools like mine with inexperienced or incompetent teachers serving the hardest to reach groups of students.

    I also like Eduwonkette for raising the issue of peer-teacher effect. As an example, I had kids come into my class from basically six different classes the year before (three from my school and three from a different school that merged). The kids from my own school were much easier to teacher because they had a better grounding in the basics and had a better grasp of school rules and etiquette (raising hands, not bothering others during study time, etc..)

    The other kids were a mess. Not only were they significantly behind academically, they were a terror to control. Obviously, my ability to work with kids passed on from better teachers directly related to my ability to improve their test scores.

    When you spend much of your time on classroom management and remedial instruction, it can shut down your gains. I had fantastic gains from almost all of the students that came from my school, and poor to mediocre results with the kids from the other school.

    I think as far as merit pay is concerned, it really should be left up to the principal to identify and reward good teaching, knowing the entire school’s problems and advantages. The principal can and should then be held responsible for those decisions.

    Thanks for the post!

  4. Colin: the trouble with leaving teacher merit pay up to the principal is that nowadays, most school administrators seem to be the ones that were too stupid to make it as teachers. Not that I’d expect an organization headed by an incompetent fool to function well no matter how much money and how many brilliant improvement plans were available…

    There are a few schools that do well at teaching the sort of students most commenters on this web site consider nearly unteachable – impoverished, with parents that don’t give a damn, and mired in a culture that denigrates academic achievement. These schools have very good principals, who are also unencumbered by regulations imposed from above.

    So how about this plan: give principals the authority and freedom to hire and fire their staff, determine merit pay, run their own budget, and choose their curriculum and books. Then fire those who do poorly, and don’t replace them with other school systems’ rejects, no matter how many degrees in education administration they have.

  5. > Colin: the trouble with leaving teacher merit pay up to the principal is that nowadays, most school administrators seem to be the ones that were too stupid to make it as teachers.

    The obvious solution then is that principals ought have their pay, and job security, determined by their competence as principals.

    Goose, gander, etc.

  6. Thanks Markm and Allen. I agree with you that many principals are incompetent. I was kind of advocating the position of both of you, which is that principals should be given wide latitude to run their school and then be held responsible for the results. If they reward poor teachers and make poor decisions, they should be fired.