Scholars back value-added’s value

Value-added data on student performance adds value to teacher evaluations, concludes a Brookings report by a group of well-respected scholars.  “We conclude that value-added data has an important role to play in teacher evaluation systems, but that there is much to be learned about how best to use value-added information in human resource decisions.”

At Teacher Beat, Stephen Sawchuk summarizes:

While an imperfect measure of teacher effectiveness, the correlation of year-to-year value-added estimates of teacher effectiveness is similar to predictive measures for informing high-stakes decisions in other fields, the report states. Examples include using SAT scores to determine college entrance, mortality rates and patient volume as quality measures for surgeons and hospitals, and batting averages as a gauge for selecting baseball talent.

Statistical predictions in those fields are imprecise, too, but they’re able to predict larger differences across providers than other measures and so are used, the authors write.

The traditional method of evaluating teachers identifies nearly all as effective, the Brookings authors write. That’s both inaccurate and harmful to students.

“When teacher evaluation that incorporates value-added is compared against an abstract ideal, it can easily be found wanting in that it provides only a fuzzy signal. But when it is compared to performance information in other fields in other fields or to evaluations of teachers based on other sources of information, it looks respectable and appears to provide the best signal we’ve got.”

By contrast, the Economic Policy Institute and the National Academy of Sciences issued reports criticizing the reliability of value-added measures and arguing the data should not be used to evaluate teachers.

About Joanne

Comments

  1. I’d suggest that value-added measurements used to assess teachers should also be used to assess school administrators. Indeed, an argument could be made that the application of these administrators should be done *first*, since the population of a whole school is much larger than that for which an individual teacher is responsible, which simplifies statistical reliability.

    I’m all in favor of teacher accountability, but there hasn’t been enough discussion of administrator accountability.

  2. Michael E. Lopez says:

    Administrator evaluation is definitely the place to start.

    As I’ve repeatedly said here and elsewhere, most principals know who the good teachers are, as do most students and many parents.

    If you make “value-added” the basis of administrator evaluation (or a basis) then the administrator will have incentive to act on his knowledge of which teachers do a good job and which do not.

    You may not even need to evaluate teachers.

  3. This dumb.

    1) Batting average has not been used a a primary measure of baseball talent for well over a decade. From really advanced stuff there is WAR and VORP. But far more simply, there is OBP, Slug and their sum (i.e. OPS). Slightly more advanced is OPS+, which takes a things like park effects into account. So, batting average should be irrelevant to the discussion.

    2) I don’t know of any college that bases admission decisions upon — or largely upon — SAT scores. In fact, there actually is a move AWAY from using SAT scores.

    3) Haven’t we long since debunked the use of mortality rates to evalaute surgeons and hospitals? Didn’t we already see that such thing just encourage them to turn away the sickest patients?

    This report, as you summarize it, seems to encourage education decision-makers to use the sorts of metrics that other fields have long since realized make for bad decision-making.

    This is dumb

  4. 1) Huh? What does baseball have to do with this unless this is a convoluted way of pointing out that metrics change, hopefully improve, over time? If that’s your point then get to it.

    2) Colleges have the luxury of having more applicants then they need to fill upper class berths. So they can afford to winnow out the never-gonna-make-its after they’ve attended for a while rather then keeping them from ever entering. Freshman year in many colleges is more or less about covering the material far too many high schools enjoy the luxury of being under scant obligation to cover.

    3) No. Mortality rates just need the proper context. A lousy surgeon that specializes in high-risk procedures is going to have a higher mortality rate then good surgeon with the same specialty. When comparing apples to apples isn’t a fine enough distinction you compare Macintoshes to Macintoshes. See? Not so tough if you have an interested in discriminating between the good and the bad practitioners.

  5. Peace Corps says:

    I would like to see value-added data on myself before I pass judgement. I think I’m great, but know that I am biased. I know my students know more math than they did before my classes, but do they know more than if they had had another teacher?

    I work hard to help my students understand the concepts that I am tasked to teach for each course, but am I as effective as other teachers? It seems that value-added data would help to answer that question.

  6. “A role to play” is a far cry from “the sole measure,” which is how the Los Angeles Times used value-added. Yes, the Times sprinkled disclaimers hither and yon noting that value-added is only one measure, but then they served up colorful graphics showing teachers, by name, on a continuum from “most effective” to “least effective,” based entirely on value-added measures — clearly overshadowing those disclaimers.

    It is far beyond the scope of journalists (especially as journalists are THE most challenged demographic in existence, ever, anywhere*, when it comes to math and logic) to determine how to evaluate teachers and then proceed to execute (so to speak) that evaluation. Once again, the L.A. Times teacher-rating project was a disgraceful collapse of journalistic ethics and standards. The Times did not serve as messenger; it served as judge, jury and executioner. That’s just mind-bogglingly wrong.

    *Anecdotally, based on many years as a newspaper copy editor. I challenge any newsroom veteran to disagree.

  7. Stuart Buck says:

    Caroline — is anyone in this thread defending the LA Times and urging that value-added be the sole measure? How about addressing the actual Brookings report?

  8. Stuart Buck, implicitly, posts defending value-added are defending the L.A. Times project. Also, the wrongheadedness, destructiveness, and total collapse of journalistic ethics and standards in the L.A. Times project cannot be emphasized too often, so I do tend to mention it whenever value-added measurement of teachers is brought up.

  9. Stuart Buck says:

    No, that’s not true at all. I know what you said about journalists and logic, and it’s illogical to think that value-added = LA Times. As the Brookings report itself says, “the question of whether these kinds of statistics should be published is separable from the question of whether such data should have a role in personnel decisions. It is routine for working professionals to receive consequential evaluations of their job performance, but that information is not typically broadcast to the public.”

  10. Mike in TExas says:

    Ahh, another “study” that was neither published anywhere else or peer reviewed.

    Then throw in this statement, only about a third of teachers ranked in the top quartile of value-added based on one academic year’s performance would appear in the top quartile again the next year. And ten percent of bottom quartile teachers one year would appear in the top quartile the next. which says their whole hypothesis is wrong and you don’t get proof of value added measurements being effective.

    In addition, nothing is mentioned of the researchers qualifications so I’m wondering how Joanne came up with the “well respected scholars” tag. I’m guessing its b/c they’re bad mouthing teachers so naturally Joanne assumes they are correct.

  11. Erin Johnson says:

    And yet the Board of Testing and Assessment (National Academies of Science) whole-heartedly renounced the use of VAMs as a method of evaluating teaching ability due to the instability of the current models used to connect student testing and teaching ability.

    The Brookings Institute is being misleading by saying that VAMs are better than the current approach to evaluating teachers. There is no study, no data and no evidence to suggest that VAMs are any better than the rubber-stamping evaluations done by principals. They are different, not better.

    So instead of “Scholars Back VAMs”; how about: “No One Really Knows How to Evaluate Teaching But We Really Wish That We Did” Not the most compelling of titles for a blog post, but one that has tremendously more validity than any of the VAM models.

  12. Stuart Buck says:

    Mike — if you look up the researchers’ qualifications, you’ll see what anyone familiar with education research already knows: that these are some of the most prestigious and respected education scholars in the country.

  13. Mike in TExas says:

    Yes,Whitehurst is an expert, at least according to the Brookings Institute

    Glazerman is a charter school hack

    Goldhaber is from a pro-charter group.

    Staiger is an economist.

    Stephen Raudenbush

    Loeb, the only actual educator in the group, is a school finace wonk.

  14. Stuart Buck says:

    Glazerman is a charter school hack? Not true.

  15. Mike in TExas says:

    From his Twitter account:

    co-founder and parent: Washington Yu Ying public charter school, DC resident

  16. Stuart Buck says:

    And that necessitates calling him a “hack” . . . why?

    He’s one of the smartest education scholars in the entire country . . . you can’t go to an academic conference without seeing his name on 3 or 4 top-notch papers.

    Dislike this group’s opinion if you must, but to deny their preeminent qualifications is objectively wrong.

  17. Mike in TExas says:

    Stuart,

    If his qualificiations are sooo great then where are they?

    Joanne didn’t post any, the Brookings Institute profile doesn’t suggest greatness

  18. Stuart Buck says:

    Try Google (http://www.google.com).