"A Mockery of Accountability"

In New York City, public schools receive annual report cards with a grade. Most of this grade is based on test scores. Miraculously, 84 percent of elementary and middle schools in NYC received an A this year, up from 38 percent last year.

Diane Ravitch points out in today’s Daily News that this bizarre situation should come as no surprise:

The problems with the report cards were apparent from the start. When the system was launched in 2007, testing experts warned that it relied too heavily on single-year changes in standardized test scores, which are subject to random error and therefore unreliable. But the Education Department did not listen.

But surprise or no, the grades are absurd, and they leave us with nothing. Ravitch continues:

The report card system makes a mockery of accountability. No one can be held accountable when almost everyone gets an A or B. No one can tell which schools are getting better or worse. Nor do parents get enough information to make good choices.

It would be one thing if they were not taken seriously and did not affect the lives of students, teachers, parents, and administrators. But they do. Schools with low grades are threatened with closure. Students want to go to a good school and get mixed messages about the schools they attend. Parents don’t know how to reconcile the grades with the ratings given by the state. Teachers may receive bonuses (or not) based on these grades. In the meantime, Chancellor Joel Klein has asserted that there is no problem with these grades, no problem with anything, in fact.

Accountability is in a bind. On the one hand, the DoE (in NYC and elsewhere) wants to appear successful–for PR, for elections, for funding, for renown, and what have you. We have something of the Creon syndrome, where leaders do not want to admit that they are wrong, lest they cede to the populace. On the other hand, the whole purpose of an accountability system (if it has integrity) is to help us see our problems and accomplishments clearly, not to trumpet success.

But to gain such insight, we have to know what we are trying to accomplish. With dumbed-down tests, vapid literacy programs, an overwhelming focus on test prep at the exclusion of essential subjects, and unreliable rating systems, we end up taking a yardstick to a void–and declaring miracles whenever we please.

Diana Senechal

Comments

  1. Yeah, our district makes big deals about yearly shifts in scores –nevermind that we really don’t know what these shifts mean. We make all-too-real policy changes on shaky or bad interpretations of the ever-proliferating data. It’s folly. No data is better than badly-used data. Data mania is wasting our time and sapping our strength. We need to work on deciding WHAT to teach and then teaching it amazingly well.

  2. Is it fair to compare the scoring of schools in NYC to grade inflation? If so, this is nothing new. But, just because it is part of a long tradition does not mean that it should continue.

  3. [i]Data mania is wasting our time and sapping our strength. We need to work on deciding WHAT to teach and then teaching it amazingly well.[/i]

    Well there’s the problem. Many people don’t realize that education, particularly public education, is inherently unmeasurable and any attempt to measure the efficacy of public education will result in catastrophe.

    Or not.

    What’s really happening is that the folks who enjoyed employment in a profession in which professional, and organizational, competence was neither measured nor valued are having to come to terms with the ending of that situation. Naturally, they’re trying to present the illusion of the measurement of organizational competence since that’s the next best thing to not measuring at all.

    As a reason to not measure performance cheating seems like a particularly lousy one.

  4. <<< With dumbed-down tests, vapid literacy programs, an overwhelming focus on test prep at the exclusion of essential subjects, and unreliable rating systems, we end up taking a yardstick to a void–and declaring miracles whenever we please.

    Other than that, test-driven accountability is a fine idea. (Other than that, Mrs. Lincoln, how'd you like the play?)

    Fine post, Diana.

  5. No one I know in the cooperate world feels that their annual standardized performance rating system is complete, 100% fair, or all encompassing of the work they have done all year. But some companies are better than others.
    This train is coming to the public schools and it is on the fast track. I think most people are aware of this and are beginning to accept this fact…..even the unions.

  6. Don Bemont says:

    I think the comments here tell the story very well.

    Posters say that statistical performance rating must be used because it is a part of the American culture, not because it works very well in education. In fact, it may be the sense that it is unfair and invalid in one’s corporate workplace which makes people particularly avid to see it applied to schools.

    One would think that the whole point was to express frustration rather than fix schools. Any truth to that impression?

    In the corporate world, the system can only be gamed so far. In the end, there is a concrete profit or loss, and everyone agrees that that is the only purpose of the enterprise.

    In education, the more people talk, the less they agree on the purpose of education, and regardless of the official statistics being gathered, there is no defining bottom line. This reality makes abuse not only possible but inevitable.

    The more teeth you put in the accountability scheme, the more you get manipulation, cheating, spin, and distortion of curriculum to prepare for the tests. In other words, it makes education for our young people worse, not better.

    While I sympathize with the public’s desperation, I despair at the public’s inability to think this problem through.

  7. So what’s the solution?  Abolish public schools, or public education altogether?

  8. The solution is modeling our public education system on those that work in other countries. A common denominator in more successful systems (e.g. Ireland’s or France’s) is that they have content-specific national curricula. Many of you lay people don’t realize how devoid of actual content many of our schools are, especially at the elementary level. It’s lots of “engagement”, activities, reading of uninformative fictions, rather than acquisition of important knowledge. Read E.D. Hirsch’s new book The Making of Americans for a brilliant and revolutionary expose of what we’re doing wrong.

  9. “In the corporate world, the system can only be gamed so far. In the end, there is a concrete profit or loss, and everyone agrees that that is the only purpose of the enterprise.”

    As a parent, I can attest that in the end, there is profit or loss to education. It’s “out there,” in the world experienced by students after school is over. You can count it in employment statistic, success or attendance in college or post secondary education. Heck, you can even perceive it in crime and health statistics. The standardized test scores are just markers along the way.

    Any individual test score at any point in time (“n” of one) is highly unreliable. This should not lead us to believe that the proliferation of indicators amassed across populations and remaining consistent over time are merely collections of random points.

    Is the ability to read, write and cipher a valid indicator that education is occurring? Pretty much. We can slice and dice it in all kinds of ways to show growth, or to try and tease out the in-school vs the out-of-school variables that have an impact. But, in the end, we can be pretty certain that some kids are coming out of school with a whole lot more of it than others. Some schools have looked at this and taken steps to improve. Others admire the problem. Some blame the test.

  10. Agreed, M/M, but there’s the rub. In the long term, the direction is clear. In the short term, as you say “any individual test score at any point in time (”n” of one) is highly unreliable.”

    The breakdown in logic for me is when otherwise sensible people say, “Yes, I know it’s unreliable, but we need to measure individual teachers and the fact that we don’t really know how to do it is no reason not to.”

    This is why jokes like “close enough for government work” gain traction.

  11. Robert–but, an “n of one” would be the single score of one student on one test. Highly unreliable (although this has never stopped anyone from presenting it to a parent or student as being a fair indicator of learning, intelligence, aptitude, etc.) But the way to increase reliability is to increase the numbers. The scores of an entire class–more reliable. The scores of all such classes in a school, even more. A three year trend of such, more reliable still. Throw in a means of accounting for where the student was previously and we are really homing in on something like actual learning, and teacher effect.

    One of the great benefits to me as a parent of a kid with disabilities was to be able to look beyond his individual scores and see that the preponderance of similarly identified kids in his school were, or were not doing well. Looking across other schools to see that in fact some were doing far (really far) better in those groups, said a lot about what was possible.

    The NYT article said that many principals were sitting on the teacher reports. While I am not certain that these should necessarily be distributed to parents (although in my state a teacher’s personnel records are public and can be requested by news agencies, etc) at this point, I don’t get why these would not be shared with teachers as a matter of course. Looking over the sample, there is nothing that statistically complex about them. I would hope that teachers would have sufficient mathematics background to be able to understand them.

  12. As a parent, I can attest that in the end, there is profit or loss to education.

    No, you can attest that there’s a profit or loss to the *student* and by extension the parent.

    If those outcomes are immaterial to the teacher, and as importantly, the principal then they’re profits and loses without meaning to the people who have contracted to do the educating. Since that’s the condition that applies from top to bottom in public education, for all practical purposes there’s no profit and no loss in public education.

    That’s one of the reasons public education is in the pickle it’s in.

    It’s also the reason for the popularity of edu-fads. If the professionals aren’t gauged by the quality of their professional performance that doesn’t mean they have no way to distinguish themselves. They just have to get creative and find those people whose approval has professional consequences and appeal to whatever motivates them.

    In public education that, unfortunately, consists of edu-fads that don’t have to result in kids actually learning anything since that’s not measured and thus unimportant.

    Superintendents and school boards who do have a say over the professional lives of teachers and principals may or may not be motivated by educational considerations dependent on local conditions but they’ll always be motivated by ego and where educational considerations aren’t important ego that’s all that’ll motivate them. So nonsense like “child-centered learning” and “whole language” become important not because they result in kids learning but because they allow education officials to indulge their conceits.

  13. The whole conversation is silly. Parents know, those that give a darn, which schools are performing and which aren’t. Parents are quite capable of looking around their neighborhoods and schools and determining which are preparing their students and which are failing them. We twist our undies trying to create an accountability that already exists. Stop forcing students and parents into failing schools. Provide choice via competition; charters and vouchers, and the only standardized testing required will be the SAT. The paternalism of our current system is smothering it.

Trackbacks

  1. [...] welcome development).  Guest-blogging over at Joanne Jacobs, the always insightful Diana Senechal captures the dilemma of nuance-averse accountability well.  “With dumbed-down tests, vapid literacy programs, an [...]