The risk of testing

In our zeal for accountability, we’re Assessing Ourselves To Death, writes Matthew Di Carlo in a characteristically thoughtful post on Shanker Blog.

To start with, “educational outcomes, such as graduation and test scores, are signals of or proxies for the traits that lead to success in life, not the cause of that success,” he writes. Pumping up graduation rates won’t improve students’ prospects — or the economy — unless they’ve actually learned the academic and non-cognitive skills employers associate with a high school diploma.

Our relentless focus on test scores has risks, Di Carlo writes. Test-based accountability “has a useful role to play, both for measuring performance and for incentivizing improvement (and, of course, the use of testing data for research purposes is critical),” but “we need to stop putting more and more faith in instruments that are not really designed to bear that burden.”

If we mold policy such that livelihoods depend on increasing scores, and we select and deselect people and institutions based on their ability to do so, then, over time, scores will most likely go up.

The question is what that will mean. A portion of this increase will reflect a concurrent improvement in useful skills and knowledge. But part of it will not (e.g., various forms of score inflation). To the degree the latter is the case, not only will it not help the students, but we will have more and more trouble knowing where we stand. Researchers will be less able to evaluate policies. We’ll end up celebrating and making decisions based on success that isn’t really success, and that’s worse than outright failure.

We need to balance “the power of measurement and incentives against the risks,” Di Carlo concludes.

About Joanne


  1. FuzzyRider says:

    In my experience, educators (especially administration/bureaucratic types) never seem to be able to grasp the idea that correlation is not necessarily an indication of causality!

    • Amy in Texas says:

      That’s because they didn’t take stats in college.

    • That’s been true for decades and it has driven lots of doomed-to-fail “reforms:” Latin, 8th-grade algebra, modern foreign languages, debate, APs, higher math/science etc. All of those things correlated with higher achievement/SATs/college success, so the ed bureaucrats jumped to the conclusion that having ALL students take them will magically lead to higher achievement! No, it won’t and it didn’t; those were simply proxies for identifying the most able and motivated students, because they were the only students taking those courses. In the real world, lots of kids lack the cognitive ability, preparation and/or motivation to handle them; hence the existence of many “algebra”, “chemistry” and “AP” classes that are nothing of the sort. It’s another variety of academic fraud. Academic achievement doesn’t rise for those who don’t belong in the class at all and the kids who should be getting the real version of the class are short-changed.

  2. Roger Sweeny says:

    There are some deep issues lurking here: what exactly are students supposed to get from schools? To say that they are supposed to get various things that are not picked up by tests is obvious and useless.

    What are they supposed to get? How do we determine whether they have or not?

    • Stacy in NJ says:

      Stop being so sensible, Roger. Your habit of seeing the forest instead of the trees is annoying. (In case there’s any doubt, this is a joke).

    • What are they supposed to get? How do we determine whether they have or not?

      If your measurements show disparate impact, you’re screwed.

      • That’s what’s driving lots of really bad ideas; heterogeneous classes (including mainstreaming and full inclusion), weak &/or flawed curricula like Everyday Math, Readers’/Writers’ Workshop, discovery learning, groupwork, artsy projects, failure to include real literature, history (including art/music), geography and the sciences, failure to challenge many kids, social promotion, appropriate discipline (now doomed) etc. Real academic work, appropriately/fairly graded (without inflation) , means that there won’t be equal outcomes.

  3. Should I be the one to let Mr. De Carlo in on the secret or does someone else want to inform Mr. De Carlo that grade inflation was a problem well before educational accountability was much more then a gleam in the eye of various proponents of the idea?

    Oh, he probably knows that I suppose, writing for The Voice of the Al Shanker Institute. No teacher’s unions guy isn’t aware of the long-standing complaints about grade inflation. Even in the absence of teacher accountability.

    And I suppose it’s too much to ask of Mr. De Carlo to delve into the reason why grade inflation predates teacher accountability so I guess if falls to me to explain.

    You see Mr. De Carlo, grades are supposed to be a measure of whether learning’s occurred. But over the years grades were inflated to mollify worried parents and annoying critics. It’s easy, right? There’s no audits so you can grade kids anyway that’s convenient and convenient is handing out grades that have less to do with learning and more to do with the comfort and convenience of the professionals and elected representatives.

    But if testing that’s not easily suborned starts to become common that pleasant state of affairs is jeopardized. Bad teachers might get the boot and good teachers might be more attractive as hiring prospect, leading to higher pay.

    Can’ have that so testing has to be portrayed as a chancy business that could easily lead to catastrophes unseen since Biblical times.

  4. Mark Roulo says:

    Matthew writes:

    We could very easily increase the graduation rate by easing requirements, but this wouldn’t do much to help kids advance in the labor market. They might get a few more calls for interviews, but over the long haul, they’d still be at a tremendous disadvantage if they lacked the required skills and work habits.

    Moreover, employers would quickly catch on, and adjust course accordingly. They’d stop relying as much on high school graduation to screen potential workers. This would not only deflate the economic value of a diploma, but high school completion would also become a less useful measure for policymakers and researchers.

    I fear that he may have made an error with his tenses:

    “We could very easily increase the graduation rate by easing requirements”, but “employers would quickly catch on, and adjust course accordingly.”


    We *did* very easily …

  5. Florida resident says:

    A friend of mine, D. A., from Boulder, Colorado, taught me an expression:
    “If you are not smart, you should better get an education”.

    Indeed, Bill Gates, Michael Dell have not got University diplomas, but they were smart, and they became successful in life anyway. Steve Jobs got terrible university education, but he was smart.

    Respectfully submitted by F.r.