Who writes the tests?

“Proponents of national benchmarks seem to think that they’ll be the ones writing them,” notes Marcus A. Winters in No State Left Behind in City Journal.

Bureaucrats in the U.S. Department of Education will have other ideas. So will congressmen from lower-achieving states, which won’t want to be embarrassed by a national proficiency standard that their students can’t reach. Since any system of setting a common standard—either by federal mandate or voluntary state agreement—depends on the cooperation of lousy performers like Georgia, it’s hard to see how a demanding national standard would survive the political process. Similarly, if the NAEP became an enforceable national benchmark, pressure would grow to make it easier.

Winters proposes amending No Child Left Behind to encourage states to set high standards backed by a challenging test.

Ed Week’s Stephen Sawchuk looks at teaching common s standards and writing and scoring the tests.

Most experts in the testing community have presumed that the $350 million promised by the U.S. Department of Education to support common assessments would promote those that made greater use of open-ended items capable of measuring higher-order critical-thinking skills.

. . . The issues now on the table include the added expense of those items, as well as sensitive questions about who should be charged with the task of scoring them and whether they will prove reliable enough for high-stakes decisions.

To save the cost of human scorers and speed  turnaround time, testing companies are experimenting with software that scores open-ended responses.  Are we going to let high-stakes tests be scored by robots?

About Joanne

Comments

  1. I’m not a big fan of a national curriculum or a common standardized test, but if there is to be one, I hope the designers make it computer-adaptive. If schools want to get useful information about how students are doing, they need to get beyond a “one size fits all” grade-level test.

  2. Homeschooling Granny says:

    Winters hasn’t thought this through. If the the proficiency standard is such that teachers and principals feel most of their students will achieve it, they can teach a well rounded curriculum. Set standards high enough so that the status of their school is threatened and they will teach to the test. They will endeavor to raise those likely to score just under the cut-off over that point. There will be little incentive to work with those unlikely to make the cut or with the advanced. This, in the name that no child be left, behind is obscene.

    Winters seems to be assuming educators know what is necessary to raise levels but just haven’t been willing to do it. I’m still looking for that *magic bullet*.

  3. Homeschooling Granny says:

    oops! Embarrassing punctuation error: This, in the name that no child be left behind, is obscene.

  4. They won’t have open ended questions. The cost to score them is horrendous. You might, perhaps, get something like the ACT writing prompt, which the colleges don’t look at anyway it is so formulaic and awful.

  5. Cranberry says:

    “Set standards high enough so that the status of their school is threatened and they will teach to the test.”

    Homeschooling Granny, with all due respect, you’ve forgotten about mainstreaming. As special ed is a subcategory under NCLB, they will teach to the test no matter how low one sets proficiency. They will teach to the test because they only want to buy one set of materials for each heterogeneous class.

    In the time my children have been enrolled in the public schools, I have seen “teaching to the test” take firm hold of the curriculum. This, in a district which is in no danger of being named a failing district.

  6. Math Teacher says:

    By definition, “high” standards are not those which everyone can meet. Is it possible to have high standards, “rigorous” tests and 100% proficiency? I don’t think so… Ignoring the natural variation among humans, including cognitive variation, and expecting uniform achievement on tests is ridiculous, and sets nearly all schools up for some sort of contrived failure. Standardized testing is both abused and abusive. But boy, are those test manufacturers raking in the dough!

  7. If the the proficiency standard is such that teachers and principals feel most of their students will achieve it, they can teach a well rounded curriculum.

    Whether they will teach a well-rounded curriculum is entirely another matter. And whether, having taught this well-rounded curriculum, they will do their best to ensure that the students have learnt it, is yet a third matter.

    Judging by test scores in the USA before the introduction of No Child Left Behind, many students weren’t learning to read, let alone learning a well-rounded curriculum. See http://www.nationsreportcard.gov/reading_2007/r0003.asp – about 30% of 8th graders weren’t reaching the basic reading level.

    This, in the name that no child be left, behind is obscene.

    Okay, so in Homeschooling Granny’s view of the world, attempting to ensure that every kid except for the severely cognitively-disabled attempts to learn to read and write is not merely a good-willed but sadly flawed attempt to improve educational outcomes, it is not even an example of bone-headed stupidity, instead it is “obscene”.

    I wonder, if attempting to ensure that schools teach everyone effectively is obsence, what word is bad enough to be applied to those politicians who were quite happy to let the status quo keep running?

    Math-Teacher – the guys who wrote the NCLB law with you that 100% proficiency is unachievable. For this reason they wrote into the law that 2-3% of students could be assessed by alternative techniques, and 1% of students could be assessed against alternative standards, with the cap not applying to small schools or schools specialising in teaching the severly-cognitively disabled. See http://www.ed.gov/admins/lead/account/saa.html.

    I don’t know what it is about opponents of the NCLB who persistently spread the lie that NCLB requires 100% proficiency, I think they are far more deserving of Homsechooling Granny’s insult of “obscene” than the introducers of the law. (Note, I am not saying that Maths Teacher is a liar, probably just failed to check facts).

  8. oh, so it’s just 96-97% proficiency. that makes it all better. taht goal is a million times easier to accomplish now that 2 of my students are being judged by a different set of criteria.

  9. *that

    also, tracy, i think the goal was the by 2014 all students would be proficient. so. . .all means 100%. it’s not a lie.

    http://dese.mo.gov/divimprove/nclb/QandA.html
    check out the second question.

  10. again, i mispelled “that.” i guess i can’t type today.

  11. Julia – the law is quite clear that 1% of students can be tested according to alternative standards – in other words they are not expected to reach the same level as every student.

    I agree with you that expecting proficiency of only 96%-99% is indeed far easier than expecting proficiency of 100%, what with 100% proficiency being agreed to be impossible by the writers of the NCLB law, Math Teacher, myself and every other commentator if I have seen. If it were me writing the law, I would also have allowed a low failure rate amongst students who sit the normal tests, because weird stuff always happens even with students who know the material thoroughly.

    (And note, I think that student achievement is a whole-of-school-district-responsibility, I don’t think individual teachers of 4th/8th grade can bring an entire class up to the test standard in one year when they receive a bunch of students with widely-varying grade levels, and in the face of an obstructive school district).

  12. Margo/Mom says:

    julia–you are also overlooking the “safe harbor” provision for meeting AYP. While the general rule is that AYP is determined by graduated goals ending at 100% proficiency, safe harbor allows schools who are progressing (decreasing the number of non-proficent students by 10% each year) to merely get ever closer. Toss in some multiple year averaging (as well as the kids who are being judged proficient by alternate standards) and you will see that 100% proficiency is never a hard and fast requirement.

  13. Homeschooling Granny says:

    The people who write the best of these tests tend to include in their instructions a warning that one test should not be used as the sole criterion to evaluate students, teachers and/or schools. They will explain the limitations of their tests if people will trouble themselves to study the issues, which are arcane and require a good deal of statistics. Math Teacher writes that test manufacturers are *raking the the dough* these days but a number of them are concerned about how their tests are being used. I’ve seen quite a bit in print but the one source I recall off the top of my head is Measuring Up by Daniel Koretz. He teaches teachers about tests at Harvard. His chapter on score inflation alone is enough to give one pause.

    He also questions the merits of slicing up the normal distribution curve of test results into the quartiles carrying such labels as warning, basic, proficient, and advanced.

    I doubt that there is any regular reader of this blog who doesn’t want to see all students receive the best possible education. Where we differ is on how that might best be achieved. Along with the test makers cautions, I am concerned by the number of teachers wary, or even alarmed, by what they perceive as distorting pressures from high-stakes tests. How are the children now studying and taking these tests being affected? For instance, will they be more or less likely to drop out of school? Is this not a vast, uncontrolled experiment on our kids?

  14. Homeschooling Granny says:

    There is a new book on Amazon:
    *The Paradoxes of High Stakes Testing: How They Affect Students, Their Parents, Teachers, Principals, Schools, and Society*
    ~ George Madaus, Michael Russell, Jennifer Higgins
    No customer reviews yet.

    Product description:
    The book’s goal is to clarify for parents, the public, and policy makers what high stakes tests are and how their use affects our schools, children, and society. It explores the various uses, limitations, and paradoxical consequences of high stakes testing. The present context of testing and the reauthorization of No Child Left Behind make the proposed book timely and important. Current testing programs provide valuable information to teachers, parents, and policy-makers about students, schools, and school systems. But paradoxically, these programs have unintended yet predictable negative consequences for many students, teachers, and schools. It is essential that the public and policy-makers understand the scope and impacts that result from the inherent paradoxical nature of high-stakes testing. Testing is viewed by policy makers across party lines as an “objective” measure of student attainment and has become their tool of choice to drive educational “reform” and hold children, teachers, schools, and districts accountable. Bipartisan support for test-based accountability is firm. For example, on January of 2005 President Bush called on Congress to extend NCLB testing in math and science to freshmen, sophomores and juniors citing poor performance among high school students as a “warning and a call to action.” (NYT 01/ 13/05) Senator Kennedy, a critic of the President, nonetheless supported the President’s proposed high school testing provisions.

  15. Math Teacher says:

    Re: High-stakes standardized tests
    “Is this not a vast, uncontrolled experiment on our kids?”
    Excellent point HS Granny!

  16. The people who write the best of these tests tend to include in their instructions a warning that one test should not be used as the sole criterion to evaluate students, teachers and/or schools.

    Indeed. But politicians are in a catch-22 situation. If they use multiple tests they get criticised for testing taking up more time than teaching. If they don’t test, we don’t have the foggiest idea of what’s going on at schools. If they test without any stakes, most schools will, based on past performance, keep churning out students who haven’t even mastered the basics of reading and mathematics, let alone a well-rounded education.

    Basically, whatever politicians do, they’re going to be criticised for. I suggest keeping that in mind when you decide what words to describe what they mean.

    He also questions the merits of slicing up the normal distribution curve of test results into the quartiles carrying such labels as warning, basic, proficient, and advanced.

    Whether anyone is doing that is an open question. The National Assessment of Educational Progress sets their test results reporting by achievement levels, not by slicing up a normal distribution curve of test results. See
    http://nces.ed.gov/nationsreportcard/tdw/analysis/describing.asp,
    and
    http://nces.ed.gov/nationsreportcard/glossary.asp#achievement_levels.
    The reading levels are available at http://nces.ed.gov/nationsreportcard/reading/achieveall.asp

    I wonder what Daniel Koretz’s motive is in questioning the merits of a practice that has already been abandoned by the main source of education results in the USA. It is a very common rhetorical move, when you dislike something but can’t attack it on the merits, to just state aloud blithely that “x is a very bad pracitce”, implying your opponents are doing x, but allowing yourself an out if anyone calls you on the implication.

    I doubt that there is any regular reader of this blog who doesn’t want to see all students receive the best possible education. Where we differ is on how that might best be achieved.

    And also in our interpretations of the motives of those people who disagree with us.

    I am concerned by the number of teachers wary, or even alarmed, by what they perceive as distorting pressures from high-stakes tests.

    That’s nice. Are you equally concerned by the large numbers of students who failed to learn to read and do basic arithemtic under the old system?

    And also, isn’t the whole point of high-stakes testing to create pressures to distort education in the desired line? Perhaps this won’t work out as planned, things seldom do, but an education system undistorted by high-stakes tests wasn’t doing a good job of educating students.

    Is this not a vast, uncontrolled experiment on our kids?

    And the old system wasn’t?

  17. Homeschooling Granny says:

    Tracy W
    I lack your confidence that the current course of accountability is taking us where we really want to go. I greatly fear that when we can look back and assess the results, when children who are now in 4th grade graduate from high school, or fail to, we shall be very much disappointed.

  18. Tracy W, if you wish to criticize Measuring Up, by Koretz, you should read it.

  19. Homeschooling Granny – I’m not at all confident that the current system will wind up where you want to go. What I lack is your confidence in the old system.

    As for criticising Koretz, you are right. Please read my criticism as a criticism of your argument, drawing from Koretz, and not one aimed directly at Koretz himself.

Trackbacks

  1. […] This post was mentioned on Twitter by kriley19, JoanneLeeJacobs and JoanneLeeJacobs, Sharon Lopez. Sharon Lopez said: Who writes the tests? « Joanne Jacobs http://bit.ly/4Ev3gE […]