# Guess pass

New York students can score in Level 2 — good enough for promotion to the next grade — by guessing on the end-of-year exam, claims Diane Ravitch.

Is this really true? The guess pass works, concludes Diane Senechal on Gotham Schools.  She answered randomly on the multiple-choice question on the sixth-grade English test. She filled in A, B, C, D, A, B, C, D and so on. She left the written portions of the test blank, earning a zero for that section. Final score: Level 2.

She tried the seventh grade math test using her A, B, C, D method. Final score: Level 2.

While this approach does not result in a 2 for all the tests, it comes a bit too close for comfort, and another guessing system might work. A fifth grader told me that his father had told him, “Just mark ‘C’ for all of the answers, and you will pass.” On the fifth grade ELA test, this would indeed have resulted in a 2.

Via Core Knowledge Blog, which is back in action after a long lay-off due to technical problems.

Update: Dan Botteron used a random-number generator to take the two tests multiple times. On average, half the all-guess tries were scored Level 1 and half reached Level 2. In real life, only .1 percent of students scored at Level 1, he writes.

1. Cardinal Fang says:

Ravitch’s result is indeed strange, but not for the reason she says. The question is, why was she so lucky? There were 26 multiple-choice questions on the English exam, each with four possible answers. She would expect to get about 6 or 7 right, but instead, she got 12 right. If the test answers were truly random, that would only happen about 2% of the time– she was very lucky.

But it turns out that the student only needs to get 7 right to get to Level 2. Almost half of guessers should get 7 or more right.

2. Ragnarok says:

Cardinal,

It’s quite common to find such variations in a small number of trials. As the number of trials increases, you’d expect to see the actual outcomes get closer and closer to the expected value.

And, by the way, it was Diana Senechal who got these results, not Diane Ravitch.

I’m very surprised that New Yorkers seem to like Bloomberg. As far as I can see his principal accomplishments have been to make a bigger mess of the NY public school system than when he started, to grant very large pay increases to the public unions, and to hike property taxes. Such talent deserves to be at the helm of GM.

3. I’m disturbed by this news, yet also not surprised somehow.

4. Cardinal Fang says:

“It’s quite common to find such variations in a small number of trials.”

No. It isn’t. I did the computation before I posted, taking into account the small number of trials, though 26 is not that small. Just to make sure, I checked with Mr. Fang, who does math for a living.

The standard deviation for a binomial is the square root of [the number of trials times the possibility of success times the possibility of failure]. In this case, that is about 2.08. The expectation is of course the probability of success times the number of trials, in this case 1/4 times 26, or 6.5. So we can easily see that 12, Senechal’s result, is over two standard deviations away from the expected value. The probability of a result over two standard deviations larger than the expected value is less than 2.5%.

5. Ragnarok says:

“No. It isn’t. I did the computation before I posted, taking into account the small number of trials, though 26 is not that small.”

OK, I’ll rephrase: It’s quite common to find large variations in a small number of trials.

Does her scoring 12 by guessing invalidate anything? Nope!

If you’re really interested in this, set up a computer simulation with varying parameters and see what happens. Or, if you’d like another way of looking at it, try Mlodinow’s account of the Maris/Ruth scores.

6. Diana Senechal says:

The chances of scoring 12 are indeed very small. But the chances of scoring 7 and above or not. The point was not that I got 12 points, but that I got enough points for a proficiency level of 2–that is, 7 points or more. Through guessing alone, a student has approximately a 41% chance of scoring 7 or more points.

I think the standard deviation should be 2.21:

square root (26 * 0.25 * 0.75) = 2.20794

Diana

7. Cardinal Fang says:

You’re right, Diana. The 2.08 was a typo; it’s actually 2.208 as you say, or, rounding, 2.21.

But to return to Ragnarok’s comment, “It’s quite common to find large variations in a small number of trials.” We have a way of quantifying that. It’s called statistics. Statistics can answer the question, how likely is it to get 12 or more right in a test with 26 four-choice answers? When we do the calculations for this elementary statistics problem, we find out that it’s very unlikely, with a probability of something less than 2.5%. There is no other answer; somewhat less than 2.5% is the answer.

Tests of significance on sample sizes of 26 are used all the time in scientific experiments. As the numbers get bigger, a proportional variation like that is even more unlikely– it would be vanishingly unlikely to get 120 right out of 260 by chance, and even more unlikely to get 1200 right out of 2600– but’s still very unlikely to get 12 right out of 26 by chance.

8. Ragnarok says:

What’s the central point of this thread, Cardinal? That it’s likely that you can score Level 2 by guessing. Nobody, including you, can deny that. Right? Your posts miss the forest for the trees.

First, the fact that Diana Senechal got a raw score of 12 on one trial, which is unlikely, is irrelevant. She was looking for corroboration, not proof. That should be clear to the meanest intelligence. Right? Your nattering on about how unlikely it is is equally irrelevant.

Second, you couldn’t do the basic math to calculate the standard deviation correctly, even though you checked with Mr. Fang.

Third, you apparently didn’t grok that it was Senechal, not Ravitch, who performed the experiment.

Shoddy work, Cardinal, really shoddy work.

“We have a way of quantifying that. It’s called statistics.”

Trivial tripe, Cardinal.

And by the way, the appeal to a higher authority (“I checked with Mr. Fang, who does math for a living.”) is quite pathetic.

9. Cardinal Fang says:

Wow, Ragnarok. Why don’t you just admit you were wrong and stop with the insults.

10. Cardinal Fang says:

Another surprising thing– and this is on topic, you’ll be pleased, Ragnarok– is that even though guessing is a pretty good way to get to Level 2, almost no students are so bad that’s the way they’re getting to Level 2. (Half of guessers get Level 1 and the other half get Level 2. There are virtually no students in Level 1; therefore the number of guessers in Level 2 is correspondingly small.)

Only .1% of students are at Level 1. The rest of the students are at Level 2 or higher, then. So, 99.9% of students are passing this test. What? 99.9% of NY sixth graders are adequately proficient in Language Arts?

Bad enough that you can pass the test by guessing, though students don’t seem to be actually doing that in great numbers. Worse that you can pass the test, apparently, by getting 27% right when you were doing your best.

11. Ragnarok says:

“Wow, Ragnarok. Why don’t you just admit you were wrong and stop with the insults.”

Where was I wrong, Senora Fang? Do tell.

As far as I can see, you haven’t been able to counter any of my points.

You completely missed the point of the original post, you couldn’t do the basic math to calculate the SD, you couldn’t even tell that it was Senechal and not Ravitch who performed the experiment.

And to top it all off, you had to appeal to Mr. Fang’s authority! Can’t you stand on your own feet?

As Fangfinger might have said: “Once is happenstance, twice is carelessness, thrice is terminal ignorance.”

12. Bill Leonard says:

“In real life, only .1 percent of students scored at Level 1…”

If virtually everyone — 99.9 percent of students tested — passes, then it strikes me that the test is:

— Absurdly easy, and no test of much of anything; or,

— Easily gamed, ala the Senechal results; or,

— The answers are commonly known and/or widely distributed in advance of the test; in other words, it is ridiculously easy to cheat on this test and move on to the next grade, and perhaps a lot of kids are doing exactly that.

Bill

13. Cardinal Fang says:

“It’s quite common to find such variations in a small number of trials.” That was wrong, Ragnarok, as you later conceded. No quibbling about typos can make it right.

14. Cardinal Fang says:

I vote door number 1, absurdly easy. Remember, we’ve been discussing how students have about a fifty-fifty chance of passing if they guess on the multiple choice part. But remember, there’s also a written part, which is scored by humans. Probably a student gets a third or a half or so of the writing points just by writing anything at all. It looks like a test that is almost impossible to fail. It’s so easy, students don’t even need to cheat.

I suppose there might, possibly, be something to be learned from the results of this test, looking only at students who scored in the higher levels. Maybe.

15. markm says:

If you know the answers to just two questions, guessing the other 24 gives an expected value of 6 correct guesses, for a score of 8. If 7 is passing, more than half the students that can only answer two questions will pass. If they can get any points on the essay questions at all, they’re a shoe-in.