# Is PISA the best test?

U.S. students don’t excel on PISA (Programme for International Student Assessment), but it may not be the best test, writes Jay Mathews on Class Struggle.  He cites a math question for 15-year-olds highlighted by Brookings Institution scholar Tom Loveless, a member of the U.S. advisory board to PISA:

For a rock concert a rectangular field of size 100 m by 50 m was reserved for the audience. The concert was completely sold out and the field was full with all the fans standing. Which one of the following is likely to be the best estimate of the total number of people attending the concert?

A. 2000

B. 5000

C. 20000

D. 50000

E. 100000

Like Mathews, I answered 5,000; PISA says the answer is 20,000. Loveless agrees that the question involves trivial math and would “throw kids off.” Not every kid goes to rock concerts and not every culture is willing to cram four people in a square meter of space.

“PISA exams are written by the losing side in a century-old debate over how to teach math,” Mathews writes. The pro-PISA progressives “want to make math instruction more relevant to the real world, and emphasize mathematical reasoning more than calculation,” while the anti-PISA and pro-TIMSS  “traditionalists say you can’t reason well without mastering the fundamentals.”

Unlike TIMSS, PISA’s approach to science leans left, Mathews writes.

On PISA’s student questionnaire, those who support statements such as “I am in favor of having laws that regulate factory emissions even if this would increase the price of products” are deemed to be environmentally responsible. Those who disagree are not.

Despite the differences between PISA and TIMSS, however, some of the same countries —Hong Kong, Korea, Singapore, Taiwan and Japan — do very well on both.

The feds are spending \$350 million to help states develop common tests to go with common standards.

1. ricki says:

20000 people in a 5000 square meter area? Definitely a cultural issue – different cultures differ on what is “OK personal space.”

I know what a square meter looks like and I would not ever willingly share that amount of space with three other people.

2. Sigivald says:

ricki: You wouldn’t willingly go to a rock concert, then.

I’m pretty sure most of the developed world has the idea that popular music concerts are crowded, regardless of ideas about personal space in other contexts.

“5,000” is selected by so many people not because they really think “only one person per square meter”, I think, but because they immediately do the “what’s the area?” calculation and assume that that must be the question being asked.

In other words, it’s not just a math question, it’s a math and estimation question. Now, if you want a pure math test, that’s bad – but thinking about the real world and estimating is a very useful thing.

3. Margo/Mom says:

I am glad that you note exactly what I have, which is that there is considerable overlap among high scoring countries. Hard to find an example of TIMSS rich and PISA poor.

With regard to the cited problem, my understanding of PISA scoring is that it is somewhat deeper than evaluating whether or not the right answer was arrived at, but considering also the means by which it was gotten.

But, the survey questions, which are a rich source of information about schools, teachers, families and students, use the questions such as that cited to determine attitude toward environmentalism AMONG others in various ways. They are highly transparent, raw data is easily available to other researchers–and I believe use clusters of questions to indicate attitudes towards content–which some consider to be important.

4. Doug Sundseth says:

The correct answer to the sample question is, “Whatever the Fire Marshall allows in that jurisdiction.” The concert will sell as many tickets as the promoter is allowed to sell.

IME, a sold-out concert in a football stadium will have something like 50,000 attendees, but that includes the stands, of course. I’ll note that the question does not specify that the only area reserved for fans is the field, though a charitable interpretation would assume that.

Any question that requires charitable interpretations and specific unstated assumptions of this sort is bad, obviously.

5. Julie says:

Eh, I found the right answer and didn’t think it was that hard. SAT prep tells you to not read more into a question than what is explicitly there. They are essentially asking how many people can be crammed into that sort of space, personal space be damned. You only need to realize that your average person does not take up a square meter of space on his own. Excusing American children’s poor showing on the test by saying it comes down to cultural differences is a poor excuse, IMO.

With most word problems, you have to disregard questions of practicality and real-world applications. Come on, how many of us have done problems that go like this: “Sam has an entire pumpkin pie all to himself. He wants to eat exactly 1/4 of the pie right now, 1/8 for tonight’s dinner, save 1/6 of it for his lunch tomorrow. How much pie will then be left over?” Who in the real world eats like that?

6. Julie says:

“5,000? is selected by so many people not because they really think “only one person per square meter”, I think, but because they immediately do the “what’s the area?” calculation and assume that that must be the question being asked.

That explains it, I think. And that’s really a very dumb mistake to make.

7. Like Matthews, you don’t know (a) how big a square meter is, and (b) how crowded it is at rock concerts.

8. Cardinal Fang says:

One person per square meter? That would mean you couldn’t extend your arm and touch the person in front of you, even with the tips of your finger. Forget the rock concert– no standing room anywhere is that spacious. Five thousand is a bad answer. Even for Americans.

9. Bart says:

I’m guessing that airliners cram in about two people per square meter.

10. Nels says:

Based on what Americans are looking like these days, one square meter per person sounds about right.

11. Michael E. Lopez, Esq. says:

That is a HORRIBLE question.

In addition to the problems aforesaid, there’s also the question of whether the field in question is the only place where there are audience members.

There is a HUGE difference between “how many people attended the concert” and “how many audience members are on the field”.

And let’s look at what is being asked, really. They can’t ask how many people are on the field, becase there’s flat-out not enough information. They can’t even ask what the best estimate is, because there isn’t enough information to definitively say which estimate is better.

They’re reduced to asking which of the answers is MOST LIKELY to be the best estimate. That’s pathetic, and phrasing like that is a sure sign that your question sucks so badly you are forced to hedge and make it even worse.

Now I am no slouch at standardized tests. I know the answer to this question and I know why they are asking it. But nonetheless, whoever wrote this question had their head up their arse.

12. ricki says:

No, I never willingly would attend a rock concert. (nor do I willingly fly).

I could stand in the middle of one of the square meter sampling frames I use, stand another person in an adjacent one, and we could touch each other with extended arms. My “armspan” is close to 6 feet. A square meter really is not that large. (Regardless of whether you’re an obese person or not. I’m not obese but I am fairly tall for a woman and have long arms and legs, and have a pretty extreme sense of what is “comfortable” personal space)

And it is kind of a senseless question. And it does assume that the test subjects have been to a rock concert. And that everyone considers “crowding” to be to the same degree. Or has a good visualization of how tightly people will allow themselves to be packed.

Would a followup question be: In the event of a natural disaster, how many people would be trampled to death in the process of trying to get out of the field?

13. Lou Gots says:

In all fairness, the question does ask for the “best estimate.” Very clearly, had the question asked for a definite answer, the correct response would have been “insufficient data,” as many of us have already observed.

Nonetheless, fairness goes only so far; I concur that the question is inappropriate for being overly culture-specific.