Review: 75% of charter studies are flawed

Seventy-five percent of charter school studies are flawed because they fail to account for charter students’ differences in academic background and performance, according to a meta-analysis published in Science.

High-quality research is emerging from charters that use lotteries to pick students, write Julian R. Betts, a professor of economics at the University of California, San Diego, and Richard C. Atkinson, a former president of the University of California system who once served as director of the National Science Foundation. Students who apply to a charter but lose the lottery represent a sound control group, they write.

The relatively small number of lottery-based studies of charter schools have generally shown that they either outperform or perform at the same level as traditional public schools, according to the authors. But those studies cover only a small fraction—about 2 percent—of charter schools nationally.

However, charters that need lotteries for admission may be unusually good schools, the authors warn.

About Joanne


  1. The very reality that a lottery system is in place reveals an undocumented bias to the studies. For, the requirement that students must apply and seek the change reveals that lotteries are already cherry picking more motivated and exceptional students who have cherry picked themselves into the lottery.

    And the reality that only 20% or charters outperform the neighborhood school, while 60% actually perform worse is truly shocking.

    • Studies of charters that use lotteries to pick students compare students who applied to the school but lost the lottery to those who applied and won. That takes out the issue of motivation.

      • The very act of applying is motivation.

        • Michael,

          From the article:

          “High-quality research on charters is nonetheless beginning to emerge, they say. Much of it is coming from charters that have so many applicants that they must use lotteries for admission. Because getting in or not getting in is based on chance, the students who fail to secure a spot represent a sound “control” group necessary for a study, Betts and Atkinson say.”

          That said, you do have a point about the difference in the student population at large being different in charters vs. non-charters.

        • Parents who want their kids to get a decent education.

          it’s interesting that defenders of the status quo have to imply that’s a subset of parents rather then the overwhelming percentage of parents, in order to undercut the competition.

          • Allen, as usual you see education as a one-way street with 100% guaranteed success of academic achievement placed on teachers.

            While I 1.) do not support the status quo 2.) don’t belong to a union 3.) do not defend intractable teacher contracts, I am not unreasonable about the complexities of student achievement. But, you are.

          • And as usual Micheal, as with all proponents of the current system, if I fail to accept the assumptions you make you necessarily resort to misstating what I’ve written.

            But dishonesty is just one price that support for the current public education system exacts. Another price is learning to accept that your professional skills are immaterial to the system which employs you and you seek to protect from criticism and competition.

            Another price is buying into the notion that parents aren’t motivated. It’s a nasty, and self-protective, assumption but one that must be accepted. Otherwise one of the cornerstones of the current model of public education, mandatory attendance, is undercut.

            So charters, along with demonstrating the superfluousness of the district model, demonstrate parental motivation to get their kids the best education they can.

          • Just wondering, Allen, how much experience you have with the system. How much regular contact do you have with large groups of parents and students in which to evaluate their motivation.

            I’ve taught in Taiwan and the USA, in the city of Chicago and suburbs in Illinois and Colorado, and in public and private over a twenty year career.

            Et tu?

          • Nationwide, 2% of students pursue charter school education.

          • Ah, the inevitable appeal to authority. Game, set, match. Thanks for playing Michael. I’m sure you’ll do better when confronted by someone who isn’t familiar with all your moves, subterfuges and dissemblings.

            What’s interesting about this exchange is that the game’s already over.

            Rather soon now it’s going to seep into the public consciousness that the public education system is overdue for sweeping, substantive reform. The reform’s under weigh but the public discourse hasn’t caught up with the tide of events. But that’ll change soon and then discussion like this will be obviously as relevant to current events as re-fighting Civil War battles.

            Oh, and as for the percentage of students attending charters, that’ll change pretty dramatically once it’s apparent that kids can be educated quite well, thank you very much, without having to pay mobs of non-teaching professionals whose contribution to the function of educating kids is non-existant.

          • Just curious about your experience. Clearly, you have little. Though we know what they say about opinions.

          • According to polls, 75% of Americans are “very satisfied” with their children’s school, and 85% of Americans are “very satisfied” with their own education. So, I guess I am just baffled by your portrayals and insinuations.

          • Really? The poll that matters makes it unarguable that support for charters, and other educational alternatives, runs at about 80% to 88%. That would be the number of states – forty or forty-four depending on which source you consult – that have charter-enabling law, vouchers and tax credits.

            Oh, and save the archness for the twelve year-olds who are impressed by it. If you don’t have any worthwhile response, as is rather strongly suggested by your retreat to the appeal to authority, then the difference in the value of opinions, and presumably “what they say about opinions”, is demonstrated. My congratulations though on tip-toeing around a mundane vulgarism while fully embracing it.

            By the way, Michigan’s charter cap comes off completely in three years. Joanne seems indifferent to that interesting development and the brave soldiers of the status quo seem frozen to muteness at the development. Feel free to break the logjam.

  2. Roger Sweeny says:

    The Science article is right. Most charter studies are flawed “because they fail to account for charter students’ differences in academic background and performance.” In other words, kids who go to charters are generally more motivated and more likely to succeed whether they go to charters or not.

    There is another, larger and more influential, group of studies that make the same mistake. People with college degrees make more money than people without. But the groups are very different. People in the first group work harder, are more likely to “defer gratification,” are smarter, and a whole lot more. If colleges didn’t exist, they would still be, on average, much more successful than people in the other group.

  3. If the studies are flawed “because they fail to account for charter students’ differences in academic background and performance.” then wouldn’t it be just as likely that charter school students were having problems in the schools they left to attend the charters?

    • Roger Sweeny says:

      It certainly is possible. In fact, from what I understand, most students who apply to charter schools were not doing very well in their regular schools. Most charter applicants come from schools where few students are doing well. The question would be whether the charter charter applicants were, on average, doing better or worse than students who stayed in their non-charter schools.

      Of course, as Joanne says, studies of lottery winners and losers compare two randomly divided groups within the charter applicant pool, so how applicants compare to non-applicants is not an issue there.

  4. If you accept that comparisons of lottery-winning and lotter-losing students is an adequate control for differences in student characteristis, there is still the question of why lottery winners do better than lottery losers. In short, is it the better instruction, longer days, etc or is it that charter-attenders are not in classes with lots of students who did not enter the lottery and whose social and educational challenges create a more difficult learning environment.