Grading computer fails Churchill speech

Churchill’s speeches get low marks from a computerized grading system, British educators tell The Telegraph. “We shall fight on the beaches. We shall fight on the landing grounds. We shall fight in the fields and on the streets . . . ” Too repetitive.

His reference to the “might of the German army” lost him marks because the computer assumed that Churchill had intended to say “might have”, instead of using “might” as a noun.

Graham Herbert, deputy head of the Chartered Institute of Educational Assessors, said: “The computer was limited in its scope. It couldn’t cope with metaphor and didn’t understand the purpose of the speech.

Hemingway was rated “less than average” by the computer, which said he should include more detail. Anthony Burgess was judged “incomprehensible.”

Online marking of papers is being tested by exam boards and could be introduced within the next few years. It is already in use in America, where some children have learnt to write in a style which the computer appreciates, known as “schmoozing the computer”.

Via Core Knowledge Blog.

Essay-grading software lets teachers assign a lot more writing, concludes Teacher Magazine in a 2006 story. When students write more, they improve.

. . . Criterion and other essay-grading technologies have their limitations. They can’t judge the creativity of a writing style or the inventiveness of metaphors and symbolism. And I remain skeptical that artificial intelligence can effectively differentiate between a good essay and a truly excellent one.

(Teacher Aleeta) Johnson acknowledges that Criterion is not a good tool for very sophisticated writers. It wouldn’t appreciate the skill and creativity of a budding Shakespeare, for example.

But few students are budding Shakespeares.

About Joanne

Comments

  1. I once graded the best authors of the 20th century on the Six Traits. The results weren’t pretty. Hemmingway lost points on both mechanics and sentence fluency, but scored high on voice. James Joyce lost points on organization and sentence fluency, but scored really well on Word Choice.

  2. “When students write more, they improve.”

    I’m willing to bet this only holds true when their papers are graded by something close to their intended audience. If you grade by machine, then you’re teaching kids to write for the machines. This is not a good thing if the difference between machine and human grading is significant.

  3. In my experience “schmoozing the machine” is not quite so easy as it looks. I have played with the things a bit–and know a few other writers who have as well. Like many other short-cuts to proficiency (eg: teaching to the test, or test-taking strategies), they probably don’t work as well as just learning the skill or material well to begin with.

    If I were evaluating the British package, I would recommend that they keep looking. I think that there is better stuff out there. You can’t beat the machine for immediate and unbiased feedback. And I am not certain that writing for that particular audience provides less fulfillment than writing for the audience of a single teacher. Both are pretty stilted.

  4. Independent George says:

    James Joyce lost points on organization and sentence fluency

    Truthfully, I agree with the machine on this one.

  5. Heck, if a machine is writing the papers you might as well have a machine grade them.

  6. Richard Aubrey says:

    When educrats are seeking to avoid accountability, they will tell us that “teaching to the test” is a Very Bad Thing.
    It all depends, I guess.

  7. in re: Richard Aubrey

    It’s not educrats who are upset with teaching to the test. It’s anyone who really cares about a quality education. It’s why so many amazing teachers don’t teach to the test.

    Accountability should be about trust and transparency rather than rules and regulations.

    In teaching social studies and emphasizing writing, my students (low-SES, Title One) have always scored highest in our district and on the top half of the state in social studies and writing.

    Yet, I have never taught to the test. I ignore the practice test booklets. I ignore our “AIMS Blueprint.” Instead, I increase motivation, use authentic learning, follow research-based best practices and monitor student learning closely.

    I have never needed a machine to help me in this process.

  8. Richard Aubrey says:

    John Spencer.
    If the public were satisfied with the general level of education, there wouldn’t be any call for this kind of testing for accountability.
    It should not have to be said that results have dropped more or less paralell with increases in funding, and took a precipitous drop with the invention of the fed Dept of Ed.
    If, caveat here, the test tests what the public wants the kids to know, then teaching to the test is not a bad thing.
    Of course, there are teachers who think kids should know something else, something whose pursuit leaves the kids unable to perform on the test. And, more than likely, doesn’t generate a lot of homework and tests to be graded. Oops. Did I say that?
    Good teachers don’t need to teach to the test, if their curriculum prepares the kids for the test sort of by accident.
    The rest?
    The public is paying through the nose and would like the magisterium to show it’s worth the money. If you don’t mind.

  9. I would like to point out a few problems with this:

    1. The Fed Dept of Ed is exactly who is pushing for the testing when they are a part of the problem. In other words, it’s the educrats who are the problem.

    2. Parochial schools tend to score better on the tests (and have higher levels of growth) when they, in fact, operate on a lower budget and do not teach to the test. Yet, many of the parochial schools allow for a higher level of teacher autonomy. Public schools that do well on the test often take a similar approach. My district micromanages teachers and our performance is piss-poor. The district where my children attend that scores higher (despite similar demographics) doesn’t micromanage. While this is purely anecdotal evidence, I have a hunch this is a trend we would see throughout the nation.

    3. The tests are manufactured to create failure. If too many kids pass, the questions are re-normed, which makes the tests norm-referenced rather than criterion-referenced. Companies like McGraw Hill lobby to change the laws to be more test-friendly, create the textbooks, “help” with creating the standards, have consultants who help “failing” schools, sell more curriculum and programs for schools that are failing (ie Jamestown Intervention), etc. It’s a rigged system with plenty of conflict of interests.

    4. Marzano proved in his research that it is not the “curriculum” that makes the greatest difference. It is the teacher. Simply adopting great materials or forcing teachers to use lesson plans will not produce results, especially given the need for a teacher to use enrichment and intervention.

    I don’t disagree with your point entirely. Bad teachers exist and we need them out of there. Money isn’t the solution. However, if I look at what works (from my experience), it’s things like collaboration, an increase in teacher autonomy (with teachers working as researchers), authentic education, intrinsic motivation and other things that seem real counter-intuitive to the public.

    I don’t pretend to be an expert on educational reform. However, I know what works for my students and it has been an authentic approach based upon high standards.

Trackbacks

  1. [...] This post was mentioned on Twitter by Neil Adam and Neil Adam, kriley19. kriley19 said: Joanne Jacobs: Grading computer fails Churchill speech http://bit.ly/2PZF8N Full http://bit.ly/1f4nzs [...]

  2. [...] the original: Grading computer fails Churchill speech « Joanne Jacobs « Microsoft rolls out UK Bing (Computer Weekly) | News , Daily News … [...]

  3. [...] the example place here: Grading machine fails solon style « Joanne Jacobs Posted in Art, Education, Uncategorized | Tags: Art, chartered, deputy-head, its-scope-, [...]