Robo-writing tutors

Robo-readers are better than humans at helping students improve their writing, argues Annie Murphy Paul on the Hechinger Report. “The computer functions not as a grader but as a proofreader and basic writing tutor, providing feedback on drafts, which students then use to revise their papers before handing them in to a human,” she writes.

At the New Jersey Institute of Technology, which has used E-Rater since 2009, students are far more willing to revise their essays if they get feedback from a computer rather than a human teacher.  They write more and improve more.

Rewriting seems like a game, a way to get a higher score, said Andrew Klobucar, a humanities professor.

Instructors’ criticism is seen by students as “corrective, even punitive,”  he said. When E-Rater suggests a rewrite, students don’t take it personally.

In a study at Alexandria University, students learning to teach English as a foreign language received feedback on two essay drafts from a robo-reader program called Criterion.

As in New Jersey, students liked the immediate response, saw writing a new draft as a game and preferred non-human feedback.

Comments and criticism from a human instructor actually had a negative effect on students’ attitudes about revision and on their willingness to write, the researchers note. By contrast, interactions with the computer produced overwhelmingly positive feelings, as well as an actual change in behavior — from “virtually never” revising, to revising and resubmitting at a rate of 100 percent.

. . . the students’ writing improved; they repeated words less often, used shorter, simpler sentences, and corrected their grammar and spelling. . . . Follow-up interviews with the study’s participants suggested that the computer feedback actually stimulated reflectiveness in the students — which, notably, feedback from instructors had not done.

Robo-graders are a bad idea, concludes Paul. But robo-writing coaches may be a very good idea.

When critics like Les Perelman of MIT claim that robo-graders can’t be as good as human graders, it’s because robo-graders lack human insight, human nuance, human judgment. But it’s the very non-humanness of a computer that may encourage students to experiment, to explore, to share a messy rough draft without self-consciousness or embarrassment.

Automated software would let teachers assign more writing without creating an impossible burden for themselves.

The only way to learn to write is to write and rewrite.

About Joanne

Comments

  1. “students are far more willing to revise their essays if they get feedback from a computer rather than a human teacher.”

    Despite our ideals about inspirational, life-changing teachers, it appears that personal relationships with a teacher can backfire. How demoralizing for some teachers to realize that one’s students improve more when dealing with a computer than with one personally. Personal attention is great, but personal feedback, even where completely warranted and necessary, is really hard to take. Too often, it feels like a personal attack.

    Just musing, maybe this is why Judeo-Christian religions traditionally didn’t emphasize a “personal God;” perhaps it’s easier to improve one’s behavior for a distant, objective authority than for a God “who is one of us,” a la Jewel’s song. After all, a pal should overlook a lot out of friendship, right?

    • Stacy in NJ says:

      Kids who play video games get constant feedback on their performance. They win or lose and learn to make incremental adjustments to enhance outcome (more winning). A software program is completely nonjudgmental, unbiased, and consistent.

  2. Well, this makes sense. Running a spell-check is a less touchy ordeal than having a human point out a typo.

    But is the robo-readers’ advice good? That’s my biggest question.

    Sometimes human criticism stings because it’s true.

    It would be quite a sophisticated robo-reader that recognized when a writer was confusing “tubid” with “turgid,” for instance, or quoting an author unnecessarily, or committing a fallacy.

    • cranberry says:

      “used shorter, simpler sentences”

      Doesn’t necessarily coincide with better writing.

  3. I wonder if this has a link with the story Salman Khan tells about how he started doing his videos. He said the cousins preferred the asynch videos. Not as a slight to him but because they didn’t develop a sense they were imposing on him by not getting it right away and having to ask him to go over it again.

    Could human feedback induce that sense of imposing in a student? Not to mention the fact that human feedback will be by necessity very delayed from the creation causing a disjuncture in the mental connections.

  4. Michael E. Lopez says:

    I had to think a lot about how I wanted to respond to this post. I actually rewrote this comment twice (that is, two substantial revisions; there were a few editing passes).

    On the one hand, anything that gets students into the habit of revisiting their written work is an almost unqualifiedly good thing. If school is about anything, it is about instilling socially acceptable values and useful habits. The habit of editing is beyond useful for anyone who will be doing written work as part of their career, and it’s just generally useful for participation in a society of autonomous agents.

    But.

    This sort of practice seems to undermine the purpose of rewriting. The purpose of rewriting isn’t to satisfy your teacher, or to get an “A” on the assignment. (As much as it might seem that way to a teenager.) Rewriting is about clarifying your thoughts, about figuring out how best to express the idea that is in your head. It’s about looking at a sentence you wrote with an objective eye and asking yourself if that’s REALLY what you meant to say.

    I don’t have a problem with a grammar or spellchecker — I use one myself because as good as I might be, I’m not perfect. (I don’t use one for blog posts, by the by.) And to the extent that something like the programs under discussion here server to get students to take their heads out of their @$$3$ and pay attention to the spellchecker, I’m all for it.

    But to the extent that we’re supposed to think that a program somehow gets students into the activity of *revision*, rather than grammar-editing, I tend to think that’s more fantasy than reality. Using software to review one’s writing *misses the point* of writing in the first place. If you’re going to use some sort of glorified spellchecker to fix grammar and punctuation errors… sure. That’s fine. Software — besides not having the faintest idea what a student is really trying to say — just isn’t yet up to the task of helping with any sort of substantive revision. It may be, someday, but it is going to take a major, major revolution in AI technology. I’d guess we’re at least 40 years away from such a thing. Probably more.

    E-Rater is a joke. The fact that a lot of people have a lot of money and effort invested in it doesn’t make it less of a joke. The following link isn’t that old.

    http://www.documentcloud.org/documents/346138-essay-awarded-a-top-grade-by-e-rater.html

    So should we turn to software just because students seem to like it better than teachers? That would, I think, be a terrible mistake. For at least three reasons.

    First, I suspect that students often have negative reactions to being told to do a rewrite because they don’t know how to do a rewrite. They do know how to respond to discrete suggestions from a limited AI, but they don’t know how to engage in a process of substantive editing. One of the things that I did a few times for college students that was *incredibly* well-received was to dedicate a few class sessions to the explicit instruction of how to do a rewrite of a paper. We started with proofreader’s marks — what symbols to use for which kinds of changes.

    Then we went through old drafts of my writing that had been subject to editing, sometimes sentence by sentence. We talked about what the goals of rewriting are supposed to be (HINT: They’re the same goals as the goals of writing in the first place!) and how rewriting and editing fit into the process of writing as a whole.

    Many of my students told me that they had never, ever been given any sort of instruction on how to rewrite something.

    Second, I don’t think it’s up for debate that some students just don’t take well to criticism. Some students break down at the sight of a sea of red ink splashed across their work. Some students — like some people — just aren’t cut out for the real world.

    Yet.

    But that’s why they’re *students*. It’s a teacher’s job to prepare them. Learning to deal with criticism, and to take criticism as an opportunity for self-improvement, is one of the most important things that a student does.

    Third, and finally, Using software to spur rewrites like this actually, I suspect, encourages a sort of facile, instrumental approach to writing. In other words, it takes all the worst aspects of the inert sort of learning that happens in many classrooms and magnifies it. Many of us complain of “teaching to the test”. But teaching students to “edit to the program’ is no better — and is in some ways much, much worse.

    This is hardly a comprehensive response to the issue, and I still feel like I’ve not quite expressed what I meant to say as clearly as I could have. But one of the other lessons I’ve given my students is that, at some point (usually around the due date for a paper) you have to let go. You have to let things go out into the world and then worry about amending your positions later.

    So that’s what I think.

  5. Roger Sweeny says:

    Michael, I agree completely but …

    1) As I recall, your students are college students, and not community college or remedial ones. If so, you can assume a base of competence that most high school teachers simply cannot. A “glorified spellchecker to fix grammar and punctuation errors” can be a very good thing.

    2) Some of the program’s advice is going to be bad. As cranberry says, shorter, simpler sentences aren’t always better. If the alternative were a teacher carefully going over each paper, making detailed comments, and forcing students to rewrite, the program is obviously inferior. But that’s not the case. Assuming a conservative 15 minutes of teacher time and 5 classes of 24 students each, it would take more than four 7-hour days just to read and comment on one batch of papers. And lots of students won’t read the comments or make the revisions.

    I was a terrible writer in eighth grade and failing English. So my father, who was in advertising, took it upon himself to teach me. Lots of, “What do you want to say?” “Does that sentence say what you want it to say?” “How do you think a reader will react to ______?” And, yes, I learned proofreader’s marks. It was one of the most difficult, unpleasant, and ultimately rewarding things I have ever done.

    It also took a tremendous amount of his time–and probably patience.

    I would love to see this done for every eighth grader. In fact, I would love for no one to graduate middle school who could not write a logical, grammatically correct one page essay. But it ain’t gonna happen.

    • This is the most important part of reviewing your writing: “How do you think a reader will react to ______?” No software program is going to help with that.

  6. Learning to write clear, grammatically correct prose takes a long time and much teacher input. I’m convinced that most kids who don’t get a solid foundation in k-5 will always struggle.(in all subjects) It’s far easier to learn the correct spelling/grammar from the beginning than to unlearn years of bad habits in HS. I am delighted that my oldest grandkids’ (incoming 3rd-graders) school has explicit grammar and spelling instruction every week, reinforced by in-class and st-home assignments.

  7. “Learning to write clear, grammatically correct prose takes a long time and much teacher input.”

    The problem with that statement is that implies increasing cost to provide the teacher input due to increasing cost of teachers. As the output of a teacher is relatively fixed, but salaries are increasing, the only possible outcome is it becoming costly to teach students how to write clear, grammatically correct prose. At some point, the cost will be more than the benefit. It already suffers from the defect that the costs are on society while the benefit accrues to the student.

    The robo-writing tutors could improve the more mundane aspects of the students writing, leaving the teacher input to the more value-added aspects. This would perhaps alter the cost/benefit to the students favor.

  8. I wish Annie Murphy Paul had qualified her praise of these robo-tutors. It’s one thing to say that they save time and expense–by giving rudimentary grammar and spelling corrections. It’s another to suggest that, because they’re less intimidating than humans, they are better than humans at helping students improve their writing.