Automated essay grading leads to more writing

My standard advice for learning how to write can be boiled down to six words: Read a lot. Write a lot. If brevity is essential, three words are enough: Write a lot. I can even make do with one word: Write.

So I’m sympathetic to the argument that students will write better if they write more, with feedback on their efforts. But teachers don’t have the time to read and respond to every draft of every paper.

Automated essay scoring lets teachers assign more writing and focus their own time on “higher order feedback,” argues Tom Vander Ark on Getting Smart. In response to an attack on scoring engines in the New York Times, Vander Ark summarizes and links to the case for automation.

Measurement is a friend to creativity, he writes in another post.

The online scoring engines use the same rubrics to score essays as human graders.  Any ‘standardization’ of writing is not a function of the method of scoring but the nature of the prompt, i.e., if a state requires every 8th grader to write a five paragraph essay every year it may lead to formulaic teaching—that’s a teaching issue driven by a testing issue, not a scoring issue.

People are sick of standardized tests “because most states are using old psychometric technology to administer inexpensive tests with little real performance assessment.”

. . . we’ve been using these tests for more than they were designed for—to hold schools accountable, to manage student matriculation, to evaluate teachers, and to improve instruction.But remember the state of the sector in the early 90s before state tests were widely used. There was no data, chronic failure was accepted, and the achievement gap was largely unrecognized. Measurement is key to improvement.

Soon, “essay graders will soon be incorporated into word processors and will be used as commonly as spell-check,” Vander Ark predicts. Students will get more assessment to help them improve.

Update: Machines Shouldn’t Grade Student Writing — Yet, writes Dana Goldstein on Slate.

About Joanne

Comments

  1. >My standard advice for learning how to write can be boiled down to six words: Read a lot. Write a lot. If brevity is essential, three words are enough: Write a lot. I can even make do with one word: Write.

    So you start by saying two things are essential, and then boil it down to just one of those.

    There’s no evidence that reading a lot makes better writers. I agree that writing is important to become a good writer, but that’s quite different from saying an English teacher (who is absolutely not a guaranteed arbiter of good writing) is essential to the writing process.

    And automation being a good way to get feedback? Good lord.

    • Reading a lot doesn’t make better writers? Any other skills or areas of knowledge in which studying the greats is of no value?

      And as for the value of automated essay grading perhaps you could go into more detail then an appeal to a deity.

    • Artists observe other artists in addition to performing. One can learn method and technique.

      Athletes observe other athletes in addition to playing. One can learn method and technique.

      When it comes to writing, reading is the observational analogue. It is reasonable to think the parallel holds.

      Unless one can show the results of a study that contradictions this reasonably intuitive hypothesis, I suspect most people will find it silly to dismiss the role of reading in becoming a better writer.

  2. The interesting research to me will determine what’s the optimal percent of essays for a teacher to grade optimally. 5% seems too low to encourage students to write with intelligent content, but 35% seems too much. More writing – some randomly machine graded, some not.

  3. Lightly Seasoned says:

    Seems like a more sensible approach would be to simply reduce the student loads of English teachers so we can assign more writing. Some schools (not mine, unfortunately) give the AP English teachers an extra plan for that purpose.

  4. Great, make teaching even more expensive. No, thanks.

    Better to hire cheap adjuncts to do the reading. Colleges have a ton of them.

    • Lightly Seasoned says:

      If Linda is assigning more writing, she is doing the same amount of work. At this point in my career (and I am sure Linda’s), the grading takes far more time than lesson prep. (Except, of course, in years when everything has to get re-aligned to new standards.)

  5. Whooops–I meant make “education” more expensive. That is, give Linda an easier life and less work for the same money, making her very happy whilst not doing anything useful other than allow us to pretend that we can get everyone to write.

  6. Michael E. Lopez says:

    Let me tell you a story. This is just my particular view of this story. There are others that are not so critical. But I think my story doesn’t just have “truthiness”, but is also illuminating for the current conversation.

    Once upon a time, there was a martial art. This martial art was called fencing. This martial art was training for how to use a light, one-handed sword (and perhaps a second weapon) against an opponent who was trying to kill you. This martial art developed many principles which were useful for helping people to defeat their opponents without dying — including a general rule that you shouldn’t launch an attack at someone whose sword was already extended and pointed at your chest without doing something about that sword first. The people who studied this martial art were great warriors, adept at killing and wounding people with swords without dying themselves.

    Once upon a time, there was a sport. This sport was called fencing, and in fencing, the object of the “game” was to score touches against your opponent with a sword. It had adopted many of the principals of the martial art that was called fencing. There was an amazing correlation between the skill level of martial arts fencers and the new sports fencers — so much so that, except for the codification of the rules and the introduction of “points”, it was thought to be the same activity.

    Over time, though, the best fencers — who would have been truly formidable warriors in an earlier age — turned their attention to developing new and better ways of scoring points and taking advantage of right-of-way rules. It became important to put your arm out first, to establish your right-of-way. It became important to make sure you hit your opponent somewhere where the judges would see it. The competitors were no less skilled than they had been in years past, but their skill was turning towards a different goal.

    Then one day, someone said, “We can get a better sense of when the sword is touching the opponent by using this electric box.” There was an amazing correlation between the touches called by judges, and the touches noted by the Box. The correlation was so close, that people thought it was the same activity, just done better. And so it was that fencing the sport became fencing the modern sport, with wires and lights and buzzers, and fencers learned that what was important was making the light go off.

    And now this is the acme of fencing skill, by a world-class fencer:

    http://timmorehouse.wordpress.com/2012/04/20/cool-fencing-video-of-the-day-amazing-behind-the-back-touch-by-miles-chamley-watson/

    It’s an amazing move, showing incredible athleticism, timing, and situational understanding. It’s also guaranteed to get you killed in a swordfight.

    Now, there is every possibility that Mr. Chamley-Watson would be a formidable opponent in an actual swordfight, though almost certainly not nearly as great a warrior as he is an athlete in the sport of modern Fencing. On the other hand, he may balk at the presence of a naked blade. His incredible panache, flair, and skill, might immediately fall into fear and over-caution. He may scratch across his opponent’s shoulder, capturing the right of way, only to have his lung pierced all the way through with a counter-attack that would never set of the light of the Box (because it fell after his scratch) nor score a point (because it didn’t have right of way). One thing is certain: all the effort he put in to mastering the behind-the-back touch would be wasted; he’d never try it.

    We cannot know the answer to that question, because Mr. Chamley-Watson is not training for, and has not trained for, a swordfight, despite the fact that he is engaging in what everyone has assumed all along is the same activity.

    **********************************************

    Once upon a time there was an activity called writing…

    • SuperSub says:

      My college’s Phys ed department was so awesome that it offered fencing courses, both competition style and classical. This is exactly why I chose the classical. The instructor looked just as I’d imagine Porthos… we spent a lot of time spearing falling gloves against a wall. He really seemed like the Mr Miyagi of fencing.
      Best thing I ever saw was him doing a 3 on 1 duel with another master and two older students. There was no boundaries or rules… just a lot of awesomeness.

  7. Cranberry says:

    Tom Vander Ark is a salesman. Keep that in mind. In no way is he functioning as a disinterested party. Tom is also CEO of Open Education Solutions and a partner in Learn Capital, a venture capital firm investing in learning content, platforms, and services with the goal of transforming educational engagement, access, and effectiveness.

    Writing frequently without feedback may improve one’s handwriting, but to improve at the technical task of academic writing, one needs feedback from a reader who can understand the arguments made. A stringent teacher who sets high standards will improve a willing student’s writing over time.

    Grading is subjective. If someone guarantees the machine grades as well as human graders, the proper response should be, “the grading criteria were too simple.” A collection of teachers will give the same work different grades. There can be no comparison on subjective judgements. See: http://www.jstor.org/stable/1076706?seq=6.

    Les Perelman’s points are spot-on. Writing long sentences, using fancy conjunctions, and using multisyllabic words whenever possible may lead to a better grade. It doesn’t necessarily lead to better writing.

    The five paragraph essay is not the high point of written expression. If students are accustomed to writing longer, more complex essays, it’s relatively easy to prepare a five paragraph essay for a standardized test. I don’t think it’s easy to transition from the simplistic five paragraph essay to a 15 or 20 page paper. I worry that the state tests will (do?) force schools to focus on a limited writing exercise.

  8. Lightly Seasoned says:

    The New York Times published an essay an e-grader gave a top score to:

    In today’ssociety, college is ambiguous. We need it to live, but we also need it to love.Moreover, without college most of the world’s learning would be egregious.College, however, has myriad costs. One of the most important issues facing theworld is how to reduce college costs. Some have argued that college costs aredue to the luxuries students now expect. Others have argued that the costs area result of athletics. In reality, high college costs are the result ofexcessive pay for teaching assistants.

    I live in aluxury dorm. In reality, it costs no more than rat infested rooms at a MotelSix. The best minds of my generation were destroyed by madness, starvinghysterical naked, and publishing obscene odes on the windows of the skull.Luxury dorms pay for themselves because they generate thousand and thousands ofdollars of revenue. In the Middle Ages, the University of Paris grew because itprovided comfortable accommodations for each of its students, large rooms withservants and legs of mutton. Although they are expensive, these rooms arenecessary to learning. The second reason for the five-paragraph theme is thatit makes you focus on a single topic. Some people start writing on the usualtopic, like TV commercials, and they wind up all over the place, talking aboutwhere TV came from or capitalism or health foods or whatever. But with onlyfive paragraphs and one topic you’re not tempted to get beyond your originalidea, like commercials are a good source of information about products. Yougive your three examples, and zap! you’re done. This is another way thefive-paragraph theme keeps you from thinking too much.