Measuring 'soft skills'

How do we tell if students are learning “soft skills,”  such as “the ability to work with others, process information from disparate sources, communicate persuasively, or work reliably?”  On Taking Note, John Merrow and Arnold Packer look at the challenge of creating valid, reliable assessments.

With a Kellogg Foundation grant, they’re asking mentors at 28 community-based organizations to assess high and middle school students on “responsibility, work ethic, collaboration, communication, problem-solving, critical thinking, and creativity.”

We ask the mentors to write a two -sentence description of the context in which each of the traits was demonstrated.  Was the teenager responsible about picking up trash in the park or helping out on the surgical ward?  Communicating to a friend about the homework assignment requires a different skill level than communicating about obesity to a large community audience.  There is no reasonable rubric that will cover this amount of variation.

Finally, mentors also grade the students’ performance on a scale of one (“cannot do it”) to five (“does it well enough teach others”).

This produces a Verified Resume of performance that could be used for job applications and college admissions.

The pilot project will survey employers to see if  the mentors’ evaluations match the new hire’s performance.

We got into this because we believe performance traits like responsibility, tolerance for diversity, ability to communicate and work ethic matter.. Because they matter, we must also figure out how to measure them reliably.

Recommendations are supposed to fill this purpose, but it’s difficult to judge whether the recommenders are setting the bar high or low. And many people are afraid to be honest in recommendations.

About Joanne

Comments

  1. Forget mentors – they have every reason to sugar-coat the truth.

    Why not ask those teachers in classes where lack of these soft skills strongly impacts the grade – physical education and science? In those classes, students need to be able to work together well – without arguing, and for the benefit of a group effort. We see the students who won’t pull their weight, but want the same grade as those who worked. We see the disruptive student, and their effect on the functioning of the group.

    Ultimately, the grade in those classes is part (40-50%) of the level of soft skills that student demonstrates. The student who is lacking, does very poorly in lab or on the team.

  2. I wonder how such a measure, however designed, will affect autistic children (and those with other diagnoses that affect social behavior). My son is autistic, though not diagnosed until his mid-30s, and he has (now, not so much in elementary school) excellent soft skills, they’re just not exactly the same as those neurotypicals have. Would he have been shut out of learning the intellectual skills he uses now in his career because he didn’t, and couldn’t, conform to social norms for first graders?

  3. georgelarson says:

    If schools want to teach teamwork why not expand or require participation in team athletics, theater groups, musical ensembles, 4H , JROTC or similar groups? I do not agree that science teaches teamwork. Many of the great scientists of history were extremely solitary thinkers. The phrase “too great for a hypocrite’s humility” comes to mind. I agree that mentors will probably not be honest judges of their student’s soft skills. I do not see how these skills can be measured for a future employer by a grade on a transcript. This is a need that can only be met by honest letters of recommendation or an honest resume. I think this is unlikely to come from a school system.

  4. These things cannot be measured in a valid or reliable fashion. That is what a job is supposed to do. In the old days when we had common sense, that is the way it was done and it worked quite well, as I recall.

  5. I am adamantly opposed to requiring participation in any non-academic endeavor, including community service. If kids want to do extracurriculars of any kind; fine. If kids choose to do regular fitness activities (sports, dance etc), I think they should be able to use those to substitute for school PE, which I don’t think belongs in school in the first place. Have recess and leave it at that.

    Schools should stick to academics. Period. It isn’t as if they are doing such a wonderful job on academics that they need any distractions. I agree with the above comments that teamwork is not inherent to science and have also observed that all this teamwork/groupwork/touchy-feely stuff is very hard on kids who are shy, quiet and/or value their privacy, let alone those who are not neurotypical. It’s one – more – aspect of school that favors the teacher-pleasers; most often they are girls.

  6. Cranberry says:

    “We got into this because we believe performance traits like responsibility, tolerance for diversity, ability to communicate and work ethic matter.. Because they matter, we must also figure out how to measure them reliably.”

    What does tolerance for diversity mean? How do you recognize it in a teenager?

    None of this “Verified Resume” approach addresses the main problem with recommendations today. It does nothing to address the fear of being sued for a bad, or honest, recommendation. “Tolerance for diversity” also comes perilously close to the use of “dispositions” in teacher training (see: http://www.thefire.org/article/7826.html).

    I agree with LindaS, as well. This sort of “resume” seems designed to knock out the sort of students who are either on the autistic spectrum, or are honest.

  7. Here’s why one writes the classic “damning with faint praise” rec letter:
    http://blogs.riverfronttimes.com/dailyrft/2010/07/careful_who_youre_calling_a_je.php

    (Article talks about a girl who sued her guidance counselor for rating her below average on what sounds like the common application — in the category of academic honesty a bad rec is almost always due to cheating/plagiarism in the school record.)

    Personally, as a mentor, I wouldn’t touch this with a 10-foot stinky crocodile.

  8. Richard Aubrey says:

    “what does tolerance for diversity mean”
    You sure you want to know?

  9. Michael E. Lopez says:

    I imagine that “tolerance for diversity” in this context means something like, “Doesn’t sit and petulantly refuse to work with Jews and Blacks, and doesn’t gratuitously insult people who are different in some value-neutral way merely because of that difference.” It’s a motivational take on “plays well with others” that focuses on WHY a person might not play well with others.

    Contra anon’s assertion, I don’t think that measuring things like this is hard at all. We do it all the time and our measurements are valid and reliable: I know which of my students have these skills and don’t have these skills. If I say that Student M is more responsible than Student N, it’s a pretty fair bet that *everyone* not personally involved in the lives of one of those students (and thus biased) will agree that M>N on the responsibility scale.

    What’s hard is finding a transferrable measurement for something that is innately subjective. Take the classic doctor’s question: “On a scale of 1-10, how much pain are you in?” That number tells you everything about how much pain the patient thinks that they are in. But there’s absolutely no reason to think that if Patient A says “6″ and Patient B says “6″ that they are experiencing anything even remotely similar. Likewise, these sorts of “soft skills” are subjective measurements — just of other people rather than the self. I might say that Student M is a “7″ on the responsibility scale, and Student N is a “5″. Someone else might rate them as “9″ and “8″, respectively. We couldn’t even use percentages (as many LoR forms do) because, frankly, different evaluators deal with different populations.

    If you think about it, though, academic grades face the *exact same problem*. I might think a paper deserves a C+, but my colleague down the hall thinks it deserves a B. We get around this, though, in terms of academics, by taking context into account. Top 10% at Lowell HS in San Francisco means something entirely different than Top 10% at Locke HS in Los Angeles. The Locke Valedictorian might not even be in the top 70% at Lowell in terms of academic achievement, and that’s simply a reality of the school environment.

    The difference, of course, is that we have a lot of practice in dealing with academic grades, and we’ve got a thick institutional framework to help us sort through the data. So all it probably would take is a little bit of effort to start developing opinions about the various mentors and/or mentor programs. Do some programs “grade” harshly? Are some good and some bad? Certainly this will be the case. Experience alone will allow us to tell the difference.

Trackbacks

  1. [...] This post was mentioned on Twitter by kriley19 and Joanne Jacobs, Monte Tatom. Monte Tatom said: Measuring ‘soft skills’ http://tinyurl.com/26dd5v4 #fhuedu501 #fhuqep [...]