Software meltdown

Educational software has “no significant impact” on student performance, according to a study by the U.S. Department of Education, reports the Washington Post. The study evaluated 15 reading and math programs.

It is the largest study that has compared students who received the technology with those who did not, as measured by their scores on standardized tests. There were no statistically significant differences between students who used software and those who did not.

Software industry people blamed poor implementation of their products and the common tendency to skimp on teacher training.

About Joanne


  1. I would have to agree with the software industry in this case, as this is exactly what my district has done. Lots of money spent, very few people trained on how to use it, and no inservice time offered for others to be trained on it.

  2. I tend to be suspicious of non-person-oriented teaching (software, videos, distance learning) anyway.

    But if you’re gonna buy it, for the love of teaching, spend the money to train people how to use it! And give the teachers adequate time to learn the software! I’ve been in situations where people were using a software package to teach that they really weren’t up to speed on, and it was frustrating all around.

  3. I just read the study. The zero effect was reported as an aggregate score. Individual programs (especially in 6th grade math) did quite well.

    Technology by and in of itself may not have much use, but certain products do.

    I also noted some districts did quite well with the products. I would guess that this is because those districts received the proper training.

    Net knowledge gained from study… nilch.

  4. Walter E. Wallis says:

    How much time to train a child to play a Wii game?
    My grandson and his buddies play some very complex games with no tutilage from me or any other adult. Perhaps game makers should write educational software and educators guide gamemakers in educational directions. I understand the Army has done quite well with gaming software. Check Ender’s Game for an example.
    Happy Easter.

  5. wayne martin says:

    The Washington Post article was not particularly helpful, so here is a link to the study–

    Effectiveness of Reading and Mathematics Software Products:

    Given the high cost of teachers, and the never-ending demand for smaller classes, as well as the demand for a wider array of topics delivered in the classroom, having educational software and on-line delivery of education is the only way that a technological society will be able to provide educate its children.

  6. Frank Zavisca says:

    I would question the districts that said the software helped. Likely motivated parents, teachers, and students would do well with or without computers.

    Why would some districts NOT say the scores improved? They have to justify all the money spent.

  7. wayne martin says:

    > Why would some districts NOT say the scores improved?

    NCLB, as well as pre-existing standards-based testing, provides the data to make performance improvement assessments. This data would be managed (and available from) by the Department of Education for each State.

  8. Why would some districts NOT say the scores improved?

    Wayne got there ahead of me but NCLB is one good answer. Notice the contrast to pre-NCLB days. The question then would have been “Why would any district bother to improve scores?”

    They have to justify all the money spent.

    That would certainly be novel.

    It’s also interesting to observe the implicit assumption that poor implementation consists of inadequate teacher training. The use of technology in other venues is to reduce personnel requirements, improving the productivity of the employees that technology can’t replace. But not in public education.

    Like every other socialist organization, in public education the political subordinates all other considerations…

    Will the use of technology improve education? That’d be nice.

    Will the use of technology reduce the need for personnel? That’s just crazy talk!

    …which gives a good measure of what’s important and what isn’t in public education.

  9. I really hate that there’s no “preview” button.

  10. GradSchoolMom says:

    Might there be something wrong with how educational research is conducted? Perhaps they are too biased to look for answers that differ from their current practices? I’m currently pursuing a Masters in Instructional Technology and the research has really confused me. My background is in advertising. Educational multimedia research has basically shown that advertising should not work but we all know it has been a powerful influence. I’m about to present at the eLearning Guild Annual Conference on how to choose graphics for eLearning and the research says that adding details that are emotional or more complicated than necessary will actually hurt the learning. That leaves educational eLearning still using charts, graphs and line drawings. I will be using the example of John who in 1975 learned the parts of a cell but can not remember them today. In 1975, he also learned the parts of a Big Mac and today he can still rattle off “Two all-beef patties, special sauce, lettuce, cheese, pickles, onions on a sesame seed bun.” I believe that somehow we are missing the educational power that can be found in multimedia technology and games.

  11. wayne martin says:

    I developed productivity software in the early days of the Microcomputer industry (spreadsheet and word-processor software for CP/M, DOS 1.0, PC Ver.1.0, etc). At the time, there were many, many “zones of resistance” to PCs in the workplace. A number of studies were done after five years, or so, that suggested that PCs were of no value because the “bottom line” of the industries studied did not increase in measurable ways, so the introduction of PCs/Software could not be supported as “productive” for the workplace. After a bit, a round of rebuttal papers appeared which pointed out that the economists that conducted the first studies were “smokestack” economists, who had the wrong tools (or vision) for measuring the value of PCs. These studies pointed out that issues like product quality generally were not measured in a pre-PC environment, so any increase in product quality, or customer service, or information sharing about products would be invisible to a pre-PC point-of-view.

    It wasn’t too much later that the World Wide Web (WWW) was created, and by 1993 (a little over 15 years after the introduction of the first microcomputers around 1976) the PC has established its worth in business, and the WEB introduced the 2nd generation of value-added functionality that PCs brought with them. Industrial control, which is also PC-based, didn’t seem to be studied in the same light that office PCs were, but have had incredible impact on manufacturing environments during this same period of time.

    Schools were the hardest of the sectors to introduce PC hardware/software in the early ‘80s. Public schools were almost impossible. Teaching professionals in universities/colleges who were over 50 were almost not open to converting from typewriters to PC-based word-processors. Over time, this group retired, and the younger teachers who were not so blind to the possibilities of change took over and the world is a different place today—only twenty-five years later.

    As to education-based studies about PC software, these need to be read suspiciously I fear. My experience in the business sector was that there were no metrics in place to measure productivity prior to PCs, and it took almost a decade before people began to see how to actually see productivity improvements that were measurable. Once the “low hanging fruit” of these sorts of studies has been identified, subsequent studies become convoluted in “theory” or introspection that might not be relevant to the process as a whole.

    For instance—many schools complain about not having books enough for every student. E-books/On-line distribution of materials provides a way for every student (with access to a PC) to have access to the materials required by the course. How many current education delivery studies include “access to required materials” as a variable in their assessment of a given study situation? And then there matters like “read all of the assigned material”. In a paper-based environment, there would be no way to evaluate this sort of thing, other than by asking each student—who may, or may not, tell you the truth. In a well-designed on-line system, counters for each page of required material could be established, so that teachers (and researchers) would be able to determine the student’s engagement with the classroom assignments.

    There might be a cottage industry out there for a small group of people to study these studies and help schools understand which ones are good, and which ones and not very well designed and executed.

  12. In response to GradSchoolMom, my father was in the advertising business. Some famous ad man once said: “Half of all money spent on advertising is wasted. I just wish I knew which half.”

    A lot of money and talent goes into designing ads because it’s so expensive to buy the media time and/or space. Despite that, plenty of ad campaigns flop. If the advertiser doesn’t see results, the ad campaign is changed — and often the ad company is dumped.

    I do remember early studies showing that introducing technology doesn’t improve workplace productivity. As Wayne writes, that changed over time as technology changed and researchers learned what to measure. I think some uses of technology — quick-feedback tests, for example — will prove very effective in supporting teaching. Some teaching programs may pan out too, though I suspect only the kids who are capable of teaching themselves from books will prove able to teach themselves via computer and books.

  13. Anyone care to hazard a guess as to the common rationale behind the purchase of educational software by a school district?

    Is it because the software will increase the number of kids marching out of the school just stuffed to bursting with critical thinking skills?

    Will the software increase teacher efficacy enough to preclude the hiring of additional staff? The laying-off of superfluous teachers? A rapid rise in AYP?

    Just why do school districts purchase education software?

  14. wayne martin says:

    > Just why do school districts purchase education software?

    Seems like one of those “trick” questions, to me.

    So, why do school districts purchase education software, anyway?

  15. Not a trick question. You provided the rationale for the purchase of software in business by calling it, properly, productivity software. Purchase of the software meant more work could be done with fewer people, i.e. productivity goes up, profits go up, cost of the product goes down, consumer price goes down and a “Wayne Martin” dividend is declared for being the smart guy who brought all that productivity to the organization resulting in the CEO getting a “Fortune” cover.

    Now, what’s the analogous, and similarly compelling, reason for a school board to buy education software? Productivity? Defined how? Improved quality? Again, define “how” with the addition of “measured how”?

    To teach the kids about computers? Come on, the kids probably know more about most computer-related topics then the teachers.

    More then likely they’ve downloaded plenty of music, located via search engines using p2p, distributed-index, torrent-based software and transferred it via USB to their DMP (digital music player for the acronymically challenged). They’ve got a MyFace account and with their bar mitzvah money they bought a camcorder which they hook up via firewire to their laptop to FTP, or stream, videos to their MyFace account. And, they know what all of the foregoing means, in detail.

    So? Where’s that leave us in terms of the reason school boards purchase education software?

    About nowhere I’d say but I could be wrong. Just give me a decent reason to think so.

  16. Wayne Martin says:

    > To teach the kids about computers? Come on,
    > the kids probably know more about most
    > computer-related topics then the teachers.

    Well .. to some extent that’s true.. but the problem is that the teachers don’t know nearly enough. With the integration of the Internet into the fabric of the information/data delivery systems of the “real world”, where are students going to learn about the electronic infrastructure of the emerging “new world”? Certainly parents have a role here, but given the low marks that parents get for parenting, it’s not likely that they are the best source of this information.

    It seems to me that information technology needs to be added to the curricula that public schools offer, just like science and math. The Internet is not that complicated to understand, although it by no means is the best design of how networks work. It seems a bad idea to wait until the second or third year of undergraduate school to begin to teach people what a “router” is.

    We don’t teach kids about cars, or architecture, or finance, or government in public schools at the moment. Maybe it’s time to realize that whatever we are teaching them isn’t very interesting to them and not helping them much once they leave the public school system.

    So? Where’s that leave us in terms of the reason school
    boards purchase education software?

    I listen to a Washington, DC radio station over the Internet a lot. Last week there was a news report about problems that nearby Loudoun County was having hiring teachers, stating that they were looking outside the county, and even outside the country for teachers to satisfy their needs:

    Certainly being able to purchase software, or instruction packages (on-line, distance learning) would provide districts with options for the incredibly costly task for education delivery. While there may be effective resistance for another decade or two about computer-based education from teachers’ unions and “old-school” teaching professionals, it’s clear from the results that CBE (Combuter-based Education) works, is growing in application, and will become more effective over time.

    School Districts have a vested interest in providing uniformity in the education product. Having variances in test scores which can be attributed to failures in teaching staff is simply not acceptable. Parents are not uniformly uninformed about quality issue, or innumerate as so many education professionals seem to be.

    Many teachers complain about putting in 55 hours a week (or more), yet never provide time cards, or activity reports for outside audit in order to determine what these teachers are doing with their time. Having on-line grade books, with access to each student’s weekly performance by parents, and alerts for notification by email or telephone contact removes a clear burden from teachers to monitor all of her/his students for performance dips. Certainly teachers will not be relieved totally from this aspect of their job, but having the computer flag those students likely to fail via an email to the teacher and parent will make their lives a lot easier, time wise. Teachers are always asking for more money, claiming that they are tasked with off-campus work. To some extent, this is true. But the more the school districts can do to reduce these off-campus burdens will have a payback in terms of the school district’s not being so easily bludgeoned by work-related claims of unions which previously could not be proven, one way or the other.

    As mentioned many times, instructional materials run a certain percentage of the cost-of-education of each student. Having instructional materials on-line increases access, and reduces cost. Moreover, print-on-demand technologies allow schools to use both on-line and printed materials at reduced costs from purchasing directing from a text-book vendor.

    For rural districts, the lack of teaching personnel clearly reduces the options for students. Having access to on-line teaching sources, as well as CBE-software increases the options for students which could never be fulfilled by live teachers, who are not willing to live in remote areas. Over time, it’s quite possible that rural schools could provide a educational experience that is more-or-less equal to any top-flight urban school system.

    Of course, one needs to construct a “big picture” and then create a set of metrics that provide educational and financial impacts, in a before-and-after framework. This is not impossible to do, but no small school district would ever be able perform this analysis adequately. This is the sort of project that the US DoE should sign up for, and once completed, provide to all the school districts in the US—at the executable and source level. Why the US DoE hasn’t seen fit to do something like this is an open question, but it’s clear that education professionals are not “systems people”, and are not likely to go down such a path on their own. Perhaps Congress, driven by demands from parents and voters, might one day task them to provide this tool to US schools.

    I could go on, but this should be enough to answer the “Why” question posed earlier.


  1. […] used, writes Gregg Downey of eSchool News. A federal study that found students who used software didn’t learn more than a control group is “half-baked,” Downey argues. At one point, the study itself […]

  2. […] week or so ago we had the shocking revelation that education software doesn’t help. Now we learn that giving students laptops doesn’t help, […]