When Is It No Longer An Implementation Error?

The New York Times reports that errors in the testing booklet of the Program for International Student Assessment has caused the Organization for Economic Cooperation and Development to invalidate all of the American test results. Apparently, the pagination of the test booklet was incorrect which made following the directions rather difficult. What is amazing is that apparently no one at the testing company or any of the "oversight" organizations caught the error.

The Times mentions that this is the first time there's been problems with US testing procedures on an international test, but does point out that "... a string of similar incidents brought flawed results on other standardized exams, including the SAT’s and the state math and reading tests used to judge schools under the federal No Child Left Behind Act."

And yes, the company is repaying $500,000 of the $2,000,000 that they charged the Federal Dept. of Education for the implementation of this test... but that's really a drop in the bucket since this is but one of thirteen federal contracts that RTI International holds. And this is a pretty small contract, since the total value of all their active contracts totals $196,000,000.

So here's my issue, as we move further and further down this road of high-stakes tests, we really have to question all of the mistakes we are seeing made on these tests. Because there is starting to be a fair amount of evidence to suggest that it's harder than we think to accurately mass-assess kids using these tests. And given that these tests affect kids' lives, after teachers' and administrators' jobs and lives, this isn't something we can afford to get right most of the time.

Clearly the outrageous growth in the testing industry (anyone else have a problem with that phrase?) is causing a lot of problems:

"We need to recognize that the testing industry is under immense pressure at a time when scores are being given immense importance," said Thomas Toch, who wrote a report last year detailing the problems of the American testing industry for Education Sector, an independent policy group, where he is a co-director.

There's no question that the testing industry is under a lot of pressure right now, but really, it's not the pressure they are under that I worry about. What of the kids who took that test and halfway through it, realized that the test was wrong but couldn't do anything about it? What of the kids who, a few years ago, took the NY State Physics Regents and then had to sweat out their high school graduation after they bombed it, waiting until the state admitted that they screwed up the test? We cannot create high-stakes assessments that are, more and more, showing themselves to have serious structural flaws when we try to place too much value on them.