Home > Uncategorized > VAM Flaws Glossed Over by ASCD

VAM Flaws Glossed Over by ASCD

November 29, 2012

The most recent ASCD online Journal is devoted to teacher evaluation and it features some articles on Value Added Measures, including one by McCREL consultants Bryan Goodwin and Kirsten Miller titled “Use Caution with Value Added Measures”. The article does a good job of delineating everything that is wrong with Value Added Measures, but— despite the fact they offer no antidote for any of the flaws— implicitly conclude that value added measures are here to stay, can be improved, and are better than what we have in place now.

They lead their list of VAM’s problems with this paragraph:

In many ways, the value-added teacher measurement model is still in its infancy, having emerged only in recent years as sophisticated data warehouses made it possible to measure the average growth of an entire class of students over the course of a school year. However, researchers have warned that what seems so simple and straightforward in theory is incredibly complicated in practice. Here are a few of the pitfalls.

What the paragraph fails to mention is that nearly all education researchers think that VAM is junk science– though they might state that assertion in more erudite terms. It also misrepresents VAM’s capabilities: VAM cannot  be used to “to measure the average growth of an entire class of students over the course of a school year” for reasons that are described in the list of pitfalls. One of the pitfalls is particularly comprehensive:

Data may be inaccurate. (After newspaper reports undercut the teacher ratings) multiple factual errors surfaced in New York’s data. For example, one teacher had data for a year when she was on maternity leave; another teacher taught 4th grade for five years but had no data (Clawson, 2012). Moreover, small samples—for example, classes with only 10 students—can paint inaccurate pictures of teachers because they are subject to statistical fluctuations (Goe, Bell, & Little, 2008).

Despite the fact that “data may be inaccurate”, the federal government is championing the use of this approach and the teachers whose ratings were published using this inaccurate data have no way of clearing their names after they’ve been published— as has occurred in NYC and LA. The computer aphorism “garbage in, garbage out” comes to mind!

The last section of the article poses the question “Still Better Than the Alternatives?” and appears to answer in the affirmative.

In general, the year-to-year correlation between value-added scores lies in the .30 to .40 range (Goldhaber & Hansen, 2010). Although this correlation is not large, researchers at the Brookings Institution note that it is almost identical to the correlation between SAT scores and college grade point average (.35); yet we continue to use SAT scores in making decisions about college admissions “because even though the prediction of success from SAT/ACT scores is modest, it is among the strongest available predictors” (Glazerman et al., 2010, p. 7).

So at the same time that elite colleges and universities are questioning the efficacy of SATs as a predictive metric, schools are being asked to embrace it because while it is a weak  predictor of success it is– what?— a predictor that is numeric? That seems to be the argument based on the logic presented in a subsequent paragraph:

…in general, principals appear to be fairly accurate in identifying top and bottom performers, but they struggle to differentiate among teachers in the middle

And what is the purpose of differentiating “among teachers in the middle?” A workmanlike teacher is as important to the operation of the school as a workmanlike employee is in any organization and assigning a numeric value to the ratings makes them appear more scientific and exacting than they are in reality.

The irony of this VAM article that implicitly advocates the use of evaluation to sort and rank order teachers is that it appears in the same newsletter with articles that describe the benefits using evaluations to identify meaningful staff development that will result in the improvement of  the performance of ALL teachers. VAM isn’t designed for any purpose except naming and shaming. It should be abandoned.

%d bloggers like this: