Search Results

Keyword: ‘VAM’

AFT Kills VAM in Houston! Major Victory for Common Sense!

May 5, 2017 Comments off

It is a sad day when the courts are the only means of ensuring that common sense prevails. This reality makes this “victory” bittersweet at best.

Source: AFT Kills VAM in Houston! Major Victory for Common Sense!

Categories: Uncategorized

The Mathbabe Pushes Back Against VAM Critics— But Overturning VAM will be a Daunting Challenge

October 9, 2016 Comments off

In a blog post a few weeks ago that Diane Ravitch linked to yesterday Cathy O’Neill (a.k.a the Mathbabe) offered some counterarguments to critics who pushed back when she slammed VAM (Value Added Model) in her recent book Weapons of Math Destruction”. As one who was seeking a way to make use of the test scores that are generated due to the NCLB mandates that emerged in the early 2000s, I was drawn to the ideas that William Sanders proposed regarding “value added” testing. But I quickly saw that the rigorous methods he initially advocated were being oversimplified and in virtually all cases the tests that many “reformers” wanted to use to measure “value added” were NOT designed for that purpose. Moreover, as statisticians like Ms. O’Neill noted, VAM was a wrongheaded approach to begin with. Nevertheless, despite all the flaws in VAM, it gained traction among politicians who saw it as a means of “weeding out” bad teachers and saw the critics of VAM as either union apologists or etherial intellectuals. Consequently, when President Obama was elected and passed an overly modest stimulus package for public education, he used VAM as the centerpiece of his Race to the Top (RTTT) grant program, effectively requiring that it be used as the basis for teacher evaluations in order for States to receive any of the funding. The two States I was working in at the time, NH and VT, were among the last to seek RTTT funds, in large measure because the leadership in the State got pushback from either State Boards or Superintendents.

In her recent post, Ms. O’Neill responds to one of the frequent rebuttals she’s received as a result of her criticism of VAM, with my emphasis added:

Here’s an example of an argument I’ve seen consistently when it comes to the defense of the teacher value-added model (VAM) scores… Namely, that the teacher’s VAM scores were “one of many considerations” taken to establish an overall teacher’s score. The use of something that is unfair is less unfair, in other words, if you also use other things which balance it out and are fair.

Ms. O’Neill makes one clearly straightforward logical rebuttal to this “one of many considerations” argument, with my emphasis added:

The obvious irony of the “one of many” argument is, besides the mathematical one I will make below, that the VAM was supposed to actually have a real effect on teachers assessments, and that effect was meant to be valuable and objective. So any argument about it which basically implies that it’s okay to use it because it has very little power seems odd and self-defeating.

While the use of the “one of many” argument IS “odd and self-defeating”, it is also an argument that has intuitive appeal and one that would enable the use of a “valuable and objective” tool that is also— conveniently— cheap, easy, and seemingly exacting. But what if the exactitude is pointless and meaningless? As Ms. O”Neill notes, when everything else that constitutes a teacher evaluation yields very little variance, as is the case in teacher evaluations, the pointless and meaningless but exact measures can ultimately be the determining factor.

The VAM was brought in precisely to introduce variance to the overall mix. You introduce numeric VAM scores so that there’s more “spread” between teachers, so you can rank them and you’ll be sure to get teachers at the bottom.

But if those VAM scores are actually meaningless, or at least extremely noisy, then what you have is “spread” without accuracy. And it doesn’t help to mix in the other scores.

In a statistical sense, even if you allow 50% or more of a given teacher’s score to consist of non-VAM information, the VAM score will still dominate the variance of a teacher’s score. Which is to say, the VAM score will comprise much more than 50% of the information that goes into the score.

In the end, I have to believe that some statistician at the USDOE knew this whole concept was flawed but supported it anyway because VAM is easy to implement, relatively inexpensive, and intuitively appealing. The shame is that once a concept like this takes hold, correcting it is extremely difficult as is replacing it with something new. And with ESSA now in place, it will require a change of heart in 50 State capitols since virtually every state in the union embraced the VAM precepts when they accepted the RTTT funds. The “Weapon of Math Destruction” will be the Obama-Duncan legacy….

NY Court Slams “Arbitrary and Capricious” VAM…

May 11, 2016 Comments off

In a victory for common sense over spreadsheets, Justice Roger D. McDonough of State Supreme Court in Albany vacated the value-added calculations that were the basis of NY teacher Sheri Lederman’s unfavorable rating “…in part because of the difficulty in measuring growth for students who already perform above grade level on state tests.” The NYTimes article explaining the court decision described the basis for Ms. Lederman’s “unfavorable” rating as follows:

For the 2012-13 school year, Sheri G. Lederman, a longtime teacher in the high-performing Great Neck public school district, on Long Island, received what was known as a growth score of 14 points out of a possible 20; the score was meant to calculate student progress over time. Her students scored substantially higher than the state average on annual standardized tests in English and math, and her score put her in the “effective” range.

The next year, her students scored a bit better on the math test than they had the year before, and slightly lower on the English exam. But her growth score plummeted to one out of 20, or “ineffective.”

This phenomenon results when a standardized test based on a bell curve is used since it is mathematically impossible for high performing students to “grow” on such a test due to the lack of “head room”. If one teacher’s class gets 48 out of 50 correct on a test in the baseline year and another teacher’s class gets 25 out of fifty correct that same year, in the subsequent year it is impossible for the high performing students to get 3 or more questions correct and, thus, impossible for them to show as much “growth” as the low performing students. Despite this inherent flaw, NYS, goaded on by Race-to-the-Top, decided to use “growth” as the primary metric for determining teacher performance. After all, test scores are an “objective” and “quantifiable” means of measuring the effectiveness of teachers. Fortunately for Ms. Lederman, the judge who heard her case— unlike the Board of Regents who adopted the evaluation system— understood basic statistics ad saw the flaw in the methodology.

While the judge limited his decision to only Ms. Lederman, as Carol Burris notes in Valerie Strauss’ Washington Post blog the decision was based on the evidence presented by academics who pointed out flaws in the system, evidence that could undermine the basis for all evaluations based on VAM. Here’s hoping other teachers in other States will follow Ms. Lederman’s lead and challenge their unfavorable ratings based on VAM and help drive a stake through the “testocracy” that drives public education today.