Both Diane Ravitch and Cathy O’Neill cited the news that Chris Christie has announced a major change in the use of VAM to evaluation teachers, with Diane Ravitch providing a link to an article in the NorthJersey.Com web site where a detailed article appeared. According to the article, Christie announced he intended to rollback the use of VAM from 30% to 10%. To help him determine the future use of VAM to higher levels, he issued an executive order creating a commission that would “…review the effectiveness of all K-12 tests used to assess student knowledge… look at volume, frequency and impact of student testing throughout New Jersey school districts.” But here’s the kicker (with my emphases added:
Christie will appoint all nine commission members, who should have expertise or experience in education policy or administration, according to his order. The commission will issue an initial report with recommendations by Dec. 31, and a final report seven months later.
The commission “should have” this expertise… but if Christie’s appointments to State operated districts are any indication of the kind of qualifications he will be using to identify people with “expertise” no one in New Jersey should be assured the commission will know anything about the construction of standardized tests. Instead of requiring anyone on the commission to have expertise in psychometrics Christie is appointing people with expertise in “education policy or administration”. As one with expertise in both of those spheres I can assure readers there are few “education policy or administration” folks who know anything about psychometrics… and that one reason why so many of them have bought into VAM. If Christie is sincere about improving the use of tests in NJ he should appoint someone like Cathy O’Neill (aka the Mathbabe) or someone from FairTest to serve on this commission.
Finally, a political sidebar: the report of this commission will be issued in July 2015— right about the time the Christie for President campaign will be ramping up. Should be interesting times in NJ!
Malcolm Gladwell’s book, Outliers, posited that in order to become successful one had to practice at a skill or craft for at least 10,000 hours. This “10,000 hour rule” got twisted to read that IF one practiced a craft for at least 10,000 hours they would become successful— a small but significantly different conclusion than Gladwell made. Gladwell suggested that 10,000 hours was a necessary but not sufficient cause of success. The twisted rule suggests that 10,000 hours was necessary and sufficient…. and the twisted rule is the one that fond its way into the mainstream media and became a statement of faith and not an evidence based conclusion.
Sunday’s NYTimes had an article titled “The Limits of Practice” that refuted Gladwell’s “findings” regarding the 10,000 hour rule. The article acknowledged that Gladwell fruitlessly attempted to clarify his findings in a subsequent New Yorker article but nevertheless picked up on the flawed “10,000 hour” meme, (superfluously) demonstrated that it was wrong. In the comment section I made the following observation:
This line jumped out at me after reading Nyhan’s article on Sunday: “But despite this, and despite recent research, the idea that you can be awesome at anything with 10,000 hours of work continues to hold sway.” The 10,000 hour meme, like many faith based memes, is easy to remember, is intuitively appealing, and ESPECIALLY appealing to those who want to shift the burden away from government to the individual. After all, if doing well in school is just a matter of hard work and grit and nothing to do with being born into the right zip code, then no government intervention is needed to improve the lives of children or to improve education.
The “10,000 hour rule” is another in a long line of bad idea holds sway in the minds of the public because IF it is true difficult decisions can be avoided. If trickle down economics works we don’t need to tax people who are earning a lot of money; if VAM works we don’t need to spend money on administrators to coach and counsel bad teachers out of the profession; if the “10,000 hour rule” works we don’t need to provide support to children— THEY need to work harder and have more grit. Bad ideas are uncomplicated, intuitively appealing, and require little or no effort or cost to implement. That’s why they stick even if they are proven fully or partially false.
As is almost always the case, the Mathbabe, Cathy O’Neill, has written a blog post full of thought provoking questions on the use of data. The big question O’Neill poses in today’s post is this: “What Constitutes Evidence?” She poses this question in the context of the collection of health data that will presumably be used to rank and evaluate doctors and patients in ways that have not been clearly defined but seem to be universally accepted. She asks readers if she is paranoid in her fear that this data, once collected, will be misused? Virtually all her readers assure her that she is NOT irrational in her fears or her concerns. I share her concern… but also fear that this will be yet another instance of us using quantification in lieu of judgment… and when that occurs, bad things often follow. I left the following comment to her post, providing an example from education where statistical algorithms have replaced human judgment… namely… VAM:
What evidence is there that ANY statistical models used to draw conclusions from aggregated data are valid and reliable? We want to believe that there is a way to quantify things to replace human judgment. Human judgment based on data can be flawed and colored by misjudgment… but, in a well conceived bureaucracy it can be subjected to further review by others. We seem to think that “dispassionate quantification” that substitutes precision for accuracy requires no review.
An example of this bogus quantification is best found in education where statistically flawed algorithms are used as the basis for judging performance with no recourse available. You’ve written frequently and eloquently about the VAM hoax… what makes you think the medical field is going to develop a more sophisticated means of “scoring” doctors or patients?
Here’s what I observe happening: we have no faith in fairness or due process and less faith in “the government” or “the system” and WAY too much faith in quantification… Readers of this blog know that I have questions and concerns about the officials we’ve elected to lead us and some of the decisions these officials made… but I prefer an imperfect elected human being to an unelected computer program developed by a mid-level contract employee who’s company submitted the lowest bid for a project.