Two data scientists, Alex Peysakhovich (Facebook) and Seth Stephens-Davidowitz (formerly of Google) wrote an op ed piece for today’s NYTimes titled “How Not to Drown In Numbers“. The article describes how Big Data ideas have swept the country using the concepts described in “Moneyball” to underscore the limitations of data and the importance of human observation and insight in making key decisions on personnel. As I read the opening paragraphs I hoped that the authors would take on education… and they did so near the end of the article. After offering the cheating scandals in Chicago and Atlanta as evidence of teachers spending their time “...worrying about gaming the test, (instead of) what was happening in class“, Peysakhovich and Stephens-Davidowitz suggest that times might be changing in education:
Education, despite all the debate about test scores, the Common Core and value-added methods, is actually moving in a similar direction as baseball and tech companies.
It’s gotten much less press than the test score debate, but there is also a huge national effort to collect and evaluate small data. Student surveys have proliferated fast. So have parent surveys and teacher observations, where other experienced educators watch a teacher during a lesson.
Thomas Kane, a professor of education at Harvard, told us, “School districts realize they shouldn’t be focusing solely on test scores.”
A three-year study by the Bill and Melinda Gates Foundation bears out the value of both big and small data. The authors analyzed value-added models, student surveys and teacher observations. They tested how to best predict student outcomes on both traditional state tests and more cognitively demanding challenges in math and English. When they put the three measures together into a composite score, they got the best results. “Each measure adds something of value,” the report concluded.
As at Facebook and in baseball front offices, small data can find holes in the big data. If a teacher raises her students’ test scores but students say she wastes a lot of time, and outside observers rank her poorly, this raises big questions. Conversely, if a teacher does not improve test scores but students say she inspires them and principals think she is imparting profound lessons, we may give her the benefit of the doubt. Most important, while big data can tell us whether certain teachers are helping their students, small data gives us the best hope to answer a crucial question: How are they doing it?
While Bill Gates is seen as the primary promoter of Value Added Metrics, this is not the first article I’ve read citing this research finding— one that effectively undercuts the notion that VAM should be the sole or even primary metric for determining a classroom teacher’s effectiveness. I am inclined to give people the benefit of the doubt in general, and I am inclined to think that Bill Gates, who has pushed mightily to eliminate polio and help fight mundane diseases in impoverished foreign countries is NOT trying to destroy public schools. I do have one bone to pick with Mr. Gates, however, and that is his failure to trumpet changes in his positions on education as loudly as he broadcasts his initial findings. Several years ago Gates saw the creation of small high schools as the best means for engaging students. When he found that size of the school was an incomplete metric he neglected to share that conclusions loudly… and so many districts are continuing to scale down schools despite recent evidence that it makes no substantial difference. Now we have governors like Cuomo claiming that Big Data should be used for 50% of the teacher’s rating based on Gates’ initial research, we learn that such a weighting is not based on science… it’s a politically convenient number.
But Peysakhovich and Stephens-Davidowitz’ article ultimately does a disservice to those of us who believe there is an overemphasis on testing. Their use of Kane’s quote and Gates’ research without informing readers that the vast majority of statisticians fail to see any evidence that VAM is valid leads readers with the impression that VAM is unquestionably the best means of using Big Data. VAM is bad science the same way that climate change denial “science” and “creation science” are bad science… and two data scientists would realize that and share that with their readers.
I hate to sound like a broken record when it comes to calling out the NYTimes… but I intend to take every chance I get to make the point that VAM is flawed and the Times is complicit in the public’s misunderstanding of that mathematical and statistical fact. Sunday’s column by Nick Kristof, “Are You Smarter than an Eighth Grader” gives me such a chance. The column offers three questions from a recent international test of eighth graders and used them as examples of how poorly our students fared as compared to students in other countries. This led me to offer the following comment:
This paper contributes to the public’s misunderstanding of mathematics and statistics by supporting flawed ideas like “value added” measures as a basis for measuring individual teacher performance despite the rebuke of the methodology by the American Statistical Association and by publishing test data on individual schools without explaining their statistical significance.
I could have made the response more political by noting that the Times reports clearly incorrect and/or incomplete mathematical information when it comes to budget proposals, giving column inches to budget balancing ideas that lack specifics or, in some cases, don’t add up at all. When the “newspaper of record” supports statistical measures that are rebuked by professionals in the field and fails to provide its readers with mathematically accurate facts it is failing the public far more than its schools who need to defend themselves against baseless and inaccurate charges of “failure”.
And then this morning I read a letter to the editor to the Lubbock Avalanche Journal from George McFarland, a local superintendent, pointing out how their media have jumped onto the “failing schools” meme without looking at the facts, which are:
For example, news media like to grab onto quotes that public schools are clearly failing because there are 146,000 students trapped in almost 300 failing public schools. However, considering that 146,000 students is 2.8 percent of the 5,151,925 Texas students, simple math can identify more than 97 percent of Texas public school students are not enrolled in “failing” schools.
Likewise, 300 schools represent 3.5 percent of the 8,574 public school campuses in Texas, meaning 96.5 percent of campuses are not “failing.” These numbers might suggest there are areas where public education can improve but certainly don’t necessitate the need to completely trash an entire system which is serving so many successfully.
Thankfully the newspaper published the article… but if they were doing their job every time a politician said schools were failing they would note that 96.5% are NOT failing… but that FACT undercuts the narrative that is stuck in the minds of readers and voters.
In what could have national ramifications, a post in Politico suggests that the STATE may have the power to withhold FEDERAL funds from districts who fail to participate in the testing.
Here’s the context for this story:
Last year, parents across NYS launched a campaign to opt out of the state tests because they feel that the emphasis on test results is undermining the curriculum in their districts, placing inordinate and inappropriate pressure on their children, and providing them with no information whatsoever about their child’s mastery of the information tested. Teachers unions tacitly supported this movement for the same reasons, emphasizing the flaws in using value added measures for evaluating them and the lack of useful information made available following the testing.
Sensing the growing opposition to the testing regimen, Governor Cuomo included a provision that 50% of the teacher evaluation be used on test results in his budget proposal. When this proviso didn’t fly in the legislature, he accepted a compromise that would allow the Regents to determine the extent to which testing would be the basis for evaluations. As noted in earlier posts, this effectively gave Cuomo a green lift to proceed with VAM since the majority of the Regents and the Regents chair are supporters of the testing regimen Cuomo wants to put in place.
Last year the opt-out campaign was marginal… but this year nearly 200,000 parents have opted out of the testing program, teachers unions have explicitly supported the opt-out movement, and some school boards and superintendents have formally and publicly endorsed the movement. As it became evident that parents were not in support of the movement, Tisch and Cuomo both made claims that they their hands were tied win it came to withholding of federal funds… but as Politico notes that may not be the case:
State officials had previously suggested that the matter was out of their hands. Representatives for the U.S. Department of Education and the state Education Department have said the federal government could withhold Title I funds—grants for schools that serve low-income students—if fewer than 95 percent of students in an individual school or district take the tests, and Governor Andrew Cuomo on Tuesday also said the federal government holds the power to decide whether to withhold funding.
But public statements and regulatory guidance from both the U.S. and state education departments suggest the decision is not totally up to the feds.
“They [federal officials] seem to indicate—I’m hearing that we have discretion, but we will find out how much discretion we have,” state Board of Regents chancellor Merryl Tisch told Capital on Tuesday. “If we do have discretion, we intend to use it.”
Duncan has put Tisch and Cuomo in a bind! Here’s why. Affluent districts with high opt-out percentages and low Title One allocations have less to lose than districts serving children raised in poverty that have high opt-out percentages and high Title One allocations. Thus, if Tisch and Cuomo use the withholding of Federal Title One funds as a penalty they will be hurting children raised in poverty more than those in affluent districts.
And Tisch may have put herself in a bind with her assertion that the Regents intend to use any discretion they have because if they DO assert themselves by withholding the marginal funds from affluent districts they will unleash a massive protest. IF the Regents intend to withhold funds they need to do so quickly because local budgets will be adopted in May and presumably Boards will be advised of their State funding in advance of those budget votes.
Finally, Duncan’s position in NYS may have created a problem for himself: If NY State can withhold federal funds as a penalty, why couldn’t ANY state do the same? And does this ability to withhold funds mean that States have the authority to re-allocate the federal dollars they receive?
It seems to me that a can or worms has just been opened in NYS and conceivably across the country. We may see some interesting fireworks in the coming weeks!
I re-blogged Diane Ravitch’s post on the NYTimes article by Kate Taylor and Mokoto Rich because it perpetuated the myth that unions and not parents were the primary force behind the opt-out movement. This post will provide a blow-by-blow listing of everything that is wrong about their article… and bear with me because the list is a long one!
- Taylor and Rich write: “In Florida, the teachers’ union has lobbied to limit the use of standardized tests, and the governor last week signed a bill thatlimits the number of hours students can spend taking them”, linking the union lobbying to the passage of the law when anyone familiar with FL politics knows that it was the conservative right who compelled Governor Scott to abandon the testing regimen first put in place by presumptive presidential aspirant Jeb Bush.
- They write: “Lawmakers are considering a bill that removes the most punitive consequences for schools and makes clear that states do not have to use test scores to evaluate teachers” as if this will put an end to testing… which, as previous posts have noted is NOT the case. The new bill will require that states develop their own accountability measures and those measures may (and probably will) include annual tests.
- Taylor and Rich quote testing advocates such as Joshua Edelman who is quoted as follows: “It’s right at the point when we finally actually have the kind of improved tests that so many folks petitioned for and advocated for for years,” and while “Mr. Edelman said that the organization supports legislation to reduce unnecessary testing (he felt that) “encouraging parents to opt out is not an effort to reduce over testing… “It’s an effort to undermine accountability”. This is wrong on two scores: first the tests being used now are NOT the kind of “improved tests so many folks petitioned for” because their results cannot be used by teachers to help individual students improve their knowledge and understanding nor can they prescribe the kind of instruction each child needs. And parents and unions are not opposed to “accountability”, they are opposed to the kind of accountability model imposed on schools by politicians who want to privatize public education.
- The article gives several inches of space to those who oppose the campaign against testing, and Taylor and Rich then write: “The union argued that it was not fair to make test scores so big a part of a teacher’s rating because many factors outside the classroom can influence scores.” That statement is accurate but incomplete: it omits the fact that the unions argue the use of test scores is unfair because there is no statistical validity to their use for evaluating teachers and even the use of the tests to evaluate schools is debatable.
In the article there is no mention whatsoever of the statistical science community’s virtually unanimous agreement that “value added” measures are impossible… which is akin to having an article denying climate change published based on the findings of a scientist on the payroll of the petroleum lobby. By turning a blind eye to the statistical worthlessness of value added measures the Times is effectively telling the public that test scores can be a valid measure teacher performance… which is clearly untrue. The Times is doing a disservice to students, parents, taxpayers, and– yes— teachers.
Several stories this past week described the increase in the number of students who opted out of tests and contrary to the narrative promoted by the pro-privatization crowd and mainstream media it is being led by parents and NOT the teacher’s union.
USA Today reported the 155,000 figure on Thursday, based on information provided by United to Counter the Core, whose Facebook page featured this statement:
BRIEF STATEMENT FROM UNITED TO COUNTER THE CORE
As we complete the first round of counts for ELA and move into the first round of counts for math, it is important to remember why parents do this.
Make no mistake, this wave of civil disobedience is not just about Andrew Cuomo and his teacher evaluation plan. Cuomo is the flavor-of-the-month in a long line of ill-prepared, ill-advised education reformers, each worse than the one before. These sometimes well-intentioned reformers have nevertheless damaged an entire generation of America’s schoolchildren going all the way back to No Child Left Behind.
Hundreds of thousands of parents are not making political statements, they are looking at crying, defeated children around their kitchen tables and demanding meaningful change. NY parents and teachers want education reform that is educator-driven, that is tested and proven, that addresses the real problems facing our schools and our children, and that is implemented with a modicum of competency.
A reduction of testing or evaluations does not address the underlying issue. NY parents want what parents have wanted since time began – a better education for our children.
As Democracy Now reported, this figure might be an understatement:
Protest organizers say at least 155,000 pupils opted out — and that is with only half of school districts tallied so far. … More than a decade after the passage of No Child Left Behind, educators, parents and students nationwide are protesting the preponderant reliance on high-stakes standardized testing, saying it gives undue importance to ambiguous data and compromises learning in favor of test prep.
Nadia Prupis’ synopsis of the opt out movement in Common Dreams included the reports from NYS, referenced Democracy Now’s coverage and also included a quote from Juan Gonzalez’ NY Daily News account referenced earlier this week in this blog. Her post noted that the 155,000 figure dwarfed the 49,000 who opted out last year and only included half of the districts in NYS. From these reports, it appears that NYS’ hurried and bungled roll out of last year’s test may be the undoing of the standardized test movement in that state… but Governor Cuomo and the Regents will likely find some way to downplay the opt-outs and/or continue to promote the notion that the unions are behind it. One of the most reprehensible ideas advanced is to permit “high performing districts” to opt out completely, thus creating a de facto two tiered system of assessments whereby affluent schools are not required to take standardized tests.
Say tuned for Week Two of the testing cycle to see how many children stay home during the math tests in NYS… and for the coming weeks win standardized tests are administered across the nation.
Today’s blog post title is a play on yesterday’s NYTimes featured an op ed article by Will Miller titled “Want Reform? Principals Matter, Too”. In the article, Miller, who is president of the Wallace Foundation, breathlessly reports that the Principal of a school plays a key role in school improvement… a fact that true school reformers like Ron Edmunds knew decades ago. Miller’s op ed piece recounts all of the reasons this is the fact, touches on some of the research that demonstrates this, and offers some recommendations on how this can be addressed.
One point Mr. Miller overlooked was the impact of VAM on school administrators, especially in New York. The latest thinking on “reform” in NY insists that test scores take greater precedence than principal evaluations. Indeed, Governor Cuomo has so little regard for Principals’ ability to evaluate that he wants to institute a system that requires independent third party evaluations. Why? Because the failure rate for teachers is way too low! To paraphrase Mr. Miller, it’s hard to think of another profession where so little attention is paid to leaders. Organizations like the military, corporations and universities listen to and respect their leaders. If we’re going to do this in public education, a lot has to change… beginning with abandoning the notion that test results can replace direct observation in the classroom as a means of judging teacher and administrator performance.
The fundamental principle that test scores cannot measure the human interactions between a teacher and a student and a leader and a subordinate needs to be brought to the forefront…. because THAT principle matters A LOT more than any Principal.