Posts Tagged ‘value added’

NYTimes Op Ed Acknowledges Limits of Big Data… Now Politicians Need to Follow Suit

May 3, 2015 Leave a comment

Two data scientists, Alex Peysakhovich (Facebook) and Seth Stephens-Davidowitz (formerly of Google) wrote an op ed piece for today’s NYTimes titled “How Not to Drown In Numbers“. The article describes how Big Data ideas have swept the country using the concepts described in “Moneyball” to underscore the limitations of data and the importance of human observation and insight in making key decisions on personnel. As I read the opening paragraphs I hoped that the authors would take on education… and they did so near the end of the article. After offering the cheating scandals in Chicago and Atlanta as evidence of teachers spending their time “...worrying about gaming the test, (instead of) what was happening in class“, Peysakhovich and Stephens-Davidowitz suggest that times might be changing in education:

Education, despite all the debate about test scores, the Common Core and value-added methods, is actually moving in a similar direction as baseball and tech companies.

It’s gotten much less press than the test score debate, but there is also a huge national effort to collect and evaluate small data. Student surveys have proliferated fast. So have parent surveys and teacher observations, where other experienced educators watch a teacher during a lesson.

Thomas Kane, a professor of education at Harvard, told us, “School districts realize they shouldn’t be focusing solely on test scores.”

A three-year study by the Bill and Melinda Gates Foundation bears out the value of both big and small data. The authors analyzed value-added models, student surveys and teacher observations. They tested how to best predict student outcomes on both traditional state tests and more cognitively demanding challenges in math and English. When they put the three measures together into a composite score, they got the best results. “Each measure adds something of value,” the report concluded.

As at Facebook and in baseball front offices, small data can find holes in the big data. If a teacher raises her students’ test scores but students say she wastes a lot of time, and outside observers rank her poorly, this raises big questions. Conversely, if a teacher does not improve test scores but students say she inspires them and principals think she is imparting profound lessons, we may give her the benefit of the doubt. Most important, while big data can tell us whether certain teachers are helping their students, small data gives us the best hope to answer a crucial question: How are they doing it?

While Bill Gates is seen as the primary promoter of Value Added Metrics, this is not the first article I’ve read citing this research finding— one that effectively undercuts the notion that VAM should be the sole or even primary metric for determining a classroom teacher’s effectiveness. I am inclined to give people the benefit of the doubt in general, and I am inclined to think that Bill Gates, who has pushed mightily to eliminate polio and help fight mundane diseases in impoverished foreign countries is NOT trying to destroy public schools. I do have one bone to pick with Mr. Gates, however, and that is his failure to trumpet changes in his positions on education as loudly as he broadcasts his initial findings. Several years ago Gates saw the creation of small high schools as the best means for engaging students. When he found that size of the school was an incomplete metric he neglected to share that conclusions loudly… and so many districts are continuing to scale down schools despite recent evidence that it makes no substantial difference. Now we have governors like Cuomo claiming that Big Data should be used for 50% of the teacher’s rating based on Gates’ initial research, we learn that such a weighting is not based on science… it’s a politically convenient number.

But Peysakhovich and Stephens-Davidowitz’ article ultimately does a disservice to those of us who believe there is an overemphasis on testing. Their use of Kane’s quote and Gates’ research without informing readers that the vast majority of statisticians fail to see any evidence that VAM is valid leads readers with the impression that VAM is unquestionably the best means of using Big Data. VAM is bad science the same way that climate change denial “science” and “creation science” are bad science… and two data scientists would realize that and share that with their readers.

How Standardized Tests Undercut Parenting, Teaching, and Schools

April 29, 2015 Leave a comment

Huffington Post blogger Stacy Steinberg’s latests post, “Set Up to Fail: High Stakes Testing in Public Schools” describes the impact of testing on administrators, teachers, and especially parents. Steinberg asserts that standardized tests are designed to “prove” schools are failing knowing that the tests will prove what we already know: the schools serving children raised in poverty will “fail” while the schools serving children raised in affluence will “succeed”.:

High stakes testing sets a child up to fail.

Even when a school employs hardworking teachers and has the support of parents and local business partners, some schools serve students that live in homes that fall below the poverty line. Research tells us that the number one indicator of how a child will perform on a standardized test is whether or not that child lives in poverty. Our flawed standardized testing model sets even the most talented teacher up to fail.

Unlike many critics of the standardized testing regimen, Steinberg appreciates the bind administrators face, noting that they:

…are caught between trying to provide quality education to… students and maintaining the financial backing from the state to do so. Unfortunately, these goals are often mutually exclusive. High stakes testing sets up even the most altruistic school administrator up to fail.

But the lion’s share of her post is directed at parents. Noting that parents work hard to provide opportunities for their children to thrive and be creative before they enter school, once they arrive to school they find an environment that stifles creativity and focuses on test preparation:

You watch as important subjects like history, art, and music drop from the curriculum as these subjects are not covered by the common core curriculum. You know your child needs an empathetic and patient teacher, yet you watch as teachers shift focus because of standardized testing standards. Assets like empathy and patience work against our best teachers who know their performance review will be based on how much and how quickly your student is able to learn. Perhaps worst of all, your child takes her first standardized test and comes home reporting that she was required to sign a piece of paper promising not to talk to you about the test or the testing procedure itself.
This leaves Steinberg with a sense that she, as a parent, is being set up to fail! She concludes by encouraging parents to write to their legislators sharing their concerns and urging them to do whatever is possible to put an end to this testing. This compelled me to leave the following comment:
Before you write, you might want to see what the Presidential candidates and your legislators think of this approach to education… and in the case of the Presidential candidates who is funding them. Are they claiming “schools are failing” based on these test results? Are they supporting the state or corporate takeover of “failing schools”? Are they claiming the solution to “failing schools” is to give parents more choices WITHOUT requiring that ALL schools open their doors to ALL students?
At this juncture (Bernie Sanders has not officially announced his candidacy), every single Presidential candidate supports the continued emphasis on testing and at least 33 Governors insist that standardized tests are the best way to measure school performance. At this juncture NCLB is the law of the land and the pending federal legislation will not end the testing regimen. And, at this juncture states have invested heavily in the new computerized tests and expect to use those tests for the foreseeable future. At this juncture, it is not too late to change the course of public education, but the odds of making the change are approaching the odds of us taking action to address climate change.

The Media’s Responsibility to Improve Voters’ and Parent’s Mathematical Reasoning

April 25, 2015 Leave a comment

I hate to sound like a broken record when it comes to calling out the NYTimes… but I intend to take every chance I get to make the point that VAM is flawed and the Times is complicit in the public’s misunderstanding of that mathematical and statistical fact. Sunday’s column by Nick Kristof, “Are You Smarter than an Eighth Grader” gives me such a chance. The column offers three questions from a recent international test of eighth graders and used them as examples of how poorly our students fared as compared to students in other countries. This led me to offer the following comment:

This paper contributes to the public’s misunderstanding of mathematics and statistics by supporting flawed ideas like “value added” measures as a basis for measuring individual teacher performance despite the rebuke of the methodology by the American Statistical Association and by publishing test data on individual schools without explaining their statistical significance.

I could have made the response more political by noting that the Times reports clearly incorrect and/or incomplete mathematical information when it comes to budget proposals, giving column inches to budget balancing ideas that lack specifics or, in some cases, don’t add up at all. When the “newspaper of record” supports statistical measures that are rebuked by professionals in the field and fails to provide its readers with mathematically accurate facts it is failing the public far more than its schools who need to defend themselves against baseless and inaccurate charges of “failure”.

And then this morning I read a letter to the editor to the Lubbock Avalanche Journal from George McFarland, a local superintendent, pointing out how their media have jumped onto the “failing schools” meme without looking at the facts, which are:

For example, news media like to grab onto quotes that public schools are clearly failing because there are 146,000 students trapped in almost 300 failing public schools. However, considering that 146,000 students is 2.8 percent of the 5,151,925 Texas students, simple math can identify more than 97 percent of Texas public school students are not enrolled in “failing” schools.

Likewise, 300 schools represent 3.5 percent of the 8,574 public school campuses in Texas, meaning 96.5 percent of campuses are not “failing.” These numbers might suggest there are areas where public education can improve but certainly don’t necessitate the need to completely trash an entire system which is serving so many successfully.

Thankfully the newspaper published the article… but if they were doing their job every time a politician said schools were failing they would note that 96.5% are NOT failing… but that FACT undercuts the narrative that is stuck in the minds of readers and voters.

The Consequences of Opting Out MAY be the Loss of Federal Funds

April 23, 2015 Leave a comment

In what could have national ramifications, a post in Politico suggests that the STATE may have the power to withhold FEDERAL funds from districts who fail to participate in the testing.

Here’s the context for this story:

Last year, parents across NYS launched a campaign to opt out of the state tests because they feel that the emphasis on test results is undermining the curriculum in their districts, placing inordinate and inappropriate pressure on their children, and providing them with no information whatsoever about their child’s mastery of the information tested. Teachers unions tacitly supported this movement for the same reasons, emphasizing the flaws in using value added measures for evaluating them and the lack of useful information made available following the testing.

Sensing the growing opposition to the testing regimen, Governor Cuomo included a provision that 50% of the teacher evaluation be used on test results in his budget proposal. When this proviso didn’t fly in the legislature, he accepted a compromise that would allow the Regents to determine the extent to which testing would be the basis for evaluations. As noted in earlier posts, this effectively gave Cuomo a green lift to proceed with VAM since the majority of the Regents and the Regents chair are supporters of the testing regimen Cuomo wants to put in place.

Last year the opt-out campaign was marginal… but this year nearly 200,000 parents have opted out of the testing program, teachers unions have explicitly supported the opt-out movement, and some school boards and superintendents have formally and publicly endorsed the movement. As it became evident that parents were not in support of the movement, Tisch and Cuomo both made claims that they their hands were tied win it came to withholding of federal funds… but as Politico notes that may not be the case:

State officials had previously suggested that the matter was out of their hands. Representatives for the U.S. Department of Education and the state Education Department have said the federal government could withhold Title I funds—grants for schools that serve low-income students—if fewer than 95 percent of students in an individual school or district take the tests, and Governor Andrew Cuomo on Tuesday also said the federal government holds the power to decide whether to withhold funding.

But public statements and regulatory guidance from both the U.S. and state education departments suggest the decision is not totally up to the feds.

“They [federal officials] seem to indicate—I’m hearing that we have discretion, but we will find out how much discretion we have,” state Board of Regents chancellor Merryl Tisch told Capital on Tuesday. “If we do have discretion, we intend to use it.”

Duncan has put Tisch and Cuomo in a bind! Here’s why. Affluent districts with high opt-out percentages and low Title One allocations have less to lose than districts serving children raised in poverty that have high opt-out percentages and high Title One allocations. Thus, if Tisch and Cuomo use the withholding of Federal Title One funds as a penalty they will be hurting children raised in poverty more than those in affluent districts.

And Tisch may have put herself in a bind with her assertion that the Regents intend to use any discretion they have because if they DO assert themselves by withholding the marginal funds from affluent districts they will unleash a massive protest. IF the Regents intend to withhold funds they need to do so quickly because local budgets will be adopted in May and presumably Boards will be advised of their State funding in advance of those budget votes.

Finally, Duncan’s position in NYS may have created a problem for himself: If NY State can withhold federal funds as a penalty, why couldn’t ANY state do the same? And does this ability to withhold funds mean that States have the authority to re-allocate the federal dollars they receive?

It seems to me that a can or worms has just been opened in NYS and conceivably across the country. We may see some interesting fireworks in the coming weeks!

Times Article on Testing Wrong Many Counts! Here’s a List!

April 21, 2015 Leave a comment

I re-blogged Diane Ravitch’s post on the NYTimes article by Kate Taylor and Mokoto Rich because it perpetuated the myth that unions and not parents were the primary force behind the opt-out movement. This post will provide a blow-by-blow listing of everything that is wrong about their article… and bear with me because the list is a long one!

  • Taylor and Rich write: “In Florida, the teachers’ union has lobbied to limit the use of standardized tests, and the governor last week signed a bill thatlimits the number of hours students can spend taking them”, linking the union lobbying to the passage of the law when anyone familiar with FL politics knows that it was the conservative right who compelled Governor Scott to abandon the testing regimen first put in place by presumptive presidential aspirant Jeb Bush.
  • They write: “Lawmakers are considering a bill that removes the most punitive consequences for schools and makes clear that states do not have to use test scores to evaluate teachers” as if this will put an end to testing… which, as previous posts have noted is NOT the case. The new bill will require that states develop their own accountability measures and those measures may (and probably will) include annual tests.
  • Taylor and Rich quote testing advocates such as Joshua Edelman who is quoted as follows: “It’s right at the point when we finally actually have the kind of improved tests that so many folks petitioned for and advocated for for years,” and while “Mr. Edelman said that the organization supports legislation to reduce unnecessary testing (he felt that) “encouraging parents to opt out is not an effort to reduce over testing… “It’s an effort to undermine accountability”. This is wrong on two scores: first the tests being used now are NOT the kind of “improved tests so many folks petitioned for” because their results cannot be used by teachers to help individual students improve their knowledge and understanding nor can they prescribe the kind of instruction each child needs. And parents and unions are not opposed to “accountability”, they are opposed to the kind of accountability model imposed on schools by politicians who want to privatize public education.
  • The article gives several inches of space to those who oppose the campaign against testing, and Taylor and Rich then write: “The union argued that it was not fair to make test scores so big a part of a teacher’s rating because many factors outside the classroom can influence scores.” That statement is accurate but incomplete: it omits the fact that the unions argue the use of test scores is unfair because there is no statistical validity to their use for evaluating teachers and even the use of the tests to evaluate schools is debatable.

In the article there is no mention whatsoever of the statistical science community’s virtually unanimous agreement that “value added” measures are impossible… which is akin to having an article denying climate change published based on the findings of a scientist on the payroll of the petroleum lobby. By turning a blind eye to the statistical worthlessness of value added measures the Times is effectively telling the public that test scores can be a valid measure teacher performance… which is clearly untrue.  The Times is doing a disservice to students, parents, taxpayers, and– yes— teachers.

Parents Opting Out: 155,000 So Far in NYS

April 20, 2015 Leave a comment

Several stories this past week described the increase in the number of students who opted out of tests and contrary to the narrative promoted by the pro-privatization crowd and mainstream media it is being led by parents and NOT the teacher’s union.

USA Today reported the 155,000 figure on Thursday, based on information provided by United to Counter the Core, whose Facebook page featured this statement:


As we complete the first round of counts for ELA and move into the first round of counts for math, it is important to remember why parents do this.

Make no mistake, this wave of civil disobedience is not just about Andrew Cuomo and his teacher evaluation plan. Cuomo is the flavor-of-the-month in a long line of ill-prepared, ill-advised education reformers, each worse than the one before. These sometimes well-intentioned reformers have nevertheless damaged an entire generation of America’s schoolchildren going all the way back to No Child Left Behind.

Hundreds of thousands of parents are not making political statements, they are looking at crying, defeated children around their kitchen tables and demanding meaningful change. NY parents and teachers want education reform that is educator-driven, that is tested and proven, that addresses the real problems facing our schools and our children, and that is implemented with a modicum of competency.

A reduction of testing or evaluations does not address the underlying issue. NY parents want what parents have wanted since time began – a better education for our children.

As Democracy Now reported, this figure might be an understatement:

Protest organizers say at least 155,000 pupils opted out — and that is with only half of school districts tallied so far. … More than a decade after the passage of No Child Left Behind, educators, parents and students nationwide are protesting the preponderant reliance on high-stakes standardized testing, saying it gives undue importance to ambiguous data and compromises learning in favor of test prep.

Nadia Prupis’ synopsis of the opt out movement in Common Dreams included the reports from NYS, referenced Democracy Now’s coverage and also included a quote from Juan Gonzalez’ NY Daily News account referenced earlier this week in this blog. Her post noted that the 155,000 figure dwarfed the 49,000 who opted out last year and only included half of the districts in NYS. From these reports, it appears that NYS’ hurried and bungled roll out of last year’s test may be the undoing of the standardized test movement in that state… but Governor Cuomo and the Regents will likely find some way to downplay the opt-outs and/or continue to promote the notion that the unions are behind it. One of the most reprehensible ideas advanced is to permit “high performing districts” to opt out completely, thus creating a de facto two tiered system of assessments whereby affluent schools are not required to take standardized tests.

Say tuned for Week Two of the testing cycle to see how many children stay home during the math tests in NYS… and for the coming weeks win standardized tests are administered across the nation.


Of COURSE Principals Matter… but Principles Matter, Too

April 18, 2015 Leave a comment

Today’s blog post title is a play on yesterday’s NYTimes featured an op ed article by Will Miller titled “Want Reform? Principals Matter, Too”.  In the article, Miller, who is president of the Wallace Foundation, breathlessly reports that the Principal of a school plays a key role in school improvement… a fact that true school reformers like Ron Edmunds knew decades ago. Miller’s op ed piece recounts all of the reasons this is the fact, touches on some of the research that demonstrates this, and offers some recommendations on how this can be addressed.

One point Mr. Miller overlooked was the impact of VAM on school administrators, especially in New York. The latest thinking on “reform” in NY insists that test scores take greater precedence than principal evaluations. Indeed, Governor Cuomo has so little regard for Principals’ ability to evaluate that he wants to institute a system that requires independent third party evaluations. Why? Because the failure rate for teachers is way too low! To paraphrase Mr. Miller, it’s hard to think of another profession where so little attention is paid to leaders. Organizations like the military, corporations and universities listen to and respect their leaders. If we’re going to do this in public education, a lot has to change… beginning with abandoning the notion that test results can replace direct observation in the classroom as a means of judging teacher and administrator performance.

The fundamental principle that test scores cannot measure the human interactions between a teacher and a student and a leader and a subordinate needs to be brought to the forefront…. because THAT principle matters A LOT more than any Principal.