Home > Uncategorized > Meaningful Metrics in Short Supply

Meaningful Metrics in Short Supply

November 27, 2015

Two articles, one from the Tuck School of Business on-line newsletter on the short-changing of the Bureau of Labor Standards (BLS) and one from the NYPost criticizing NYC Mayor deBlasio’s abandonment of the grading system imposed on public schools are based on the same concern: the public’s lack of credible data is leading to flawed decision making.

Writing in the Tuck School’s newsletter, Dean Matt Slaughter and his colleague Matt Rees call the US Government’s decision to level fund the BLS as part of sequestration process “pound die and penny foolish”. Why? Slaughter and Reese contend that:

Healthy societies create, fund generously, and zealously protect from political meddling public agencies that collect and disseminate economic data: on workers, on families, on companies, and on all related economic actors and transactions.  The sounder the public economic data, the sounder the economic decisions of individuals, firms, and policymakers alike. Thus do these data contribute to long-run economic growth and prosperity.

The solution to the complaints about the impact of underfunding is predictable: turn the process over to the private sector! Rees and Slaughter disagree and in an open letter to Congress wrote:

To be sure, private-sector data initiatives such as ADP’s well-known monthly employment reports have a role to play. But they analyze only a slice of the labor market, and they benchmark their findings to BLS data, the gold standard for measuring employment. BLS remains the sole entity in either the public or the private realm with both the mandate and capacity to collect and provide to the public information on the labor market in its entirety. Data collection efforts by independent government agencies like the BLS also benefit from a stamp of legitimacy and neutrality that is difficult to match in the private sector.

They could have asked rhetorically “What happened when monitoring of mortgages was left to bond rating agencies?” Their point about the collection of insubstantial data and using that data to issue flawed public pronouncements is most prevalent in public education, where several statisticians bend over backward to provide politicians with simplistic metrics designed to inform the public about the quality of their schools. There is one BIG difference between the BLS data and that gathered by governments: data on school performance does NOT benefit from “…a stamp of legitimacy and neutrality”. Instead, the Bloomberg administration in its zealous effort to promote the need for and use of charter schools set cut scores on tests that initially “proved” scores of schools were failing and then used different standards to prove charter schools succeeded where public schools fell short of he mark. As the author of the NYPost article, Charles Sahm noted:

From 2007 through 2013, the New York City Department of Education released “Progress Reports,” which included an A-F letter grade for every public school in the city. These grades were based on year-to-year student progress on tests, overall student performance and school-environment surveys. While far from perfect, they provided parents with objective, data-driven information in a relatively easy-to-understand format.

Sahm went onto criticize de Blasio’s decision to abandon this letter grade system and replace it with “School Quality Reports” that label schools as excellent, good, fair or poor. Sahm’s complaint?

…while they still contain information on student achievement, the new format focuses mostly on the results of surveys of parents, teachers and (older) students.

The Department of Education said it wants to “focus on multiple measures of school improvement,” including the “six essential elements of the Framework for Great Schools: Rigorous Instruction, Collaborative Teachers, Supportive Environment, Effective School Leadership, Strong Family-Community Ties, and Trust.”

But parents probably care more about whether children can read or do math at grade level rather than whether teachers “feel respected” — one of the survey questions.

Sahm’s solution? You guessed it: he has a proprietary plan that is better than Bloomberg’s ratings or de Blasio’s!

If parents are looking for a simpler, more objective analysis of their child’s school, they may want to look to a new school-rating system from the Manhattan Institute at SchoolGrades.org.

In comparing the old Bloomberg plan that based the letter grade on a black-box calculation derived from student test scores or the spiffy Sahm plan that is an enhanced black box set of calculations, I’ll take de Blasio’s system that looks at a broader set of data that incorporates thinks like parent surveys and surveys of teachers.

Slaughter and Rees’ conclusions, too, illustrate the flaws in Sahm’s “solution”: which privatizes something the government should be doing at the national level and, in doing so, substitutes robust findings with ones that are “easy to understand” but ultimately meaningless.

Categories: Uncategorized Tags: ,
%d bloggers like this: