Jerry Z. Muller’s recent article in The American Interest, “The Costs of Accountability“, describes the measurement mindset that undergirds the “education reform” movement, today’s updated version of the “cult of efficiency” that Raymond Callahan wrote about four decades ago. If you want to know the mental model being used in business and being imposed on the public sector, grab a cup of coffee and spend a few minutes reading the 22 page article. If you want a sense of the article, read this string of quotes below:
The characteristic feature of the culture of accountability is the aspiration to replace judgment with standardized measurement. Judgment is understood as personal, subjective, and self-interested; metrics are supposed to provide information that is hard and objective. The strategy behind the numbers is to improve institutional efficiency by offering rewards to those whose metrics are highest or whose benchmarks have been reached, and by punishing those who fall behind relative to them…
The attractions of accountability metrics are apparent. Yet like every culture, the culture of accountability has carved out its own unquestioned sacred space and, as with all arguments from presumed authority, possesses its characteristic blind spots. In this case, the virtues of accountability metrics have been oversold and their costs are under appreciated….
Measurement schemes are deceptively attractive because they often “prove” themselves through low-hanging fruit… But, in many cases, the extension of standardized measurement may suffer diminished utility and even become counterproductive if sensible pragmatism gives way to metric madness. Measurement can readily become counterproductive when it tries to measure the unmeasurable and quantify the unquantifiable, whether to determine rewards or for other purposes. This tends to be the case as the scale of what is being measured grows while the activity itself becomes functionally differentiated, and when those tasked with doing the measuring are detached organizationally from the activity being measured….
(Accountability) picked up steam in the 1970s, and then achieved a kind of theoretical quintessence in “principal-agent theory”, developed by economists in business schools. That theory called attention to the gap between the purposes of institutions and the people who ran and were employed by them. It focused on the problem of aligning the interests of shareholders in maximum profitability and stock price with the interests of corporate executives, whose priorities might diverge from those goals. Principal-agent theory articulated in abstract terms the general suspicion that those employed in institutions were not to be trusted; that their activity had to be monitored and measured; that those measures needed to be transparent to those without firsthand knowledge of the institutions; and that pecuniary rewards and punishments were the most effective way to motivate “agents.” Here too, numbers were seen as a guarantee of objectivity and as replacement for intimate knowledge and personal trust….
In the public sector, the show horse of accountability became “No Child Left Behind” (NCLB), an educational act signed into law with bipartisan support by George W. Bush in 2001 whose formal title was, “An act to close the achievement gap with accountability, flexibility, and choice, so that no child is left behind.”
The NCLB legislation grew out of more than a decade of heavy lobbying by business groups concerned about the quality of the workforce, civil rights groups worried about differential group achievement, and educational reformers who demanded national standards, tests, and assessment.7 The benefit of such measures was oversold, in terms little short of utopian.
Thus William Kolberg of the National Alliance of Business asserted that, “the establishment of a system of national standards, coupled with assessment, would ensure that every student leaves compulsory school with a demonstrated ability to read, write, compute and perform at world-class levels in general school subjects.” The first fruit of this effort, on the Federal level, was the “Improving America’s Schools Act” adopted under President Clinton in 1994. Meanwhile, in Texas, Governor George W. Bush became a champion of mandated testing and educational accountability, a stance that presaged his support for NCLB….
Yet more than a decade after its implementation, the benefits of the accountability provisions of NCLB remain elusive. Its advocates grasp at any evidence of improvement on any test at any grade in any demographic group for proof of NCLB’s efficacy. But test scores for primary school students have gone up only slightly, and no more quickly than before the legislation was enacted. Its impact on the test scores of high school students has been more limited still.
The unintended consequences of NCLB’s testing-and-accountability regime are more tangible, however, and exemplify many of the characteristic pitfalls of the culture of accountability. Under NCLB, scores on standardized tests are the numerical metric by which success and failure are judged. And the stakes are high for teachers and principals, whose salaries and very jobs depend on this performance indicator. It is no wonder, then, that teachers (encouraged by their principals) divert class time toward the subjects tested—mathematics and English—and away from history, social studies, art, and music. Instruction in math and English is narrowly focused on the skills required by the test rather than broader cognitive processes: Students learn test-taking strategies rather than substantive knowledge. Much class time is devoted to practicing for tests, hardly a source of stimulation for pupils.
Despite the pitfalls of NCLB, the Obama Administration doubled down on accountability and metrics in K-12 education. In 2009, it introduced “Race to the Top”, which used funds from the American Recovery and Reinvestment Act to induce states “to adopt college- and career-ready standards and assessments; build data systems that measure student growth and success; and link student achievement to teachers and administrators.” This shows what happens these days when accountability metrics do not yield the result desired: Measure more, but differently, until you get the result you want.
Both the American Left and Right have trouble admitting that some problems are insoluble for all practical purposes, such as differential group achievement. Instead, there is a search for technocratic answers to achievement gaps, as if measuring a problem with precision will lead inexorably to its solution. In such cases metric goals serve as a form of wish fulfillment.
The article elaborates on the history of the measurement paradigm, including its mis-application in other fields such as medicine and police work. I offer the concluding paragraphs in full:
Accountability metrics are less likely to be effective when they are imposed from above, using standardized formulas developed by those far removed from the activity being measured. Measurements are more likely to be meaningful when they are developed from the bottom up, not by mining engineers, so to speak, but by those at the coalface. That means asking those with the tacit knowledge that comes from experience to provide suggestions for how to improve productivity and to develop appropriate performance standards.27 Such measurement can be used to inform practitioners of their performance relative to their peers, offering recognition to those who have excelled and assistance to those who have not. Measurement instruments, such as tests, are invaluable; but they are most useful for internal analysis by practitioners rather than external evaluation by publics who may fail to understand their limits. To the extent that they are used to determine continuing employment and pay, they will almost invariably be subject to the gaming of the statistics or outright fraud.
Even when metrics can be useful, however, they are not necessarily worth assembling. One should always consider the trade-offs: the costs in employee time and energy, and their diversion from other purposes. Acquiring data is not free, nor are the costs of obtaining them easily calculated. Had he lived longer, Franz Kafka himself might have written a short story of accountability mania spiraling downward in an infinite regress until reaching a point of total entropy. The challenge for organizations is to leave room for judgment and initiative, to motivate without stifling autonomy.
In the end, there is no silver bullet, no substitute for actually knowing one’s subject and one’s organization, which is partly a matter of experience and partly a matter of unquantifiable skill. Many matters of great importance are too subject to judgment and interpretation to be solved by standardized metrics. In recent decades, too many politicians, business leaders, policymakers, and academic officials have lost sight of that distinction. To paraphrase Lewis Carroll, “if you don’t know where you’re going, any metric will take you there.”
My advice: read this entire article.
Earlier this week, Michigan Public Radio ran a story on how Manistee (MI) Area School District’s newly launched on-line charter school offers programs to 2800 students, more than twice the enrollment in the current district. The result? Manistee students have a wider array of courses to choose from and the school district raked in roughly $500,000 in additional revenue or $300/student. How did a small, rural NW Michigan district pull off such a feat? They did have help from a K12, a national charter chain who, presumably, provided the technological infrastructure the district needed to implement the program. The public radio report glossed over the specifics, but did offer this background information on how charter’s can bring new revenues to a school district:
All charter authorizers take a percentage of their charter schools’ revenue for overseeing the academies they run.
The standard is to take 3%.
For Michigan’s largest charter authorizers, Grand Valley State and Central Michigan Universities, that 3% amounts to more than $6 million apiece.
From this report it sounds as if the State funded colleges are the primary beneficiaries of the charter law in Michigan and that students, presumably, benefit from the offerings. But there are often other entities who benefit as well— and the passing mention of K12 implies the for profit entities gain from the implementation:
…Manistee leaders attracted online education giant K12 Inc, signing a five year contract to run the new Michigan Great Lakes Virtual Charter Academy.
It’s hard to know what services K12 provides and how much they are compensated for their role in overseeing “…the new Michigan Great Lakes Virtual Charter Academy.” But a report from the Detroit Free Press earlier this year noted that “…charters collect nearly $1 billion a year in state aid, often with little accountability, transparency or academic achievement.” The context of this news report was the fact that 11 of the charter authorizers in Michigan were “at risk of being suspended” because of shoddy management practices, an indication that State oversight of the program was lacking.
From my perspective, a partnership between a small rural school district strapped for revenue and an “online education giant” could be promising. Small rural districts often contract with large bus fleets to deliver students to school and many schools and colleges contract with food service firms to provide school lunch programs. If an “online education giant” is providing the technological infrastructure and software needed to manage a virtual academy it seems like a win-win. The students can enroll in more courses, no teachers lose their jobs, the “online education giant” can earn a marginal profit, the district budgets are held harmless and, perhaps, restored to previous levels, and taxpayers are relieved of absorbing additional costs.
But turning some of the operations over to an “online education giant” could be inviting that giant to assume more and more responsibility over time. If a science teacher retires, the “online education giant” could offer to have science instruction delivered on-line… or if a school board wanted to save even more money it could replace the entire staff of a school with “tutors” who would “coach” students working on-line. The “online education giant” is unlikely to be satisfied with 3% of the profit when it might be possible to collect a higher percentage or it could increase it’s profits by increasing it’s “market share”.
As one who has consulted in poor, small rural districts in New England, I can see the promise and the peril… and given the many school boards in most New England states it is easy to see that the promise of on-line partnerships as opposed to the peril. And as one who believes that close government supervision of all government spending is needed I would hope that other states learn from Michigan’s “deregulation”: for-profit entities will cut corners to make a profit if they are given the opportunity.
danah boyd, an insightful writer for Education Modern Learners and a principal researcher at Microsoft, wrote a response to the NYTImes “Room For Debate” column on the topic “Is internet addiction a concern for teenagers?” boyd’s response was an unequivocal “no”. Instead, she suggested we needed to look at the stress we are placing on our children and out over-protection of them as the problem.
This is the Catch-22 that we’ve trapped today’s youth in. We’ve locked them indoors because we see the physical world as more dangerous than ever before, even though by almost every measure, we live in the safest society to date. We put unprecedented demands on our kids, maxing them out with structured activities, homework and heavy expectations. And then we’re surprised when they’re frazzled and strung out.
For many teenagers, technology is a relief valve. (And that goes for the strung-out, overworked parents and adults playing Candy Crush, too.) It’s not the inherently addictive substance that fretting parents like to imagine. It simply provides an outlet.
The presence of technology alone is not the issue. We see much higher levels of concern about technology “addiction” in countries where there’s even greater pressure to succeed and fewer social opportunities (e.g., China, South Korea, etc.).
If Americans truly want to reduce the amount young people use technology, we should free up more of their time.
This reasoning resonates with me having witnessed the intense pressure many children face at earlier and earlier ages and the many pieces I’ve written about the need for children to have unstructured time to play with each other out of doors if possible. It also resonates because I have a sense that we are becoming more like China and Korea, who rely on standardized tests as the only means of entry into higher levels of education and accept social rigidity as a by-product. But boyd should realize that parents find themselves in their own Catch 22. When they read about parents who are arrested for allowing their children to walk home from school, when they are subjected to endless reports of child abductions and dangers their children could encounter in unsupervised environments, when their child’s actions are captured on camera every moment during the school day, when they are told by their school that if their child fails to do well on a particular test they will not be able to enroll in a particular school, and when they hear repeatedly that their child needs an impressive resume to get into a competitive college and therefore be set for life, parents get the message: Monitor your child 24/7; make sure they study hard; make certain their every moment is accounted for. This message will be very hard to overcome as long as we place a huge emphasis on standardized tests and as long as we promote fear in the media. Here’s body’s concluding paragraphs:
This is why many of our youth turn to technology. They aren’t addicted to the computer; they’re addicted to interaction, and being around their friends. Children, and especially teenagers, don’t want to only socialize with parents and siblings; they want to play with their peers. That’s how they make sense of the world. And we’ve robbed them of that opportunity because we’re afraid of boogeymen.
We’re raising our children in captivity and they turn to technology to socialize, learn and decompress. Why are we blaming the screens?
Children want time to play… and the I-Phone is their new playground.
Two recent articles, one on “Screen Addiction” by Jane Brody in the NYTimes and one on “Helicopter Parenting” by Julie Lythcott-Haims in Slate indicate how difficult free-range parenting can be in this day and age of intense competition and pervasive electronic media.
Brody’s article suggests that closer parental supervision of children is needed when it comes to computer games… and given the intentional addictiveness of those games the supervision must often be heavy-handed and invasive. Brody’s article describes research on how much time children spend in front of screens, which approaches 8 hours per day on average for 8-10 year olds. She illustrates this phenomenon by describing her husband’s challenges in trying to engage their grandchildren in conversation when transporting them to and from school.
Two of my grandsons, ages 10 and 13, seem destined to suffer some of the negative effects of video-game overuse. The 10-year-old gets up half an hour earlier on school days to play computer games, and he and his brother stay plugged into their hand-held devices on the ride to and from school. “There’s no conversation anymore,” said their grandfather, who often picks them up. When the family dines out, the boys use their devices before the meal arrives and as soon as they finish eating.
Perversely, close supervision and monitoring of a child’s every action can lead to over-parenting— which can be every bit as detrimental as “screen addiction”. Lythcott-Haims’ Slate article describes how parents who try to control every aspect of a child’s life end up disabling them:
When parents have tended to do the stuff of life for kids—the waking up, the transporting, the reminding about deadlines and obligations, the bill-paying, the question-asking, the decision-making, the responsibility-taking, the talking to strangers, and the confronting of authorities, kids may be in for quite a shock when parents turn them loose in the world of college or work. They will experience setbacks, which will feel to them like failure. Lurking beneath the problem of whatever thing needs to be handled is the student’s inability to differentiate the self from the parent.
When seemingly perfectly healthy but overparented kids get to college and have trouble coping with the various new situations they might encounter—a roommate who has a different sense of “clean,” a professor who wants a revision to the paper but won’t say specifically what is “wrong,” a friend who isn’t being so friendly anymore, a choice between doing a summer seminar or service project but not both—they can have real difficulty knowing how to handle the disagreement, the uncertainty, the hurt feelings, or the decision-making process. This inability to cope—to sit with some discomfort, think about options, talk it through with someone, make a decision—can become a problem unto itself.
Both of these parental behaviors are a manifestation of our culture that values busy-ness and competition. A busy parent is happy to have their child quiet and engaged in an activity that doesn’t involve watching television and doesn’t require their undivided attention…. and playing computer games seems innocuous. Indeed, in some cases the children’s screen addiction may be a reflection of the parents’ addiction. A parent who wants “the best” for their child will carefully schedule their activities so that they “stay out of trouble” and build a resume that will help them get into a top tier college where a diploma will lead to a good job. In both of these cases the children are robbed of the chance to interact with peers in an unstructured environment, to use their imaginations, and to daydream now and then… they are not allowed to be children.
Getting out of this trap will require a free-range parenting approach that sets clear limits on the use of screens, maintains free-time on the child’s calendar, and encourages unstructured outdoor activities with other children of similar ages and/or physical abilities. This free range parenting requires a fearlessness: a belief that their children will not be abducted if they join friends at the playground; their children will not be denied entry to a top tier college if they aren’t fully scheduled 24-7-365; that their children cannot thrive without close supervision; that their children will reject them of they say “No” to every request. Here’s hoping there are enough courageous parents to offset the laissez faire group allowing their children to sit mindlessly in front of screens or the intensely engaged group who smother their children’s curiosity and imagination by packing every minute of every day with activities.
As noted in many previous posts, public schools have been collecting massive amounts of data on individual students for decades… data that has been stored in stuffed file folders and various generations of microfiche and computer formats. This inconvenient and inconsistent method of data collection made it impossible to use group data to determine the effectiveness of teaching methods, to track an individual student’s learning, or to do systematic research in education.
The advent of cloud storage, the adoption of uniform learning standards, and the extensive use of standardized tests makes it possible to gather and analyze data systematically. This should be nothing but good news for teachers and parents… but as we’ve seen with the NSA, data collection has a dark side as well. Recent articles in the NYTimes and Atlantic describe the dilemma researchers and practitioners face in making use of the data that is now available: the reluctance of parents to have information about their children stored on line.
The Times article, “When Guarding Student Data Endangers Valuable Research” looks at the Data Dilemma from the research angle. As the writer Susan Dynarski notes, the data gathered is invaluable:
Educators parse this data to understand what is working in their schools. Advocates plumb the data to expose unfair disparities in test scores and graduation rates, building cases to target more resources for the poor. Researchers rely on this data when measuring the effectiveness of education interventions.
Noting that despite the fact that no one has hacked into the student data and despite the fact the student data is not a likely target for marketers, many legislators are proposing laws that would hamstring the efforts of researchers to draw on the data to gain a better understanding of what works and the efforts of teachers to use the data to personalize instruction. To use a phrase of one of my colleagues in Maryland, the legislators are using a shotgun to kill a mosquito. Her solution to this is to provide the Department of Education with the ability “…to impose serious penalties on districts and states as soon as they are found to have violated privacy regulations” noting that “…the states, districts and the courts then need to do the hard work of enforcing laws that protect student privacy.” A noble idea, but a non-starter in Congress who, even if they passed such a law to pacify indignant parents, would fail to provide the funding for enforcement.
The Atlantic article by Andrew Giambrone describes one way to solve this data dilemma. Given the government’s seeming inability to deal with this problem, and the given the demand for data analytics on the part of schools (e.g. a 2012 survey of educational professionals indicated that 80% of the respondents “…believed analytics would become more important in the future”), developing an acceptable means of defining appropriate use of data may fall to local districts working with eager vendors. Giambrone describes how this is happening across the country… and it calls to mind a Ted Sizer quote I used frequently: “How does change occur in education? Slowly, Carefully, and All At Once”. His concluding paragraphs underscore why the systematic collection of student data is a good idea… and why this change will happen slowly and carefully:
Jose Ferreira, the founder and CEO of Knewton, a New York-based company that develops adaptive-learning tools, says a lot of student data is going to waste right now; rather than being forgotten at the end of each school year or semester, it could be harnessed responsibly to drive learning outcomes. His company tracks students’ proficiencies across a variety of subjects, but will not share that information—even with teachers—unless explicitly authorized to do so by a student’s legal guardians.
“If you’re going to touch people’s data, it’s very important that the benefits be clear,” he explains. “‘Why should I let you collect my data? The benefits are fantastic? Now you have to reassure me you’re going to use it in a way I’m comfortable with.’”
Like Ferreira, I am convinced that reams of student data is going to waste.. but like the majority of parents, I am not yet comfortable with the way the data could and might be shared. That will take some time.