Rating and ranking objects seems to be baked into our DNA, and while I find it problematic from many vantage points I accept the practical reality that it isn’t going away any time soon. This compulsion to rate and rank is particularly problematic when the basis for the rankings is narrow: for example, the simplistic report cards based on test scores results in teaching-to-the-test in order to boost a school’s rating and thereby boost its ranking.
When State departments began focussing on test scores in the early 1990s and the media obligingly provided “horse race” ranking charts that overlooked the fact that there was often no statistically significant difference between the 6th ranked school and the 96th ranked school, I encouraged our local board and staff to design a comprehensive set of metrics that we would use within our district in lieu of the narrow and simplistic ones the State was using. I hoped that my defining our own comprehensive quality standards we would be able to put the narrow and simplistic state standards in perspective. I would like to be able to say that this was an easily accomplished task that made an impact in the three districts where I attempted this… but that would not be entirely true. I found that developing “Quality Standards” was difficult process at the community, board, and staff level in each of the districts. Some wanted the results to focus on only measurable data; some wanted to extrapolate from their own personal experiences with a particular school; some wanted to focus on cost-effectiveness; and some felt that what schools accomplished for children could not be measured in any fashion in the short run and would only be known in the indefinite future. In general, staff members were the most resistant to any kind of measurement and representatives of the business community were enthusiastic about finding metrics that could be used to measure cost-effectiveness.
All of this was brought to mind by a recent news report from WMUR on recently released rankings of NH high schools from Niche. As one who served as Superintendent in two NH districts and did consulting in several others, I clicked through the rankings and found them to be congruent with my gut-level sense of how various high schools would rank: their standing would reflect the parent’s education level and affluence of the districts. But when I dug more deeply into Niche’s rating algorithm I was pleasantly surprised. It incorporated the factual and survey data in a fashion that was fair and transparent. This section from their web page was particularly good from my perspective (my emphases are added):
Our rankings are different, and for good reason. We believe that the quality of a school or district should be measured, at least in part, by the parents and students who actually go there. They should also be measured by hard data and across a number of key factors so that no one factor dominates a ranking. Most importantly, they should be measured by their results. The most unique thing about our rankings is that they incorporate student outcomes.
We go to great lengths to ensure that our rankings represent a comprehensive assessment of each school or district. Each Survey Score used in each ranking represents a composite score of several different survey questions pertaining to that topic. Of the dozens of statistical factors that go into each ranking, no one factor accounts for a majority of the overall weight. For example, in our Best Public High Schools ranking, no single factor accounts for more than 15 percent of the overall ranking.* Our outcome data doesn’t just look at college matriculation. It looks at the quality of those colleges.
From what I can tell Niche is not underwritten by Gates, the Koch Brothers, or Pearson and it’s “team” appears to be comprised of millennial computer programmers from CMU in Pittsburgh, PA.
From my perspective, it would be wonderful if Arne Duncan adopted the Niche algorithm and invited feedback from districts across the country on ways to improve it. A crowd-sourced rating system that ensured that “no one factor accounts for a majority of the overall weight” would be FAR superior to the blunt instrument of standardized test scores that dominates the ratings and rankings of students, school districts, and teachers. By using a thoughtful and (to the best of my knowledge) independently developed rating system as the framework for an improved system of rating schools the USDOE could lead states and local districts in the same direction.
Joe Nocera’s column in today’s NYTimes laments the “disruption” taking place at The New Republic” where a dot-com tycoon has decided to focus on the bottom line instead of the magazine’s traditional mission of providing thought provoking and insightful articles written by seasoned journalists.
After reading this, I saw a chilling parallel to what is happening in public schools. Traditional schools, like traditional journalism, was based on the premise that students needed a well rounded education that provided depth as well as breadth. The new owner of The New Republic, in seeking profits, is increasing that predominantly print magazine into an on-line magazine and willingly trading “long form” writing for bar graphs and bullets. After reading the article, I offered the following comment:
I hope that readers can see that an analogous transformation is happening in public education where deregulated for-profit charters are being touted as “disruptive forces”. When public schools become “profit centers” overseen by highly paid “CEOs” whose “success” is based on standardized test scores their primary function will be to satisfy investors. It should be no surprise to parents that these new “profit centers” will focus on test scores and abandon the arts and other “frills” which do not lend themselves to easy and inexpensive metrics. Oh well… at least our future McSchools will be preparing good subscribers to the McJournalism of the internet. McSchool graduates might not do well on international science tests but they’ll at least know a lot about flatulence!
When “clicks” are the ultimate measure of readership, there will be a race to the bottom in content… and “when test scores” are the ultimate measure of education there will be a squeeze toward standardization. In both cases, breadth and depth is sacrificed at the alter of profits.
A few years ago John Stewart ran a segment on the lifestyle of “greedy teachers” that made the rounds among my colleagues in NH. The segment featured Samantha Bee as an intrepid interviewer pursuing a story to show American how their tax dollars are being siphoned to pay for the opulent lifestyle of teachers, and it followed teachers home in their compact cars to walk-up apartments in the city and small suburban homes payed for in part by their spouses earnings. The segment came out around the time Scott Walker was gutting teachers’ contracts in WI asserting that taxpayers were being fleeced by unions who demanded outlandish wages and benefits.
In the intervening years, economists and “blue ribbon commissions” on the teaching profession have periodically issued reports lamenting the diminishment of respect for teachers and the need to upgrade their compensation in order to attract and retain teachers. Mokoto Rich reports on the latest such report in today’s NYTimes. This one issued by the National Council on Teacher Quality, flags the reality that most teachers today have to work for years before they earn “middle class compensation”, emphasizing that it often takes decades for teachers to progress through the step and track system to earn the top dollar.
Rich then rehashes the tired debate between the “reformers” who advocate higher compensation for teachers who achieve high VAM scores for several consecutive years and the unions who want to see more money thrown on the current step and track system OR have the steps compressed so that higher salaries can be achieved earlier.
As readers of this blog realize there IS another way to reward teachers based on their performance as determined by an array of measures… While the mechanism described in this earlier blog post has only been implemented in few districts it is an effort to apply research on teacher’s motivations and learning curves to compensation.
One last word on Rich’s report: she, the NCTQ, and many “experts” assume that turnover is a function of the changing values of today’s workforce:
With young people changing careers more often than previous generations, schools need to consider how to reward teachers earlier, said Segun Eubanks, director of teacher quality at the National Education Association, the nation’s largest teachers’ union.
“We know that unlike the profession that many of us entered 30 years ago, the idea that we’re going to recruit a work force today with the design to keep them for 30 years isn’t realistic in this labor market,” Mr. Eubanks said.
The linchpin in this analysis is the phrase “in this labor market“, which assumes that the way things are today will not change. I believe that we DO need to change the compensation plans for teachers and we DO need to hold teachers in higher esteem given their hard work and dedication. If we could make the compensation higher and more rational, public education would provide a labor market that would attract and retain individuals who desire a job that is secure, rewarding, and beneficial. If we accept “this labor market”, we are accepting jobs that are short term, driven by the desire to make money quickly, and limited to increasing student performance on tests instead of increasing each student’s ability to thrive in a democracy.
Tomorrow’s NYTimes features a thought provoking and fascinating article on recent findings that link genes to early intervention programs for troubled children. “The Downside of Resilience” by Jay Belsky describes international longitudinal research on the role genetics plays in determining whether a child is affected by developmental experiences. After examining the impact of early intervention programs on groups of children from an array of racial and ethnic backgrounds in studies that were conducted independently from each other, researchers have concluded that genetics DOES play a role in determining who will benefit most from intervention:
Every gene contains two so-called alleles — one from each parent. There is evidence that people who carry certain variations of these alleles have a greater chance of developing particular disorders. For instance, short alleles of the gene 5-HTTLPR, which transports serotonin, have been linked to depression, while long alleles of the dopamine-receptor gene DRD4 have been linked to attention deficit hyperactivity disorder. Intriguingly, these “risk” genes also turn out to be associated with heightened sensitivity to environmental conditions. Children who carry either or both of them appear to be most adversely affected by negative experiences, and seem to benefit most from supportive ones. Children without them seem relatively immune to the effects of both supportive and unsupportive environments.
My mind immediately went to the thorny question Belsky posed near the end of the article after elaborating on the various studies:
This brings up a challenging ethical question: Should we seek to identify the most susceptible children and disproportionately target them when it comes to investing scarce intervention and service dollars?
Belsky answers in the affirmative, while suggesting more research be done simultaneously. He then elaborates on his reasoning:
Those who value equity over efficacy will object to the notion of treating children differently because of their genes. But if we get to the point where we can identify those more and less likely to benefit from a costly intervention with reasonable confidence, why shouldn’t we do this? What is ethical, after all, about providing services to individuals for whom we believe they will not prove effective, especially when spending taxpayers’ money?
I appreciate Belsky’s acknowledgement that the quandary we face is in part based on the reality that funds for intervention will be limited. Most arguments for equitable treatment— including many advanced in this blog— are based on the rosy assumption that because we have a moral imperative to provide equity we will raise whatever money is needed to ensure that we can achieve equity. And most who argue for early intervention— including me— base their advocacy on the assumption that the intervention plans would be customized based on the unique needs of each child. Finally, a case can be made that we are already on the path of providing medically-based programming for children: IEPs are based on the findings of a school psychologist and 504 plans are often framed based on the recommendations of physicians. On coldly logical basis it seems to me that adding genetic counselors to the list of “intervention advisors” is not that much of a leap… and yet the notion that genetics might play a role in public policy DOES seem chilling… especially if we are unable to develop some means of intervening in cases where children are NOT affected by their developmental experiences.
Belsky concludes his essay with this paragraph that opens the doors to even more questions:
For now, after half a century of childhood interventions that have generated exaggerated claims of both efficacy and ineffectiveness, we need to acknowledge the reality that some children are more affected by their developmental experiences — from harsh punishment to high-quality day care — than others. This carries implications for scientists evaluating interventions, policy makers funding them and parents rearing children.
The last phrase is particularly problematic. IF we can determine a childs’s responsiveness to developmental experiences through a genetic test, are we ready to include such a test as part of the initial pediatric screening? I’ll leave you with that question to ponder….
A post from Diane Ravitch yesterday provided a link to a report by UCLA professor Noel Enyedy titled “Personalized Instruction: New Interest, Old Rhetoric, Limited Results, and the Need for a New Direction For Computer Mediated Instruction”.
At the outset of the report, Enyedy offers his definition for Personalized Instruction and differentiates it from Personalized Learning:
It is critical to note that “Personalized Instruction” is not the same as “personalized learning,” even though promoters and vendors of technological systems often use the terms interchangeably. Personalized instruction focuses on tailoring the pace, order, location, and content of a lesson uniquely for each student—as when a software program introduces a quiz at some point during instruction and then, based on the student’s score, either presents the student with new material or with a review of material not yet mastered. It is a rebranding of the idea of individualized instruction first promoted in the 1970s, before the widespread availability of personal computers.
Personalized learning, on the other hand, places the emphasis on the process of learning as opposed to attending exclusively to the delivery of content. Personalized learning refers to the ways teachers or learning environments can vary the resources, activities, and teaching techniques to effectively engage as many students as possible—as when, for example, students with a stronger intuitive understanding of the topic are assigned to small groups and given a challenging task to independently extend their understanding while the teacher concurrently works directly with a small group of students who have less prior knowledge of the topic. This interpretation of “personal” does not imply that each student receives a unique educational experience, but instead that students are provided with multiple entry points and multiple trajectories through a lesson.
Enyedy, after emphasizing that the scope of this study is limited to personalized instruction, does an admirable job of outlining the rationale for expanding the use of technology supported “Personalized Instruction”. He describes and analyzes the shortcomings of the factory school model, noting its inability to provide students with the “critical thinking and independent agency” needed to function in a democracy.
In his description of on-line and blended personalized instruction, Enydey identifies one major problem with its implementation to date: inequity.
Research has found that schools in less affluent areas are more likely to use the technology for remedial instruction and for drill and practice, whereas affluent schools are more likely to use technology in ways that advance problem solving and conceptual understanding. These choices, often left up to individual teachers, have serious implications for equity within the classroom and across schools and districts.
Enydey then attempted to perform a meta-analysis of personalized instruction models, an analysis that he acknowledged was limited because there were not a sufficient number of K-12 systems in place. This meant the lion’s share of the studies he analyzed were at the college level where student agency was arguably higher. But the meta-analysis also incorporated one other flaw, which this paragraph flags:
The study examined the standardized test scores for the same three blended learning schools compared with three other schools in the district to see if the gap between high and low achievers was closed by using blended instruction for one year. The study showed that neither blended learning nor face-to-face instruction in this district was particularly successful at improving the performance of lower achieving students. The gap closed 3% in the blended learning schools compared with the 2% improvement in the comparison schools that used conventional teaching methods.
The flaw is that Enydey, like most policy makers, cannot shake the age-based grade-level paradigm that is the basis of the factory school! If we are to abandon the factory model, we have to also abandon the notion that time is constant and learning is variable…. and therefore abandon the use of our current standardized tests to measure “student learning”. That is, we should not measure how much a student has learned in one year, but devise a means of measuring the extent to which a student is making progress in learning-how-to-learn. To date, we have no means of measuring that and so we continue to measure what it EASY to measure instead of what is IMPORTANT to measure, relying on a factory metric instead of a more holistic metric.
Another flaw in the study is the failure to acknowledge and advocate for more access to technology in schools and, more importantly, in the homes of students nd teachers. This paragraph touches on that topic:
In one RAND study,40 based on the actual expenditures of schools that transitioned to an Intelligent Tutoring System for Algebra 1, the cost increased an average of $120 per student for the one course. This increase was reduced to $70 per student per class in schools with a good existing technological infrastructure. However, as many as half the schools in implementation studies undertaken by SRI Education41 and RAND42 were found to need a substantial investment in their technological infrastructure before they could take advantage of Personalized Instruction.
Presiden Obama’s support for a new surtax on phone services to raise $3 billion for schools is a step in the right direction if we ever hope to address the inequities among schools… but in order to provide each and every student with the same opportunities to learn, as emphasized repeatedly in this blog, we need to provide each and every student and teacher with high speed internet at their doorsteps. Until every child can access the power of the internet in their home and every teacher can access the comprehensive data packages outside of school we will be stuck with the models for teaching and learning we have today.
Enydey does note near the end of his paper that the current models in place: on-line instruction and personalized instruction, may be replaced with something different in the future:
The type of computer technology that many believe will lead to transformational change will be technologies built around the process of learning and that attempt to enhance human-to-human interaction, not supplant it: technologies that spark conversations and inquiry; technologies that support these conversations with tools for visualization, simulation, analysis and communication; technologies that allow the students to create physical or computational objects; and technologies that allow students to share their ideas and solutions with their peers and larger social networks for feedback and refinement. There are many promising new models for how computers should be used to support learning.
These promising new models are predicated on two major changes: one a change of thinking on our part and the other an investment in technology. We need to change our thinking by abandoning the factory school model, which will lead to the abandonment of age-based student cohorts and the abandonment of standardized tests as the measure of “learning”. And, we need to make a he investment in our nation’s technology infrastructure by ensuring that each school and home has the means of providing personalized instruction AND learning to students.