Rating and ranking objects seems to be baked into our DNA, and while I find it problematic from many vantage points I accept the practical reality that it isn’t going away any time soon. This compulsion to rate and rank is particularly problematic when the basis for the rankings is narrow: for example, the simplistic report cards based on test scores results in teaching-to-the-test in order to boost a school’s rating and thereby boost its ranking.
When State departments began focussing on test scores in the early 1990s and the media obligingly provided “horse race” ranking charts that overlooked the fact that there was often no statistically significant difference between the 6th ranked school and the 96th ranked school, I encouraged our local board and staff to design a comprehensive set of metrics that we would use within our district in lieu of the narrow and simplistic ones the State was using. I hoped that my defining our own comprehensive quality standards we would be able to put the narrow and simplistic state standards in perspective. I would like to be able to say that this was an easily accomplished task that made an impact in the three districts where I attempted this… but that would not be entirely true. I found that developing “Quality Standards” was difficult process at the community, board, and staff level in each of the districts. Some wanted the results to focus on only measurable data; some wanted to extrapolate from their own personal experiences with a particular school; some wanted to focus on cost-effectiveness; and some felt that what schools accomplished for children could not be measured in any fashion in the short run and would only be known in the indefinite future. In general, staff members were the most resistant to any kind of measurement and representatives of the business community were enthusiastic about finding metrics that could be used to measure cost-effectiveness.
All of this was brought to mind by a recent news report from WMUR on recently released rankings of NH high schools from Niche. As one who served as Superintendent in two NH districts and did consulting in several others, I clicked through the rankings and found them to be congruent with my gut-level sense of how various high schools would rank: their standing would reflect the parent’s education level and affluence of the districts. But when I dug more deeply into Niche’s rating algorithm I was pleasantly surprised. It incorporated the factual and survey data in a fashion that was fair and transparent. This section from their web page was particularly good from my perspective (my emphases are added):
Our rankings are different, and for good reason. We believe that the quality of a school or district should be measured, at least in part, by the parents and students who actually go there. They should also be measured by hard data and across a number of key factors so that no one factor dominates a ranking. Most importantly, they should be measured by their results. The most unique thing about our rankings is that they incorporate student outcomes.
We go to great lengths to ensure that our rankings represent a comprehensive assessment of each school or district. Each Survey Score used in each ranking represents a composite score of several different survey questions pertaining to that topic. Of the dozens of statistical factors that go into each ranking, no one factor accounts for a majority of the overall weight. For example, in our Best Public High Schools ranking, no single factor accounts for more than 15 percent of the overall ranking.* Our outcome data doesn’t just look at college matriculation. It looks at the quality of those colleges.
From what I can tell Niche is not underwritten by Gates, the Koch Brothers, or Pearson and it’s “team” appears to be comprised of millennial computer programmers from CMU in Pittsburgh, PA.
From my perspective, it would be wonderful if Arne Duncan adopted the Niche algorithm and invited feedback from districts across the country on ways to improve it. A crowd-sourced rating system that ensured that “no one factor accounts for a majority of the overall weight” would be FAR superior to the blunt instrument of standardized test scores that dominates the ratings and rankings of students, school districts, and teachers. By using a thoughtful and (to the best of my knowledge) independently developed rating system as the framework for an improved system of rating schools the USDOE could lead states and local districts in the same direction.
Ross Douthat, the NYTimes resident conservative columnist wrote a column this morning lamenting the end of The New Republic, an “old journalism” magazine that reported on liberal arts as well as liberal politics. He contrasts the “old journalism” of The New Republic with new on-line blog sites like Vox.com and finds the “new journalism” lacking.
Unlike our era’s ascendant data journalism, it also never implied that technocracy was somehow a self-sustaining proposition, or that a utilitarianism of policy inputs and social outcomes suffices to understand every area of life…
So when we talk about what’s being lost in the transition from old to new, print to digital, it’s this larger, humanistic realm that needs attention. It isn’t just policy writing that’s thriving online; it’s anything that’s immediate, analytical, data-driven — from election coverage to pop culture obsessiveness to rigorous analysis of baseball’s trade market.
I was heartened to read that Mr. Douthat laments the passing of a non-utilitarian liberal arts magazine and I hope that he will transfer that lament to education where we are witnessing an emerging emphasis on utility. I believe the noxious notion that colleges should be measured by job placements will have the same effect on schooling as the “new journalism” is having on the breadth of our understanding… and we’ve already seen that the emphasis on test scores has pushed the arts to the sidelines in public schools… Mr. Douthat’s closing paragraph summarizes all that is wrong with the “new journalism”:
The peril isn’t just that blithe dot-com philistines will tear down institutions that once sustained a liberal humanism. It’s that those institutions’ successors won’t even recognize what’s lost.
I couldn’t help but note that education, like journalism, is an institution that is being torn down by “…blithe dot-com philistines” and “those institutions’ successors won’t even recognize what’s lost.”
The USDOE announced earlier this week that it plans to require states “…to develop rating systems for teacher preparation programs that would track a range of measures, including the job placement and retention rates of graduates and the academic performance of their students.” Unsurprisingly one of the metrics that USDOE is mandating as part of the rating system is some form of Value Added measures using standardized tests.
A NYTimes article by Mokoto Rich outlines the rationale for this mandate, and it’s full of subtle reinforcements of “reform” advocates, which are flagged in red bold italics. Early in the article Rich quotes Arne Duncan who frames this efforts as a “…nothing short of a moral issue” because when they begin their careers teachers often “…have to figure out too much on the job by themselves.” The solution to this problem is to withhold grant funds from teacher preparation programs that do not pass muster. These paragraphs from the article exemplifies the attitude of the USDOE toward teacher preparation programs, most of which are offered in state funded colleges and universities:
Education experts said the new regulations were necessary to spur change, particularly among colleges that draw most of their tuition revenue from candidates enrolled in education programs.
“I think you need to wake up the university presidents to the fact that schools of education can’t be A.T.M.s for the rest of the college or university,” said Charles Barone, policy director for Democrats for Education Reform, a group that pushes for test-based teacher evaluations and has battled teachers’ unions. (Nudge, nudge, wink, wink— the UNIONS are the problem with introducing “reform”.)
It is difficult to argue against more regulations and accountability, but there are several aspects of this proposal that are troubling:
- It reinforces the notion that teachers are the primary reason schools are “failing”: If this initiative was part of a multi-pronged comprehensive plan to increase the public’s respect for the teaching profession it would be very helpful to public education. Instead, this plan makes it sound as if State colleges that prepare students are to blame for the struggles that teachers encounter in their first year, that they are to blame for the low standardized test scores that children in poverty achieve (but presumably NOT responsible for any of the high test scores in affluent districts), and that they accept unqualified teacher candidates in order to line their pockets.
- It reinforces the notion that standardized tests can be used to measure teacher performance: VAM is a sham and the USDOE’s continued insistence that it be incorporated in accountability measures doesn’t change that reality. Oh… and Rich reinforces the “reform” meme that States CHOSE this methodology of student accountability and will therefore CHOOSE this methodology to measure teacher performance with this quote: “Although the rules do not require tests, 42 states, the District of Columbia and Puerto Rico have agreed with the Department of Education to develop teacher performance ratings that include test scores.”
- It implicitly reinforces the notion that programs like TFA are superior to “traditional” teacher training programs: One of the underreported changes that RTTT introduced was a deemphasis on districts reporting on the number of “Highly Qualified” teachers they had on the staff, a change that coincided with the promotion of programs like TFA and the expansion of deregulated for-profit charter schools. It will be interesting to see how TFA can sustain it’s standing as a quality teacher preparation program given the fact that most TFA classroom teachers leave the field after 2 years…. and even more interesting to see how USDOE takes action against State Boards who award charters to schools headed by CEOs who lack teaching credentials.
- It implies that the ultimate value of college education is employability: All of the accountability schemes I’ve read about to date imply that employability is more important than versatility: that is, learning a specific skill set is more important than learning how to learn. This is a terrible assumption to make because it assumes the entry skills required in today’s workforce are not going to change and this is clearly NOT the case in public education nor is it true in any field. USDOE and undergraduate colleges cannot predict what the workforce requirements will be in 2050 any more than my college could have foreseen that I’d be sitting at home with access to the library of congress listening to a collection of customized music selected for me by a computer algorithm sharing my views with readers across the country and (based on the information WordPress provides) across the globe. The research skills University of Pennsylvania required for my dissertation were obsolete 20 years later and the skills they require today will change in the next 20 years.
- It assumes that “market incentives” driven by the rating system will increase the number of STEM teachers. The article includes this priceless quote based on the daft logic that job placement metrics will somehow enable teacher training institutions to motivate undergraduates to change their majors:
Using metrics like job placement makes common sense, said Arthur Levine, president of the Woodrow Wilson National Fellowship Foundation, which administers a program for people training to be high school teachers, because it would force programs to train people for actual job openings.
“Education schools and universities educate a lot of elementary school teachers, an area that’s glutted,” Mr. Levine said. “On the other hand, we definitely need science and math teachers, which they don’t prepare.”
Accountability is needed… but NOT the “reform” driven accountability advocated by the USDOE that will continue to demonize teachers as the cause of “failing schools” and assumes that STEM teachers will materialize if the metrics are right…
A post from Diane Ravitch yesterday provided a link to a report by UCLA professor Noel Enyedy titled “Personalized Instruction: New Interest, Old Rhetoric, Limited Results, and the Need for a New Direction For Computer Mediated Instruction”.
At the outset of the report, Enyedy offers his definition for Personalized Instruction and differentiates it from Personalized Learning:
It is critical to note that “Personalized Instruction” is not the same as “personalized learning,” even though promoters and vendors of technological systems often use the terms interchangeably. Personalized instruction focuses on tailoring the pace, order, location, and content of a lesson uniquely for each student—as when a software program introduces a quiz at some point during instruction and then, based on the student’s score, either presents the student with new material or with a review of material not yet mastered. It is a rebranding of the idea of individualized instruction first promoted in the 1970s, before the widespread availability of personal computers.
Personalized learning, on the other hand, places the emphasis on the process of learning as opposed to attending exclusively to the delivery of content. Personalized learning refers to the ways teachers or learning environments can vary the resources, activities, and teaching techniques to effectively engage as many students as possible—as when, for example, students with a stronger intuitive understanding of the topic are assigned to small groups and given a challenging task to independently extend their understanding while the teacher concurrently works directly with a small group of students who have less prior knowledge of the topic. This interpretation of “personal” does not imply that each student receives a unique educational experience, but instead that students are provided with multiple entry points and multiple trajectories through a lesson.
Enyedy, after emphasizing that the scope of this study is limited to personalized instruction, does an admirable job of outlining the rationale for expanding the use of technology supported “Personalized Instruction”. He describes and analyzes the shortcomings of the factory school model, noting its inability to provide students with the “critical thinking and independent agency” needed to function in a democracy.
In his description of on-line and blended personalized instruction, Enydey identifies one major problem with its implementation to date: inequity.
Research has found that schools in less affluent areas are more likely to use the technology for remedial instruction and for drill and practice, whereas affluent schools are more likely to use technology in ways that advance problem solving and conceptual understanding. These choices, often left up to individual teachers, have serious implications for equity within the classroom and across schools and districts.
Enydey then attempted to perform a meta-analysis of personalized instruction models, an analysis that he acknowledged was limited because there were not a sufficient number of K-12 systems in place. This meant the lion’s share of the studies he analyzed were at the college level where student agency was arguably higher. But the meta-analysis also incorporated one other flaw, which this paragraph flags:
The study examined the standardized test scores for the same three blended learning schools compared with three other schools in the district to see if the gap between high and low achievers was closed by using blended instruction for one year. The study showed that neither blended learning nor face-to-face instruction in this district was particularly successful at improving the performance of lower achieving students. The gap closed 3% in the blended learning schools compared with the 2% improvement in the comparison schools that used conventional teaching methods.
The flaw is that Enydey, like most policy makers, cannot shake the age-based grade-level paradigm that is the basis of the factory school! If we are to abandon the factory model, we have to also abandon the notion that time is constant and learning is variable…. and therefore abandon the use of our current standardized tests to measure “student learning”. That is, we should not measure how much a student has learned in one year, but devise a means of measuring the extent to which a student is making progress in learning-how-to-learn. To date, we have no means of measuring that and so we continue to measure what it EASY to measure instead of what is IMPORTANT to measure, relying on a factory metric instead of a more holistic metric.
Another flaw in the study is the failure to acknowledge and advocate for more access to technology in schools and, more importantly, in the homes of students nd teachers. This paragraph touches on that topic:
In one RAND study,40 based on the actual expenditures of schools that transitioned to an Intelligent Tutoring System for Algebra 1, the cost increased an average of $120 per student for the one course. This increase was reduced to $70 per student per class in schools with a good existing technological infrastructure. However, as many as half the schools in implementation studies undertaken by SRI Education41 and RAND42 were found to need a substantial investment in their technological infrastructure before they could take advantage of Personalized Instruction.
Presiden Obama’s support for a new surtax on phone services to raise $3 billion for schools is a step in the right direction if we ever hope to address the inequities among schools… but in order to provide each and every student with the same opportunities to learn, as emphasized repeatedly in this blog, we need to provide each and every student and teacher with high speed internet at their doorsteps. Until every child can access the power of the internet in their home and every teacher can access the comprehensive data packages outside of school we will be stuck with the models for teaching and learning we have today.
Enydey does note near the end of his paper that the current models in place: on-line instruction and personalized instruction, may be replaced with something different in the future:
The type of computer technology that many believe will lead to transformational change will be technologies built around the process of learning and that attempt to enhance human-to-human interaction, not supplant it: technologies that spark conversations and inquiry; technologies that support these conversations with tools for visualization, simulation, analysis and communication; technologies that allow the students to create physical or computational objects; and technologies that allow students to share their ideas and solutions with their peers and larger social networks for feedback and refinement. There are many promising new models for how computers should be used to support learning.
These promising new models are predicated on two major changes: one a change of thinking on our part and the other an investment in technology. We need to change our thinking by abandoning the factory school model, which will lead to the abandonment of age-based student cohorts and the abandonment of standardized tests as the measure of “learning”. And, we need to make a he investment in our nation’s technology infrastructure by ensuring that each school and home has the means of providing personalized instruction AND learning to students.
An article by Anna North in today’s NYTimes, “Can Brain Science Be Dangerous”, suggests that some of the widely publicized research on brain science, and particularly the brightly colored brain scans that often accompany the articles on this subject, are drawing people towards conclusions that are not true or accurate. My take on the article was that the brain science findings are akin to Rorschach tests: they invite the viewer to see whatever they want to see and project it as reality. This is most troubling when scientific findings about the brain are applied to broad social issues like poverty. Some research suggests that poverty creates stresses in the lives of children that irreversibly limit brain growth while other research suggests that if interventions are applied early and effectively those same stresses can be overcome. This can enable politicians to use “science” to support whatever their views are. With this as a backdrop, one American sociologist, Susan Sered, offered this observation:
…used incorrectly, neuroscience might spread the view that poor people are lesser than others, that they are irrevocably debilitated by their experiences with poverty — or, conversely, that if they fail to respond to programs that science says will help them, it must be their own fault.
North sums up the concerns of social and neurological scientists in her concluding paragraph:
People… seem to find neuroscience extremely persuasive, even when it’s wrong. And this may be part of what critics fear — that images and facts about the brain are so powerful, they can make us believe things we really shouldn’t.
I would take it a step further and suggest that “images and facts about the brain are so powerful, they can support our wrong beliefs and close our minds to contradictory and new findings.”