Posts Tagged ‘Testing’

Robots Cannot Replace Humans When it Comes to Grading Essays— But No Matter! They are Cheap, Fast, and Unbiased.

July 17, 2018 Leave a comment

NPR recently ran a story by Tovia Smith on the use of robots (or more precisely computers) to grade student essays and found to no educators’ surprise, that they were not not up to the task. The story opens with this quiz:

Multiple-choice tests are useful because:

A: They’re cheap to score.

B: They can be scored quickly.

C: They score without human bias.

D: All of the above.

It would take a computer about a nano-second to mark “D” as the correct answer. That’s easy.

But now, machines are also grading students’ essays. Computers are scoring long form answers on anything from the fall of the Roman Empire, to the pros and cons of government regulations.

From this point forward in the story, Ms. Smith provides an account of the expansion of the use of computer-graded essays, work done by MIT research affiliate, Les Perelman, who has developed algorithms that generate nonsense responses to machine-graded essays that yield high scores, a rebuttal to Mr. Perelman’s work by ETS, who argue, in effect, that if someone is smart enough to game to the essays they deserve the high grade, and the “cat-and-mouse” game underway to catch students who use algorithms in states that have adopted computerized grading.

At the end of the report, Ms. Smith hits on the real problem with computerized grading: it compels teachers to teach students formulaic writing.

Indeed, being a good writer is not the same thing as being a “higher-scoring GRE essay writer,” says Orion Taraban, executive director of Stellar GRE, a tutoring company in San Francisco.

“Students really need to appreciate that they’re writing for a machine … [and when students] agonize over crafting beautiful, wonderfully logically coherent and empirically validated paragraphs, it’s like pearls before swine. The computer can’t appreciate what this person has done and they don’t get the score that they deserve.”

Instead, Taraban tutors students to give the computer what it wants. “I train them in fabricating evidence and fabricating fake studies, which is a lot of fun,” he says, quickly adding, “but I also tell them not to do this in real life.”

For example, when writing a persuasive essay, Taraban advises students to use a basic formula and get creative. It goes something like this:

A [pick any year] study by Professor [fill in any old name] at the [insert your favorite university] in which the authors analyze [summarize the crux of the debate here], researchers discovered that [insert compelling data here] … and that [offer more invented, persuasive evidence here.] This demonstrates that [go to town boosting your thesis here!]”

It results in a kind of mad-lib writing that is anything but artful, thoughtful, or pleasing to read. But it is cheap, easy, unbiased, and unbiased…. and the ultimate triumph of efficiency over excellence.



Can Philadelphia Ever Be Freed from Charter Mania? It Depends on the School Board Developing a Spine

July 16, 2018 Leave a comment

Diane Ravitch wrote a post yesterday drawing on a commentary written by Lisa Haver and Deborah Grill, two activists in Philadelphia for In the commentary, Mss. Haver and Grill describe the machinations of the Philadelphia School Board who recently took control of charter schools in the city after politicians determined that School Reform Commission was failing in its mission to improve the schools after 17 years of oversight. I was hopeful that the newly installed School Board would insist that charter schools adhere to the same standards and regulations as public schools. But, alas, it appears that school board members are “negotiating” standards and regulations with charter operators behind closed doors, presumably based on the fact they negotiate with teachers behind closed doors. But negotiating standards and regulations are not the same as negotiating wages and working conditions. Nor are they the same as negotiating contracts with vendors who provide indirect services to schools and students. In short, there is no rationale for negotiating standards behind closed doors or negotiating them at all. If teachers and students in public schools have different standards than students in privately operated charters the playing field is clearly NOT level… and the students who attend schools with the lowest standards will clearly suffer. Here’s hoping the Philadelphia School Board develops a spine.

Bill Gates Millions Get “Lackluster Results”… And Christiansen Institute Sees Innovation as the Answer

July 14, 2018 Leave a comment

Christiansen Institute (CI) fellow Thomas Arnett writes thoughtful blog posts that buttress the CIs belief that technology-based disruption is the way forward if we hope to improve public schools. In his recent post in the CI’s weekly digest, Mr. Arnett offers his perspective on the recently released study of Bill Gates’ initiatives, describing their findings as “lackluster” and poses two questions based on these findings:

Following a multi-year, multi-million-dollar initiative by the Gates Foundation to boost school performance through better teacher evaluation systems, RAND’s researchers found that the initiative failed to produce significant gains in student outcomes. When a promising education reform like this one misses the mark, where should education leaders turn next? If we step back and look at the big picture, what options do we really have for dramatically improving schools?

In his examination of “the big picture” Mr. Arnett astutely observes that teachers have a limited time frame. He uses a principle of economics called the “production possibilities frontier” to illustrate this reality.

Here’s a simplified example to illustrate the concept: Suppose a hypothetical non-profit called Basic Aid makes two goods—bread and shirts—for people in need. If the non-profit has its employees spend all their time producing bread, it can churn out nine thousand loaves a month. Alternatively, Basic Aid could schedule its employees to spend all their time making shirts and produce seven thousand shirts a month. Most months, however, the non-profit has its employees split their time between these two forms of production.

The graph below represents Basic Aid’s production options. At point A, Basic Aid only makes bread. At point B it only makes shirts. Connecting those two points is a curve that shows all the ways Basic Aid could maximize its production of bread and shirts given its available workforce. This means Basic Aid can make tradeoffs to produce any combination of bread or shirts that lies on or below the curve. But it is impossible for Basic Aid to produce combinations of bread and shirts that are beyond the blue curve. Put simply, the curve represents Basic Aid’s production possibilities frontier.

How does this apply to schools? Teachers, like the imaginary non-profit, have a zero-sum time budget. I was heartened to see an acknowledgement that schools have a time budget that is more limited than it’s financial budget, but was disappointed to see that Mr. Arnett had no appreciation for the driving force behind how time is spent in schools operated by “reformers”. You see, in Bill Gates view and the view of most “reformers” the best way to determine a teacher’s impact was by examining changes in test scores. This, in turn, led to teachers squandering their time budgets on test preparation. If the only metric that counts is changes in test scores, it’s no surprise that teachers worked on test preparation… and no surprise that the results of their efforts were “lackluster”. Why? Because children have their own time budgets. They begin their schooling at different places and learn at different rates and tests expect all children to start at the same place and proceed at the same rate year-after-year. Anyone who has raised or taught children realizes that they grown and mature at different rates and that growth rates often have no bearing on where a child will ultimately end up. Standardized tests, though, do not take these individual differences into account. To make matters worse, many of the “reforms” that emerged from this test driven world developed scripts for teachers to follow based on the premise that learning was uniform… in effect that children’s individual “time budgets” would somehow change. This just in Mr. Gates: if time is a constant, learning will be variable no matter how you try to write a script wishing it were otherwise.

If the “reformers” want to see changes in performance, that learning should be constant, they need to accept the fact that time should be a variable.


NPE Offers a Grading System for the States that Makes Sense

July 9, 2018 Leave a comment

The Network for Public Education (NPE), the public education advocacy group founded by Diane Ravitch, has used the letter grading system beloved of “reformers” to illustrate how states are performing in their efforts to resist two changes “reformers” are seeking: the expansion of deregulated charter schools and vouchers. In a brief overview of their work, the authors provide several paragraphs underscoring the overarching purpose of public education and offer this paragraph describing the effects of “reform” advocates who want to privatize the existing system of education and thereby undercut democratic local governance:

The attack on public education is also an attack on equal opportunity and civil rights. Although privatization advocates claim that private schools advance the quality of education, this is a tenuous argument to make in the face of the reality that too often there is little to no public accountability, fiscal transparency or maintenance of civil rights protections for students in privatized programs. History is replete with battles fought and sacrifices made to protect the civil rights and ensure the equality of opportunity for all students regardless of race, ethnicity, religion, gender, disability or other immutable characteristics. The proliferation of privatization programs in the states and the redirecting of public resources for the benefit of a small percentage of the student population belies this principle of equality of opportunity for all students. Privatization in public schools weakens our democracy and often sacrifices the rights and opportunities of the majority for the presumed advantage of a small percentage of students.

They conclude with an overview of the purpose of their report card:

This report card… provides a vital accounting of each state’s democratic commitment to their public school students and their public schools, by holding it accountable for abandoning civil rights protections, transparency, accountability and adequate funding in a quest for “private” alternatives. It is designed to give citizens insight into the extent of privatization and its intended and unintended consequences for our students and our nation.

If critics of NPE’s findings— likely to be Red State legislators and Governors— argue that their grading system is too simplistic, they might want to look at the grading “systems” they use to conclude that public education is failing and their belief that “running schools like a business” is the solution.

Christian Science Monitor Provides Excellent Overview of NYC Testing Debate

July 7, 2018 Leave a comment

In “Keep the Test! A Debate Flares Over Exam Based Public High Schools“, Christian Science Monitor  staff writers Stacy Teacher Khadaroo and Harry Bruinius provide a balanced overview of the complex issue of the use of a single standardized test to place students in “elite” public high schools. The city is attempting to provide a better racial and ethnic balance in its best high schools where nearly 7 out of 10 students in the school district are African-American or Latino but only 1 out of 10 “earn” places in these schools. I place the word “earn” in quotation marks because they gain a place in these competitive schools based on one and only one factor: their score on the city’s Specialized High School Admissions Test (SHSAT).

The point-counterpoint approach used by the writers provides a series of conundrums that arise from this policy  and allows the reader to see that defining merit is tricky. It illustrates how the replacement of the SHSAT will aggrieve one minority group, Asian Americans, in order to address the under-representation of two other groups. The most compelling quote came from an Asian American alum, who noted an irony in the ongoing debate:

Ted Chang says he and his wife, a graduate of an exam school, are for the mayor’s plan even though their children attend school in a neighborhood that would end up sending fewer students. “There’s something truly ironic about getting the alumni associations of our most popular science schools to coalesce around a test that social scientists have concluded is a very weak and inaccurate measure of academic potential,” he writes in an email.

Economic and educational justice will remain out of reach as long as high scores on standardized tests are conflated with “merit”. We need better metrics if we want a fairer and just economic and political system.

Open mindedness Essential for Democracy and Capitalism but Under-emphasized in test-driven schools

June 30, 2018 1 comment

DC Miracle Story Evidence of Traction of “Fake News”

June 30, 2018 Comments off

A few days ago AP writer Ashram Kahlil wrote an article titled “DC’S Public Schools Go from Success Story to Cautionary Tale“, a story that was picked up by NPR and some other mainstream news outlets. But alas, Time magazine is unlikely to run a cover story with Michelle Rhee sitting on a dunce stool or holding a broken broom.

In 2008, both Time and Newsweek offered overs depicting then rising star Michelle Rhee, the no-nonsense DC Superintendent who pledged to clean up the public schools in that city by implemented a test-and-punish policy that garnered support among those who thought schools needed to be operated using a no nonsense “business” approach and negative attention from anyone who actually worked in schools and realized that instead of a clean sweep their schools needed new floors, new lighting, and more money.

Since 2008, funding for schools has diminished, in some cases in real dollars and in all cases in terms of actual funding… and the consequences of test-and-punish has not been the improvement of test scores but rather the expansion of corruption in the administration of those high stakes tests. And DC has had its eyes blackened badly. As Mr. Kahlil reports:

As recently as a year ago, the public school system in the nation’s capital was being hailed as a shining example of successful urban education reform and a template for districts across the country.

Now the situation in the District of Columbia could not be more different. After a series of rapid-fire scandals, including one about rigged graduation rates, Washington’s school system has gone from a point of pride to perhaps the largest public embarrassment of Mayor Muriel Bowser’s tenure.

This stunning reversal has left school administrators and city officials scrambling for answers and pledging to regain the public’s trust.

A decade after a restructuring that stripped the decision-making powers of the board of education and placed the system under mayoral control, city schools in 2017 were boasting rising test scores and a record graduation rate for high schools of 73 percent, compared with 53 percent in 2011. Glowing news articles cited examples such as Ballou High School, a campus in a low-income neighborhood where the entire 2017 graduating class applied for college.

Then everything unraveled.

An investigation by WAMU, the local NPR station, revealed that about half of those Ballou graduates had missed more than three months of school and should not have graduated due to chronic truancy. A subsequent inquiry revealed a systemwide culture that pressured teachers to favor graduation rates over all else — with salaries and job security tied to specific metrics.

The internal investigation concluded that more than one-third of the 2017 graduating class should not have received diplomas due to truancy or improper steps taken by teachers or administrators to cover the absences. In one egregious example, investigators found that attendance records at Dunbar High School had been altered 4,000 times to mark absent students as present. The school system is now being investigated by both the FBI and the U.S. Education Department, while the D.C. Council has repeatedly called for answers and accountability.

It takes a long time to inculcate a culture of support, but a culture of fear can be implemented rapidly… and once that culture is in place it is hard to change. And that culture is especially hard to change when “salaries and job security tied to specific metrics” and those metrics can be manipulated by those who will be damaged the most: the administrators and politicians who based their careers and campaigns on their ability “…to improve public education.”

And who implemented this culture that resulted from salaries and job security tied to specific metrics?

As Mr. Kahlil reports in his closing paragraphs… it was none other than Michelle Rhee:

Critics view the problems, particularly the attendance issue, as an indictment of the entire data-driven evaluation system instituted a more than a decade ago when then-Mayor Adrian Fenty took over the school system and appointed Michelle Rhee as the first chancellor. Rhee’s ambitious plan to clear out dead wood and focus on accountability for teachers and administrators landed her on the cover of Time magazine holding a broom. But now analysts question whether Rhee’s emphasis on performance metrics has created a monster.  

Readers of this blog know the answer to that question: there is no doubt that the test-and-punish methods supported by Ms. Rhee and her follow reformers created a monster… but it’s serving their purposes: it is creating the impression that public schools are not only “failing” based on those test scores, but they are now “corrupt” because of the actions of a handful of administrators whose continued employment required them to boost them.

And here’s one fact that remains the same today as it was in 2008: the teachers who work in poverty stricken urban and rural districts like DC are giving their hearts and should to the jobs and the administrators in those same schools are being over backwards to support them. But a cover article lionizing public school teachers and principals is not nearly as compelling as one showing that an inexpensive one-size-fits-all solution is the best way to fix schools.