Home > Uncategorized > Big Data’s Potential Oversold in Some Areas, Overlooked in Others

Big Data’s Potential Oversold in Some Areas, Overlooked in Others

March 28, 2016

For at least two decades technology advocates have promoted the notion that what we now call Big Data would be the magic bullet that would help all students achieve success in public education. But as an editorial in yesterday’s San Diego Union-Tribune notes Big Data was oversold in some areas and overlooked in others.

Many “Freakonomics” readers probably assumed there must be a way to analyze voluminous education data and come up with something that has positive potential…

But it hasn’t worked out that way. Even as Big Data has transformed industry after industry, it hasn’t seemed to yield valuable insights into public education — and the education establishment often resists using empirical methods to improve schools.

As readers of this blog realize, I am among those in the “education establishment” who embraces empirical methods to improve schools, but also among those in the “education establishment” who is deeply dismayed at the public’s belief that the primary purpose of “Big Data” is to find a way to develop a standardized means of rating and ranking schools and teachers instead of using “Big Data” to gather empirical evidence on what works to motivate students in the classroom.

The Union-Tribune writers note that there are some researchers who are using “Big Data” to sift through information gathered on the hiring of teachers to identify the key determinants that administrators could use to select the best applicants from a large pool of candidates:

(A correlational analysis of) the backgrounds of nearly 8,000 teachers seeking jobs with the Washington, D.C., school district with the subsequent records of those the district hired has come up with some powerful findings. This research is detailed in “Teacher Applicant Hiring and Teacher Performance: Evidence from D.C. Public Schools,” a 75-page National Bureau of Economic Research working paper by Brian Jacob, Jonah E. Rockoff, Eric S. Taylor, Benjamin Lindy and Rachel Rosen.

The report found that two qualities — good GPAs in college and impressive 30-minute teaching “auditions” in which applicants present a lesson plan — strongly correlated with classroom effectiveness. This isn’t necessarily surprising. A good GPA reflects some combination of intelligence and hard work. The importance of auditions reflects what’s been learned from industrial psychology for decades — that some individuals are more able to translate “book learning” into strong job performance — as the report notes.

This ISN’T surprising to me. My dissertation, written over four decades ago, determined that a demanding and presumably more rigorous application process narrowed the pool of applicants but did nothing to winnow the pool. That is, the GPAs, test scores on a creativity assessment, and experience of non-applicants was no different than the pool of applicants despite the belief that making an application process more demanding. In that era there was no “Big Data” to draw from and no possible way to do the kind of correlational analysis done by the research team cited in this report. But then, as now, I had faith that gathering and analyzing data might help the district identify the best teachers, help teachers determine the best way to connect with individual students, and help students gain a better understanding of themselves.

But there is one other factor that was in play then, as now: I had no faith that standardized achievement tests could be used in any way shape or form. Why? Because the first course I took in graduate school was an Education Statistics course taught by a professor who had a deep skepticism of the way the tests were designed… and his first assignment reinforced that notion in all of his students. After lecturing us on the flaws of multiple choice tests after we read the first chapter on that topic, he distributed copies of the Stanford tests used in the early 1970s to rate and rank public schools in Philadelphia and, consequently, used by many of the suburban schools. The test had roughly 70 questions. Our assignment was to use the knowledge we now had after one class to find five errors in the construction of test questions. When we gathered a week later, we had collectively identified eleven flawed questions, three of which were glaringly incorrect.

I am confident that the tests used today to rate and rank schools, teachers, and students are no better… and presumably any journalist, politician or “reformer” could do the same exercise as my classmates in 1970 and come to the same conclusion. But, alas, everyone wants to believe that tests yielding data that is seemingly precise can make clear distinctions in the collective and individual performance of students and teachers.

Categories: Uncategorized Tags: , ,
%d bloggers like this: