“Weeks/days of learning” is well-intended bad interpretative factoid

The Institute of Education Sciences has released a new Evaluation of the Teacher Incentive Fund, or TIF, (after two years), which is generally solid research by Mathematica Policy Research, at least at a quick first read today. The main findings:

  • Most of the experimental part of TIF was implemented by the schools.
  • Some parts of the program were more difficult to implement (e.g., higher performance pay for a more limited group of educators), or more difficult to maintain.
  • Part of the logic model was hard to confirm, especially the issue of educator understanding of their opportunities to earn higher pay.
  • The bottom-line effects on student performance were weak: 0.04 standard deviations in math, 0.03 in reading. If you obsess about p values, only the association with reading was statistically significant.

I say that this is generally solid research… until you get to the part of the document where the main effect size for reading is translated into a statement that teacher and principal performance pay is associated with three additional “weeks of learning” in reading. Mathematica is using a common, well-intended attempt to translate the abstract concept of effect size into something a general audience can understand. This translation has become more common in the last few years.

It is also bad interpretation. That doesn’t mean that documents should not attempt the translation for a general audience, but there are problems with just using terms like “three weeks of learning” as naked representations. To cut to the chase:

  • “Weeks (or days) of learning” avoids the most important part of recontextualizing effect sizes: comparing the effect size in an individual study with effect sizes from empirical research in the same domain — i.e., if you are translating your research findings for use in the real world, how does this intervention or policy compare with other interventions or policies that are realistic alternatives?
  • “Weeks (or days) of learning” implies more accuracy than is realistic; it is hard to spot a difference between 3 and 4 weeks of learning (and for those tempted to publish “days of learning,” under no circumstance in the real world can research make an empirically-justified distinctions between 15 and 16 days of learning). This study does not report standard errors for the estimates, but Mathematica does report the effect sizes under different models (or sensitivity to model assumptions), and the variations easily surpass 0.01 standard deviations, or the equivalent of one week of learning. If you want to talk about weeks of learning for this study, we need to understand that depending on the model used the inferred effect on reading for the first cohort in the second year is likely to range somewhere between 0 and 4 weeks of learning. That interval may looks odd, but it’s a better representation of the research than the statement in the report.

Reporters reading such findings can ask the authors two questions before writing stories, as a consequence:

  • What are the effect sizes of potential alternatives, either in standard-deviation units or weeks/days of learning?
  • What is the error of the estimate — or the confidence intervals, in weeks or days of learning?

Continue reading ““Weeks/days of learning” is well-intended bad interpretative factoid”

Who will tell you that your utopian school-improvement gambit has been tried before?

Jack Schneider is the star of Amy Scott’s Marketplace piece last Friday on the XQ competition to redesign high schools:

[P]eople have tried to reinvent high school before.

“That idea is as old as students sitting in desks and a teacher lecturing at the front of the classroom,” said education historian Jack Schneider, an assistant professor at the College of the Holy Cross.

In the mid-20th century there was a movement to make high schools much bigger, with more courses from which to choose, Schneider said. Fifty years later, people decided high schools were too big. The Gates Foundation spent $1 billion promoting smaller schools. Neither was a smashing success.

High school is far from perfect, Schneider said, but it’s been evolving all along.

I could also point out the idea of computer literacy as a fifth “R” in A Nation at Risk (1983) — something Mayor deBlasio probably is unaware of — let alone the New American Schools Development Corporation created under President George H.W. Bush, the Coalition of Essential Schools founded by Ted Sizer, and so forth. Laurene Powell Jobs may succeed where others have stumbled, but we should not ignore the fact that high schools have changed over time, and others have likewise had utopian visions for school redesign.

Today, we also have Stephen Sawchuk’s article on the decline of history of education courses in teacher education programs. Which raises the question, if courses and jobs in the field are declining, where will historians of education be when the next ahistorical proposal for education reform comes down the pipeline?1 I have a (paywalled) article from earlier this year, Prophet or Fool: The Professional Position and Role of Historians of Education, and the brief answer is, less from inside colleges of education and more from outside colleges of education. Jack is in a liberal-arts college with a liberal-arts education curriculum, like another Jack historian of education, others are in history departments, and others who write with an historical perspective are journalists such as Audrey Watters and Dana Goldstein.

Those of us who care about new scholars in the history of education need to prepare the next generation of scholars to be able to migrate across fields, and those of us who care about ongoing scholarship in the field must value good work wherever it comes from.


  1. There is the other question: in a world where our courses and jobs are declining, why are we still portrayed as the Saruman of teacher ed? []

Kudos to the U.S. Department of Education

Hold onto that blog entry title, folks — it’s going to be pretty rare around these parts. Two events this weekend justify the praise:

  1. The Saturday-morning release of institution-level data on loans, paybacks, income, and other data, including some data specific to Pell-grant recipients. Even more to its credit, the department made tools available so that other organizations such as ProPublica could create their own interfaces. This comes two years after announcing that the Administration wanted to create a rating-and-ranking system for colleges and universities. I was against it at the time, and I am delighted that not only did the Obama administration back away from the rating-and-ranking idea, but demonstrated how it didn’t need the “crack cocaine of today’s generation of education reformers” (my language from 2013). As Sara Goldrick-Rab noted, this data release is not a panacea, but it’s a great start.
  2. The clever shifting of FAFSA tax-record information today to a prior-prior-year basis — that is, students entering college in 2016 only need their or their parents’ 2015 taxes (which are already filed) rather than 2016 taxes, which would have been the case without today’s change. Why is it clever? It only required changing the timeline so that those who expect to be students in fall 2016 can start filing the FAFSA next month–previously, FAFSA (and student-need calculations) were only available beginning in January. Again, shifting the tax-year basis for student aid is a big step — and just one step — in addressing the nightmares many students face with financial aid. But it’s still a big step.

Why a state supreme court struck down a charter-school law

A week ago, the Washington State Supreme Court ruled that the use of common-school funds for charter schools violated the state constitution, and that the state’s 2012 charter-school law as a whole was invalidated as a consequence. I think some charter-school supporters are a bit emotional, from Rick Hess to Robin Lake. Lake went further than Hess and suggested that judicial campaign donations from the Washington Education Association essentially rigged the decision: “most of the judges accepted campaign contributions from the Washington Education Association, the state’s biggest teachers’ union—and also the plaintiff.”

Continue reading “Why a state supreme court struck down a charter-school law”

Causation cadge

Note: this post is a sort of mental marker, as I am partway thinking through a particular issue and do not want to lose my place.

One of my colleagues, Micki Chi, has divided causal modeling into two sorts, the type with narrative structures and the type without, which she calls emergent and has some evidence is more difficult for students to learn.

Continue reading “Causation cadge”