How to think clearly about that clever econometrics policy paper

A late August NBER working paper by Joshua Goodman, Oded Gurantz, and Jonathan Smith argues that if every high school student took common college admissions tests twice, that would shrink the income-relevant college enrollment gap by 20%. New York Times reporter Sahil Chinoy wrote up the story, and the headline repeated the eyebrow-raising import of the paper’s conclusion: “A Surprisingly Simple Way to Help Level the Playing Field of College Admissions.”

Chinoy quoted only one person other than the authors, and the story came out the same morning as the paper. One could be excused if the NYT story felt a bit like one of those “imminent medical breakthrough” stories about various cures that are just over the horizon. Many of those imminent breakthroughs never make it through medical trials, and one might wonder, is it really true that just taking the SAT or ACT a second time would have such a dramatic effect if every high school student did it?1

Spoiler warning: the research is clever — the paper is very clever — and yet state policymakers and those who advise them should not assume that steps to encourage test-retaking would have dramatic benefits in either equalizing enrollment across the state or boosting a state’s college enrollment.2

Continue reading “How to think clearly about that clever econometrics policy paper”


  1. The conclusion in this working paper does not suggest requiring a retake but speculates on various mechanisms to encourage retaking admissions tests. []
  2. There is a reason for this conclusion at the beginning: This blog post has been brought to you by the letter We Can Torture Finer Points of Analysis Until They Bark Like Chihuahuas. Also, my thanks to Joshua Goodman, Oded Gurantz, and Daniel Klasik for the Twitter conversation that helped clarify some things for me. []

Dorn vs. Oliver

In May 2015, more than three years ago, the major segment of John Oliver’s Last Week Tonight focused on standardized testing. Eighteen minutes is a lifetime on television, and in his segment Oliver argued that testing was not worth the pain or money. Critics in education policy world responded with serious quibbles, because I guess then we were fact-checking comedians. Yes, that’ll stop Abbott and Costello: explain that there really is no team where Who’s on first. But in 2015, we had this quaint notion that checking facts could shame those who didn’t speak the truth.

Oh, how we wish that had turned out to be so.

Continue reading “Dorn vs. Oliver”

First day of school, admin version

Today at Arizona State was the first day of classes in the fall semester. I know: a Thursday. Maybe Arthur Dent couldn’t get the hang of Thursdays, but we can.

Continue reading “First day of school, admin version”

The post-Janus world, first legal response

Abood did not abide.1

As I explained elsewhere,2 teachers unions and union activism will survive the Janus decision that ends mandatory representation fees for public employees who are not members of unions that represent their bargaining unit.3 When I was thinking about the post-Janus world, I thought primarily of the broader strategies of unions that could no longer function on semi-autopilot with agency fees, at least in terms of their relationships with bargaining units.4

But less than a week later, we are also seeing the potential legal response by teachers unions, in Florida.

Continue reading “The post-Janus world, first legal response”


  1. Abood was the 1970s Supreme Court ruling that mandatory representation fees do not violate the rights of public-employees who are represented by unions but not members. Yes, I’m willing to go this far for a bad pun. []
  2. Note for 7/2: I have some edits I need to make to that piece, primarily adding sources that I didn’t have the chance to add last week. []
  3. Yes, that’s a mouthful. Every part of that is required. []
  4. This is not necessarily what many unions did, but it is a temptation, to treat the relationship as transactional. []

How to write an external review letter

It is summer, and many of my tenured colleagues around the country are writing external review letters evaluating the scholarship of candidates for tenure or promotion. In an era with a shrinking proportion of academic jobs holding the chance of tenure, it becomes more important for those of us who hold tenure to understand the role of the tenure and promotion process.1 This blog entry has a brief description of the role of external review letters, the advice I generally give colleagues when they have the chance to nominate external reviewers for either the tenure/promotion-to-associate step or promotion-to-full step, and what is important for an external review letter to contain.

The brief, tl;dr version of my plea to letter-writers:

  1. Spend 80% of your time/words explaining how you look at the candidate’s vitae and scholarship in the context of your and the candidate’s area, written for fellow academics who are in other disciplines — either a college/university-wide faculty committee or a provost may treat your letter as critical to the decision. Trust that a provost and faculty can count articles; what your readers need is your disciplinary (or interdisciplinary!) perspective.
  2. Read the writing samples and comment on them — explain what you learned from them, or if necessary, why you did not learn from them.
  3. Update your vitae before sending it with your review letter, and include everything that puts your position in context — that is, brag to make clear your expertise as a member of the field. Do not omit editorial board service, service as a program officer in a funding agency, or the like.
  4. Keep your tone civil.
  5. Do not accept an invitation to write a letter if there is a clear, discoverable conflict of interest and/or if the candidate explicitly asked you to be willing to be an external reviewer.

Continue reading “How to write an external review letter”


  1. … and not botch it through carelessness. []