Chingos/Peterson vouchers study does not support authors’ claims

This morning, I recommend downloading and reading Sara Goldrick-Rab’s review of the Brookings Institute Chingos/Peterson study of vouchers’ effects on college attendance. Bottom line (from p. 7):

Policymakers and practitioners interested in the effectiveness of school voucher programs should indeed attend to the results of this study, which—contrary to the interpretation of the authors—convincingly demonstrates that in New York City a private voucher program failed to increase the college enrollment rates of students from low-income families.

Update (9/29/12): The link above also has Chingos and Peterson’s response and Goldrick-Rab’s rebuttal. One of the questions between them is about the focus on African-Americans as a subgroup without highlighting the non-significance of the overall effect (0.6% increase in college enrollment for those offered vouchers, where the standard error of that estimate is 2.2%). If you look just at the sample from the study, the non-African-American sample has to have decreased college attendance by 4.1%, but what is the standard error (i.e., can we be reasonably certain that if the increase for African-Americans is statistically significant, then is the decrease for everyone else also statistically significant)? I decided to simulate a batch of 1000 samples that each had a subgroup consisting of 42% exposed to a random effect of 7.1% increased college attendance (3.4% standard error) and where the total sample was exposed to a random effect of 0.6% increased college attendance (2.2% standard error), and where the total sample and the 42% subsample effects varied normally. Bottom line: a simulated median and mean decrease of the low 4% range in college attendance for the 58% of the simulated samples whose effects were inferred. Depending on the random seed for the simulation, between 80 and 82% of the simulations have negative effects, and the 95th percentile effect was about 3.3%. This makes sense if you think about the sample: if a minority of a population has strong effects in one direction, and the overall effect is zero, the remainder would have weaker effects in the other direction (and here I mean strong and weak in both the effect’s magnitude and the probabilistic sense of not likely to be zero).

Is it likely that the college-matriculating effect of offering vouchers to African American families is clearly positive but a weaker negative effect for everyone else? Having thought about this study for over a month, I can’t think of the relevant mechanisms that would give you that result.

Further update (12/25): I am trying to learn a little R while running through John Kruschke’s text on Bayesian stats (impression thus far: very good), so here’s a little code if you want to run this yourself. ((I apologize for the lack of indentation; I couldn’t figure out how to do that within HTML’s code tags.))

# Specify known values from study data
muT = 0.006 # effect for total population
sigmaT = 0.022 # effect estimation standard deviation
muAA = 0.071 # effect for African American participants
sigmaAA = 0.034 #effect estimation standard deviation for African Americans
mixPropAA = 0.42 # proportion of study participants who were African American
#
# Simulation
#
# Designate an arbitrarily large number of simulated samples.
nSimSamples = 10000
# Set aside correctly-sized vectors in which to store the simulation results.
simSampleTrecord = vector( length=nSimSamples ) # total effect
simSampleAArecord = vector( length=nSimSamples ) # African American effect
simSampleNBrecord = vector( length=nSimSamples ) # effect for non-African Americans
#
# Generating MC "sample" estimates from the study results.
# sampleIdx is an index .
for ( sampleIdx in 1:nSimSamples ) {
# Generate an effect value for the simulated sample from the study results.
sampleDiffTotal = rnorm( 1 , muT , sigmaT )
sampleDiffAA = rnorm (1, muAA, sigmaAA)
# Generate an effect for non-African Americans.
sampleDiffNB = (sampleDiffTotal - sampleDiffAA * mixPropAA)/ (1-mixPropAA)
# Store the sample data in the vectors created above.
simSampleTrecord[ sampleIdx ] = sampleDiffTotal
simSampleAArecord[ sampleIdx ] = sampleDiffAA
simSampleNBrecord[ sampleIdx ] = sampleDiffNB
}
# Make a histogram for each variable of interest.
hist( simSampleTrecord )
hist( simSampleAArecord )
hist( simSampleNBrecord )
# Summary stats
summary( simSampleTrecord )
summary( simSampleAArecord )
summary( simSampleNBrecord )

Way into the weeds: I’m using a normal distribution for effects, which may or may not be the best choice in theory–attending/not-attending college is a dichotomous variable, and a few other options might work better. I suspect the gist of these results would stay the same with a different choice of distribution: if a minority of a population has strong effects in one direction, and the overall effect is zero, the remainder would have weaker effects in the other direction.

Even further into the weeds: Another approach would be to generate a very large simulated sample through a two-step process: first simulating a set of sample parameters, and then drawing a simulated individual outcome based on the sample mean for each simulated sample. This would require a distribution choices for individual outcomes, not just sample parameters.

Update (1/7/2015): The research now has appeared in the Journal of Public Economics.