On occasion, students and reporters ask me what makes me trust or distrust folks who claim to be education researchers, and it’s a harder question to answer than one might think. As an historian with some quantitative training, I am eclectic on methods–I have no purity test other than “the evidence and reasoning have to fit the conclusion.” It’s not the existence of error: even great researchers make occasional errors, and it’s a good thing in the long run for researchers to take intellectual risks (which imply likely error/failure). Further, we all have the various myside biases cognitive psychologists write about.
But when I come across something like the following produced by the Cato Institute’s Andrew Coulson and displayed by Matthew Ladner twice on Jay Greene’s blog (including on Thursday), I start to wonder. Here’s Coulson’s chart:
Look at both vertical edges, and you will see that this is a two-vertical-axis chart, with the per-pupil costs on the left and some (unstated) measure of achievement that is labeled “[subject] scores” in percentage terms. Because one can often manipulate units and axes to leave almost any impression one might wish, I wondered if I could use the same underlying data to leave the opposite impression.
First, once I looked at Table 182 from the 2009 Digest of Educational Statistics, it became clear that the cost figure increases (supposedly the total cost of a K-12 education taken by multiplying per-pupil costs by 13) are false. If you look at the columns in the linked data (Table 182), the per-pupil costs when adjusted for inflation approximately double rather than triple as asserted in this figure. Second, there is no possible source for the approximate “0%” line from NAEP long-term trends data, unless there is an additional calculation unexplained by Coulson.
But let’s look at the real data and see if you can manipulate that to leave an opposite impression:
Whoa, Nelly! It looks at first glance from this figure that Coulson’s dead wrong: when comparing the trend lines for per-pupil costs after inflation (the green line: like that, green = money?) to reading (blue) and math (brown) trends, it looks like reading trends may not be great, but math looks to have had a pretty good return on the total investment in all K-12 education.
How did I manipulate the data to get this result? First, I chose a measure of per-pupil expenditure change that was both acceptable academically and also would shrink the apparent change: the natural log of the ratio of current per-pupil expenditures to 1971-72 per-pupil expenditures. Then I put average NAEP long-term scale scores on another academically-acceptable measure, using the starting scale score for the interval as 100 for each subject and age (1971 mean scale = 100 in reading, 1978 mean scale = 100 in math). Then I made sure the vertical axes had the “right” low-high range to contrast the greatest increases in NAEP trends (math for 9- and 13-year-olds) with a visually-shrunken per-pupil trend line: the vertical scale on the left included nothing more than the total range of the re-calculated trend scores, while the vertical scale on the right was just a little more than twice the range of the natural log of the expenditure ratios. Voila!: a figure that looks like it shows the exact opposite of what Coulson’s figure looks like it shows. Addendum 1: Two commenters misunderstood my point even with the phrase “manipulate the data” in the first sentence of this paragraph. So maybe I should make clear that, yes, the figure I prepared is also a demonstration of what not to do, except that in contrast with Andrew Coulson, I am telling you exactly how I am making the pretty colors dance to my tune. Addendum 2: Ladner responds, arguing that I am missing the forest for the trees. Or maybe that I am dodging the point, or dodging the trees (dodging a forest would be much more difficult, as some folks in Middle-Earth found out). That might be the case if my point here is about some measure of productivity. But it isn’t; it’s about presentation of data. In general, I think it’s wise to separate discussions of this sort from the substantive question. On that, it is fair to ask questions about whether or what part of the rising per-pupil spending on schooling in the past 40 years has been a good investment.1 Addendum 3: Coulson responds. I’ll let the difference between the data trees and chart manipulation forest here be an exercise for the reader, and the trees have some problems, too.2
This manipulation of data and presentation by Coulson is the type of behavior that makes me distrust not only the piece in which something like this appears but the broader work of an individual.
Notes
- Also, apropos the picture Ladner used: I have more hair on my chin than hobbits, but less on my feet. [↩]
- The un-fuzzed chart in Coulson’s response shows dotted lines for extrapolation beyond the data… uh, right. We don’t have 2009 financial data in the 2009 or 2010 Digest of Education Statistics. I think the endpoints with real data also look off by about $3K or more, depending on the data series he used, which he did not specify. [↩]
Visual rhetoric has *always* been a persuasive tool, ever since Florence Nightengales’ Rose graph used to demonstrate the casualties from the Crimean War. Before that, it was the statistics in William Petty’s Political Arithmetick. Statistics are always in service of another master.
Without context — i.e., who is using it and why — the presentation is meaningless. This is why it is not a matter of “distrust[ing] not only the piece in which something like this appears but the broader work of an individual,” but of *using* the broader work of the individual (or corporation) tp provide context for establishing meaning.
Coulson’s mismatch of modes asserts an implied connection between expenditures and achievement. Ok. But this raises more questions than it answers.
In the second graph I noticed a mismatch between the use of exponentially-presented costs (right vertical axis, unlabelled) and linear NAEP data (itself a constructed nominal variable presumed to be continuous and representative, by scholars). Context is provided, I think, through reference to academically-accepted data (itself cause for concern). Why don’t the year-ends correlate in the two graphs (2008 versus 2004)?
Glen: “academically acceptable” means I chose a method that had been used by others at some point in respectable research, not that it is appropriate for the question asked. Remember that this is an exercise in demonstrating how to manipulate presentation.
Why ending in 2004? My basic professional instincts kicked in here: the long-term trend scales changed in 2004, so trying to patch together trend lines simplistically from before to after 2004 is inappropriate.
Isn’t this more misleading, though? Taking the log of expenditures obviously mutes any rise (doubling real spending from $5,000 to $10,000 makes the natural log rise only by 8% or so). As a society, we care about what we’re actually spending. We don’t report government budgets in terms of what they’re the log of, and I’m not sure why the log of spending would ever be used for anything except if you think there’s a multiplicative relationship or if you want to interpret a parameter in terms of percentages.
Moreover, putting achievement on a Y axis such that it looks like achievement sometimes quadrupled or even went up by a factor of 10 is way more misleading than a graph implying that a 10 point gain isn’t that much.
So, the graph you’re criticizing may be misleading, but your alternative is much more so.
Stuart: Yes, that’s the point: anyone can manipulate the presentation of data with multiple axes to present conclusions that one wants to draw, as Coulson did.
The log for any variable, including spending, will be used (correctly) when it better normalizes the data for the assumptions of the statistical analysis being performed. Because no statistical analysis is forthcoming in this case, no, it shouldn’t be taken at face value.
But I don’t see that as Sherman’s point. His point is that the scales on graphs can easily be misused. For instance, what does it mean to plot costs in [fill in the year] dollars? It usually means that someone claims to have applied some adjustment for inflation. But what adjustment? The U.S. Government has changed the formula for the various inflation indexes more than 20 times since 1970. The current formula leaves out universal expenses like utilities and mortgage/rent – ostensibly because these are ‘too volatile,’ but in reality because many large companies tie employee raises (and banks tie lending rates to governments) to inflation. Wouldn’t a more honest conversion be something such as % of total money in ‘circulation,’ or something along those lines?
And while we’re at it, shouldn’t the cost of educating a student exclude some significant portion of the local education budget? Say, a big chunk of what goes to the bloviated administrative structure of the District? Almost certainly – if the goal is an honest assessment.
But that is what I took as Sherman’s point. He posited the question of whom to “trust” as an expert. The answer, in education, is no one, fully … because education research, being almost exclusively non-experimental, is almost always tainted by politics and personal bias.
Eric,
See Jeff Henig’s Spin Cycle for a good discussion of education research and politics. Henig argues that education research improves what we know, but asymptotically rather than in the heat of the moment.
As an historian, I’d disagree with the implication that research that is non-experimental is “almost always tainted by politics and personal bias.” But let me come at this from a different direction: major public-health research studies are often epidemiological–the Framingham heart study, for example. There has been a huge amount of knowledge derived from that non-experimental study. The problems with education research are different and more complicated than “almost exclusively non-experimental.”
shouldn’t the cost of educating a student exclude some significant portion of the local education budget? Say, a big chunk of what goes to the bloviated administrative structure of the District?
No, not by any means. What we care (or should care) about as a society is how much money we’re spending, not how much we’re spending after excluding all of the spending that you deem as wasteful. If we’re spending a lot more due to wasteful bureaucratic structures, that’s something we should know, not something we should ignore, as you suggest,
Yes, but it’s not exactly a refutation of Andrew Coulson’s graph to point out that you can come up with something totally different that is, by any objective measure, much more misleading.
Stuart, the decision to use a two-axis chart for arguments about ROI is either incompetent or deceptive at its base.
It is most certainly a refutation of the utterly arbitrary association of $150,000/13 with 100% and the use of some measure that hides the increase in math scores.
Why not use dollars per year, which is what we pay?
Why not pair up $150,000 with 200%, since that is what it actually corresponds to, making the comparison even worse?
Why not use the construction cost deflator rather than the GDP deflator (if that is what they used), since construction (along with energy) is a big factor in education costs?
And has anyone come up with a measure of report production per pupil?
I also missed your point — I took it to be a “corrective.” But Addendum 1 clarified this for me. Much of the time, I miss the point of sarcasm and irony — I don’t think these are communicated very well in cyberspace.
Instead of trying to make some meaningless graph in an attempt to discredit the author, why don’t you take data that you trust and see if an honest graph made by you presents the same picture. I think this approach would be more meaningful and useful to readers. I think there is a good chance your results would look very similar to the original graph.
I like to see apples compared to apples not apples compared to rocks.
Frank
Frank,
You mean with a separate post on the substantive question of ROI? Already done.