Look at both vertical edges, and you will see that this is a two-vertical-axis chart, with the per-pupil costs on the left and some (unstated) measure of achievement that is labeled “[subject] scores” in percentage terms. Because one can often manipulate units and axes to leave almost any impression one might wish, I wondered if I could use the same underlying data to leave the opposite impression.<\/p>\n First, once I looked at\u00a0Table 182<\/a>\u00a0from the 2009 Digest of Educational Statistics, it became clear that the cost figure increases (supposedly the total cost of a K-12 education taken by multiplying per-pupil costs by 13) are false. If you look at the columns in the linked data (Table 182), the per-pupil costs when adjusted for inflation approximately double rather than triple as asserted in this figure. Second, there is no possible source for the approximate “0%” line from\u00a0NAEP long-term trends data<\/a>, unless there is an additional calculation unexplained by Coulson.<\/p>\n But let’s look at the real data and see if you can manipulate that to leave an opposite impression:<\/p>\n Whoa, Nelly! It looks at first glance from this figure that Coulson’s dead wrong: when comparing the trend lines for per-pupil costs after inflation (the green line: like that, green = money?) to reading (blue) and math (brown) trends, it looks like reading trends may not be great, but math looks to have had a pretty good return on the total investment in all K-12 education.<\/p>\n How did I manipulate the data to get this result? First, I chose a measure of per-pupil expenditure change that was both acceptable academically and also would shrink the apparent change: the natural log of the ratio of current per-pupil expenditures to 1971-72 per-pupil expenditures. Then I put average NAEP long-term scale scores on another academically-acceptable measure, using the starting scale score for the interval as 100 for each subject and age (1971 mean scale = 100 in reading, 1978 mean scale = 100 in math). Then I made sure the vertical axes had the “right” low-high range to contrast the greatest increases in NAEP trends (math for 9- and 13-year-olds) with a visually-shrunken per-pupil trend line: the vertical scale on the left included nothing more than the total range of the re-calculated trend scores, while the vertical scale on the right was just a little more than twice the range of the natural log of the expenditure ratios. Voila!: a figure that looks like it shows the exact opposite of what Coulson’s figure looks like it shows. Addendum 1<\/strong>: Two commenters misunderstood my point even with the phrase “manipulate the data” in the first sentence of this paragraph. So maybe I should make clear that, yes, the figure I prepared is also\u00a0<\/em>a demonstration of what not\u00a0to do, except that in contrast with Andrew Coulson, I am telling you exactly how I am making the pretty colors dance to my tune. Addendum 2<\/strong>: Ladner responds<\/a>, arguing that I am missing the forest for the trees. Or maybe that I am dodging the point, or dodging the trees (dodging a forest would be much more difficult, as some folks in Middle-Earth found out). That might be the case if my point here is about some measure of productivity. But it isn’t; it’s about presentation of data. In general, I think it’s wise to separate discussions of this sort from the substantive question. On that, it is<\/em>\u00a0fair to ask questions about whether or what part of the rising per-pupil spending on schooling in the past 40 years has been a good investment. ((Also,\u00a0apropos\u00a0the picture Ladner used: I have more hair on my chin than hobbits, but less on my feet.)) Addendum 3<\/strong>: Coulson responds<\/a>. I’ll let the difference between the data trees and chart manipulation forest here be an exercise for the reader, and the trees have some problems, too. ((The un-fuzzed chart in Coulson’s response shows dotted lines for extrapolation beyond the data… uh, right. We don’t have 2009 financial data in the 2009 or 2010 Digest of Education Statistics<\/em>. I think the endpoints with real data also look off by about $3K or more, depending on the data series he used, which he did not specify.))<\/p>\n This manipulation of data and presentation by Coulson is the type of behavior that makes me distrust not only the piece in which something like this appears but the broader work of an individual.<\/p>\n","protected":false},"excerpt":{"rendered":" On occasion, students and reporters ask me what makes me trust or distrust folks who claim to be education researchers, and it’s a harder question to answer than one might think. As an historian with some quantitative training, I am eclectic on methods–I have no purity test other than “the evidence and reasoning have to […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"jetpack_post_was_ever_published":false},"categories":[4,18],"tags":[],"class_list":["post-4675","post","type-post","status-publish","format-standard","hentry","category-research","category-the-academic-life"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/pag0MB-1dp","_links":{"self":[{"href":"https:\/\/shermandorn.com\/index.php?rest_route=\/wp\/v2\/posts\/4675","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/shermandorn.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/shermandorn.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/shermandorn.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/shermandorn.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=4675"}],"version-history":[{"count":21,"href":"https:\/\/shermandorn.com\/index.php?rest_route=\/wp\/v2\/posts\/4675\/revisions"}],"predecessor-version":[{"id":4687,"href":"https:\/\/shermandorn.com\/index.php?rest_route=\/wp\/v2\/posts\/4675\/revisions\/4687"}],"wp:attachment":[{"href":"https:\/\/shermandorn.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=4675"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/shermandorn.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=4675"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/shermandorn.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=4675"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}![\"\"](\"https:\/\/shermandorn.com\/wp-content\/uploads\/2012\/03\/Coulson-fudge-original.jpg\" "\\"Andrew")
<\/a><\/p>\n![\"Comparison](\"https:\/\/shermandorn.com\/wp-content\/uploads\/2012\/03\/Coulson-fudge-redone-1024x616.png\" "\\"\\"")
<\/a><\/p>\n