{"id":1427,"date":"2008-09-09T00:56:20","date_gmt":"2008-09-09T04:56:20","guid":{"rendered":"http:\/\/shermandorn.com\/wordpress\/?p=1427"},"modified":"2008-09-09T00:56:20","modified_gmt":"2008-09-09T04:56:20","slug":"cold-permutations","status":"publish","type":"post","link":"https:\/\/shermandorn.com\/?p=1427","title":{"rendered":"Cold permutations"},"content":{"rendered":"<p>First, to provide a minor update on <a href=\"http:\/\/www.shermandorn.com\/mt\/archives\/001386.html\">this morning&#8217;s news items<\/a>:<\/p>\n<ul>\n<li>Semi-success on the reserving-time front. I had a lunch meeting and then a 3 pm meeting, and the time in between was too short to do much, so I exchanged one parking sticker for another. Whee. At least my wonderful grad student assisting with the journal did a monster job helping on a long MS, giving my head-cold-affected mind a much easier job going through the next article. I <em>WILL<\/em> climb on top of this mountain of work. Just not today.<\/li>\n<li>It&#8217;s a semi-full-blown cold now. Proof: I should be asleep, and I&#8217;m exhausted, but I can&#8217;t sleep.<\/li>\n<\/ul>\n<p><!--more--><\/p>\n<hr width=\"25%\">\n<p>I&#8217;ve been trying to wrap my mind around permutation tests and exchangeability for about a week, and I figure that my typical head-cold mentality may be the best shot I can take at it both in terms of the orthogonal way I think at way-too-late-on-a-head-cold evening and also the fact that once I&#8217;m up this late and in this state, no student or MS author wants me to be making decisions right now. (For the record, I&#8217;m on antihistamines. I know, I know: Never take Benadryl and grade. No. That&#8217;s not funny, not even in my state of mind.)<\/p>\n<p>A few weeks ago, I was pondering the NYC achievement gap controversy, a debate over the summer that among other things spawned a <a href=\"http:\/\/www.tcrecord.org\/Content.asp?ContentId=15366\">Teachers College Record commentary by Jennifer Jennings and me<\/a> (available just to subscribers for now, but to the world in a few weeks). And while the limits on TCR commentaries and op-eds require a fairly narrow argument, I kept thinking about trends and time series data as I looked at the New York City Department of Education&#8217;s claims. I kept thinking to myself, <em>There has to be something an historian can contribute to this debate that is specific to the way historians think<\/em>. I&#8217;ll probably write something at length when I&#8217;m more coherent and have some time, but there <em>was<\/em> an obvious answer that came to mind: to historians, the order of events matter. An argument about causality depends on contingency which depends on a sequence. (Historians often focus on contingency rather than causality, except when we&#8217;re playing the counterfactual game. The obvious answer to the question, &#8220;What caused Gore&#8217;s defeat in 2000?&#8221; is &#8220;everything, or almost everything.&#8221;) The sequence doesn&#8217;t prove causality (or contingency), but it&#8217;s necessary.<\/p>\n<p>That logic is usually not applied in policy. In the case of New York City, as is typical in this type of reform publicity, someone pointed to a time series of data and claim, &#8220;Aha! See this trend? Ignore its tentative nature: it&#8217;s <em>PROOF<\/em> that we&#8217;re on the right track.&#8221; One obvious problem with the NYC data is the reliance on threshold-passing percentages; that&#8217;s the focus of the TCR commentary. But the <a href=\"http:\/\/www.nysun.com\/files\/doeppt.pdf\">NYC Department of Education made claims<\/a> about the achievement gap more broadly, and the data is a lot messier than the folks in Tweed would state. Below are three permutations of the &#8220;z-scores&#8221; of achievement gaps (the differences in Black-White means on the 4th-grade state math tests, scaled to the population&#8217;s standard deviation). One is the <em>real<\/em> time series that runs between 2002 and 2008. The other two are permutations. Before you look for the data (it&#8217;s on p. 13 of the PDF file linked above), see if you can tell the differences among them, and which is the observed order:<\/p>\n<table align=\"center\" border=\"1\">\n<tr>\n<td>0.74<br \/>0.79<br \/>0.73<br \/>0.67<br \/>0.72<br \/>0.67<br \/>0.71<\/td>\n<td>0.79<br \/>0.67<br \/>0.72<br \/>0.67<br \/>0.71<br \/>0.74<br \/>0.73<\/td>\n<td>0.79<br \/>0.72<br \/>0.71<br \/>0.74<br \/>0.73<br \/>0.67<br \/>0.67<\/td>\n<\/tr>\n<\/table>\n<p>My professional judgment as an historian is also common sense: if the order of events does not make a discernible difference, even if you ignore measurement error and standard errors, then it&#8217;s hard to conclude that there&#8217;s a trend. How to test that is the realm of statistics, and when I explained the issue to my colleagues Jeffrey Kromrey and John Ferron, the answer from them was clear: permutation tests. That&#8217;s a general family of nonparametric tests of inference that&#8217;s the formal version of the question I asked: if you jumble up the data in all the possible ways they could be permuted, and if you look at a particular measure of interest (a test statistic), where in the distribution of all permutations does the observed data set fall? In the case of the 4th grade Black-White gap on New York state math tests measured as a z-score, we have 7 points of data, which have 7! = 5040 permutations. If you choose an appropriate test statistic for each permutation and the observed time series is about 125 from either end of the distribution, that excludes the 95% or more permutations in the middle of the distribution.<\/p>\n<p>No, I haven&#8217;t had the time or inclination to follow up, learn how to calculate one of the possible test statistics and how to get the R statistics program to do a permutation test. There are two problems, as I&#8217;ve learned from my colleagues: choosing the right test statistic is a matter of art as well as science; and there may be a problem with exchangeability. As far as I understand it, exchangeability is a less constricting assumption than the standard &#8220;independent, identically-drawn&#8221; sample assumption in parametric inferential statistics. From what I understand, the practical definition of exchangeability means roughly that you <em>could<\/em> theoretically exchange all the data points without screwing up the distribution. Again, if I understand correctly, one situation that violates the assumption of exchangeability is in autocorrelated data&#151;i.e., when one data point influences the next one (or the next few). And if there&#8217;s anything that&#8217;s likely to be autocorrelated, it&#8217;s a time series. That&#8217;s not a serious problem if you&#8217;re just looking to see if a trend exists at all; for that, autocorrelation is a form of trend (though an artifactual one). But if you&#8217;re trying to make causal inferences or anything more complicated when there&#8217;s autocorrelation (i.e., if achievement data levels or trend slopes are different before and after a policy change), I think you have to throw permutation tests out the window.<\/p>\n<p>And that&#8217;s such a shame, because the concept is still right when extended beyond the question of a trend: if a policy makes a difference, then it should make a difference on which side of the policy change you&#8217;re sitting. So if you&#8217;re a clever person with statistics, <em>please<\/em> provide some ideas in comments for where to go with this or if, as I suspect, the best we can do with permutation tests is ruling out possible trends\/autocorrelation. <\/p>\n","protected":false},"excerpt":{"rendered":"<p>First, to provide a minor update on this morning&#8217;s news items: Semi-success on the reserving-time front. I had a lunch meeting and then a 3 pm meeting, and the time in between was too short to do much, so I exchanged one parking sticker for another. Whee. At least my wonderful grad student assisting with [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"jetpack_post_was_ever_published":false},"categories":[16,11,18],"tags":[],"class_list":["post-1427","post","type-post","status-publish","format-standard","hentry","category-accountability-frankenstein","category-education-policy","category-the-academic-life"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/pag0MB-n1","_links":{"self":[{"href":"https:\/\/shermandorn.com\/index.php?rest_route=\/wp\/v2\/posts\/1427","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/shermandorn.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/shermandorn.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/shermandorn.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/shermandorn.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=1427"}],"version-history":[{"count":0,"href":"https:\/\/shermandorn.com\/index.php?rest_route=\/wp\/v2\/posts\/1427\/revisions"}],"wp:attachment":[{"href":"https:\/\/shermandorn.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=1427"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/shermandorn.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=1427"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/shermandorn.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=1427"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}