{"id":8037,"date":"2015-08-18T07:31:10","date_gmt":"2015-08-18T14:31:10","guid":{"rendered":"http:\/\/shermandorn.com\/wordpress\/?p=8037"},"modified":"2015-08-18T07:31:10","modified_gmt":"2015-08-18T14:31:10","slug":"another-dirty-little-secret-of-test-based-measures","status":"publish","type":"post","link":"https:\/\/shermandorn.com\/?p=8037","title":{"rendered":"Another dirty little secret of test-based measures"},"content":{"rendered":"<p>New York State&#8217;s department of education recently reported that <a href=\"http:\/\/www.nytimes.com\/2015\/08\/13\/nyregion\/new-york-state-students-standardized-tests.html\">approximately 20% of students in testing grades refused to participate<\/a> in this year&#8217;s state assessments, the high-water mark thus far of the opt-out movement. Among the various stories and arguments flowing from that is the argument that 20% refusal is easily over the threshold of non-participation that invalidates conclusions drawn from testing. This is an argument made by both proponents and opponents of test refusals.<\/p>\n<p>Here is the dirty little secret of the existing system of accountability: plenty of measures\u00a0already are working on less than 80% coverage for many\u00a0schools. Three come to mind this morning, with only one cup of coffee in me:<\/p>\n<ul>\n<li>The federal graduation rate definition excludes students who move away from a school after ninth grade &#8212; thus the <em>adjusted-cohort<\/em> part of <em>adjusted-cohort graduation rate<\/em>. In areas with high student mobility, well over 20% of the original cohort will be uncovered in the graduation rate. (Are those mobile students truly counted\u00a0<em>somewhere<\/em>? No one knows.)<\/li>\n<li>Value-added algorithms\u00a0require multiple years of test-score data. Again, with high student mobility, plenty of schools and a higher proportion of\u00a0teachers have value-added indicators\u00a0that come from far fewer than 80% of the students taught in a year.<\/li>\n<li>In many states, even cross-sectional test data is limited to students who attended a particular school for at least a good part of the school year. High-mobility schools have far more than 20% turnover in a year.<\/li>\n<\/ul>\n<p>If 20% nonparticipation is a measure-killer, we need to worry about far more than New York state&#8217;s accountability indicators.<\/p>\n<p>What can happen with missing information in a statistic? At least two\u00a0things:<\/p>\n<ul>\n<li>Bias from the nonrandom nature of missing values. If the participants are different in fundamental ways from nonparticipants, any measure will not reflect the complete population.<\/li>\n<li>An incorrect assumption that a statistical\u00a0estimate is more accurate than it truly is. This is an important finding of the research of Donald Rubin in the 1980s: correcting for missing values\u00a0generally leads to larger standard errors for statistics.<\/li>\n<\/ul>\n<p>There are other problems, but these are the main ones to keep in mind.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>New York State&#8217;s department of education recently reported that approximately 20% of students in testing grades refused to participate in this year&#8217;s state assessments, the high-water mark thus far of the opt-out movement. Among the various stories and arguments flowing from that is the argument that 20% refusal is easily over the threshold of non-participation [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_hide_featured_image":false,"_jetpack_featured_image_duplicate_dismissed":0,"_jetpack_feature_clip_id":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"jetpack_post_was_ever_published":false},"categories":[16,11],"tags":[],"class_list":["post-8037","post","type-post","status-publish","format-standard","hentry","category-accountability-frankenstein","category-education-policy"],"jetpack_publicize_connections":[],"jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/pag0MB-25D","jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/shermandorn.com\/index.php?rest_route=\/wp\/v2\/posts\/8037","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/shermandorn.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/shermandorn.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/shermandorn.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/shermandorn.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=8037"}],"version-history":[{"count":1,"href":"https:\/\/shermandorn.com\/index.php?rest_route=\/wp\/v2\/posts\/8037\/revisions"}],"predecessor-version":[{"id":8038,"href":"https:\/\/shermandorn.com\/index.php?rest_route=\/wp\/v2\/posts\/8037\/revisions\/8038"}],"wp:attachment":[{"href":"https:\/\/shermandorn.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=8037"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/shermandorn.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=8037"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/shermandorn.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=8037"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}