{"id":4635,"date":"2012-03-04T11:22:43","date_gmt":"2012-03-04T15:22:43","guid":{"rendered":"http:\/\/shermandorn.com\/wordpress\/?p=4635"},"modified":"2012-03-04T19:14:33","modified_gmt":"2012-03-04T23:14:33","slug":"grades-for-coverage-of-nyc-teacher-data-report-release","status":"publish","type":"post","link":"https:\/\/shermandorn.com\/?p=4635","title":{"rendered":"Grades for coverage of NYC Teacher Data Report release"},"content":{"rendered":"<p>Before the New York City Department of Education released the name-and-number spreadsheet on the now-defunct Teacher Data Reports, I wrote and released\u00a0<a href=\"https:\/\/shermandorn.com\/?p=4598\">grading criteria for news coverage<\/a>. And now, with more than a week of coverage from a number of outlets, the grades (limited to major outlets where I read a critical mass of coverage). ((Thanks to Leo Casey of UFT for forwarding voluminous news clips, which provided a reality check on my separate online reading. Any errors in judgment are my own.))<\/p>\n<h1>Grades<\/h1>\n<h2>Gotham Schools: A<\/h2>\n<p><em>Gotham Schools<\/em> climbed into the A-C range by <a href=\"http:\/\/gothamschools.org\/2012\/02\/23\/why-we-wont-publish-individual-teachers-value-added-scores\/\">declining to publish<\/a> specific teacher names. Here are some of the stories GS published on the TDR dump to earn the A:<\/p>\n<ul>\n<li>Perspectives from <a href=\"http:\/\/gothamschools.org\/2012\/02\/24\/as-ratings-near-a-teacher-reiterates-what-test-scores-dont-say\/\">a teacher<\/a>, <a href=\"http:\/\/gothamschools.org\/2012\/02\/27\/many-parents-in-the-dark-on-teacher-ratings-on-first-day-back\/\">parents<\/a>, Dennis Walcott both <a href=\"http:\/\/gothamschools.org\/2012\/02\/24\/city-releases-teacher-data-reports-and-a-slew-of-caveats\/\">with caveats<\/a> and then\u00a0<a href=\"http:\/\/gothamschools.org\/2012\/03\/02\/following-bloomberg-walcott-shifts-on-teacher-ratings-release\/\">backing away from caveats<\/a>, and UFT head Michael Mulgrew (<a href=\"http:\/\/gothamschools.org\/2012\/02\/27\/at-ps-321-mulgrew-finds-universal-opposition-to-ratings-release\/\">in situ<\/a>).<\/li>\n<li>A corrective follow-up to the <a href=\"http:\/\/www.nytimes.com\/2012\/02\/25\/education\/teacher-quality-widely-diffused-nyc-ratings-indicate.html?_r=1\">New York Times reporters&#8217; superficial analysis<\/a> of score distributions, explaining <a href=\"http:\/\/gothamschools.org\/2012\/02\/29\/why-its-no-surprise-high-and-low-rated-teachers-are-all-around\/\">why the statistics were structured<\/a> to leave the impression ratings were evenly distributed across schools.<\/li>\n<li>A piece putting the TDR and the release <a href=\"http:\/\/gothamschools.org\/2012\/03\/01\/citys-value-added-initiative-early-entrant-to-evolving-landscape\/\">in a national context<\/a>.<\/li>\n<\/ul>\n<div>At the same time, GS kept up coverage on other school issues in the last week, despite having fewer resources than the metro dailies. That is not to say that GS coverage was perfect, but I do not require perfection in A work. Some reporters (including but not restricted to GS staff) could have explored how the TDR agreement went off the rails, from a closely-held pilot to a <a href=\"http:\/\/www.cjr.org\/cover_story\/tested.php\">Klein-provoked court case<\/a>.<br \/>&nbsp;<\/div>\n<h2>New York Times: D<\/h2>\n<p>The <em>New York Times<\/em>\u00a0earned a D because it published teacher names and scores. It avoided an F by including some indication of margin of error&#8211;not very smartly, because the point estimate had two nonzero digits, which is inconsistent with the brontosaur-wide error bands. But if I am not going to require perfection for an A, why should I quibble too much with work that is no higher than a D? NYT reporters added plenty of context, sometimes smartly and sometimes foolishly (see the superficial-analysis article linked above, including the GS response). My bottom-line judgment: so much effort, not enough value added. ((Pun fully intended.))<\/p>\n<h2>New York Daily News: D<\/h2>\n<h2>New York 1: D<\/h2>\n<h2>Wall Street Journal: D<\/h2>\n<p>Publishers, producers, and editors of the <em>Daily News,<\/em>\u00a0NY1, and the <em>Wall Street Journal<\/em>\u00a0comprised the rest of the herd in publishing teacher names and scores but avoiding the worst of tabloid coverage and displays. The WSJ has taken a different tack from others in displaying the labels (high, above-average, etc.) rather than quantities. The central problem of such reports is that when the data are so untrustworthy, there are no good options for displaying them with integrity. Quantify and you falsely imply precision (e.g., the too-many-significant-digits issue). Use categories and you imply the labels are applied with precision.<\/p>\n<h2>New York Post: F<\/h2>\n<p>What can you say about the New York Murdoch tabloid? For several days the NYP only displayed point estimates with no margins of error, and in several cases the paper wrote an entire story about a single low-rated teacher, which qualifies as sensationalizing. Either of those would have earned the paper a failing grade.<\/p>\n<h1>Observations on the TDR race to the bottom<\/h1>\n<p>I am struggling with explaining why the publishers and producers of major news outlets in New York took the bait Joel Klein offered more than a year ago, despite clear evidence that a broad range of people in journalism and education thought publishing individual teacher ratings was a lousy idea. The major dailies and NY1 continued down that path through the long court case and after Klein&#8217;s departure, even after editors knew or should have known that TDR was a pilot, fragile, and based on tests revealed to be problematic since the original NYCDOE-UFT agreement creating the pilot. The easy answer is that the TDR coverage was the equivalent of nightly-news &#8220;if it bleeds, it leads&#8221; practice. But I think it&#8217;s more complicated, since several of the outlets had plenty of time to have extensive internal discussions before the release, and there were plenty of options between &#8220;publish everything with names&#8221; and &#8220;publish nothing.&#8221;<\/p>\n<p>It is important to keep pushing on this story, because unlike the <em>Los Angeles Times<\/em>\u00a0VAM publishing project, we have several news organizations and a longish period of time in which decisions could have been made, reversed, and so forth.\u00a0The <a href=\"http:\/\/www.cjr.org\/cover_story\/tested.php\">Columbia Journalism Review article<\/a> from March\/April 2011 should not be the last word on this, because this is likely to be a continuing ethical dilemma with large data sets and journalism. I have seen at least one comparison between the NYT coverage of TDR and the Judy Miller Iraq-WMD scandal, but it feels to me to be\u00a0a different challenge to the\u00a0<em>Times<\/em>&#8216; integrity, even if it is the same publisher (&#8220;Pinch&#8221; Sulzberger) who made the misjudgments.\u00a0On the other hand, in both cases, some form of heady rush (pre-war coverage or data on public employees) trumped caution on ethics and professionalism. Sadly, it looks like bad data are publishers and editors&#8217; crack cocaine.<\/p>\n<p>I have seen a few efforts nationally to spin the coverage in NYC in different ways, from &#8220;well, it got a conversation started&#8221; to &#8220;<a href=\"http:\/\/www.nytimes.com\/2012\/02\/23\/opinion\/for-teachers-shame-is-no-solution.html?ref=sunday\">Gates&#8217; op-ed<\/a> in the NYT was an Overton Window thing&#8221; (paraphrases, not direct quotations). Folks: please stop this effort to spin. Just. Stop. When people with widely-varying views of value-added measures all think publishers and editors at the <em>Times<\/em>\u00a0et al. jumped the shark with their coverage, there is no obvious effort at spin that doesn&#8217;t come across as tenuous reasoning. The coverage mostly stank, it bodes poorly for how news organization managers respond to data dumps in general, and that&#8217;s about all to be said for it for now. ((&#8230; until well-researched post-mortems.))<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Before the New York City Department of Education released the name-and-number spreadsheet on the now-defunct Teacher Data Reports, I wrote and released\u00a0grading criteria for news coverage. And now, with more than a week of coverage from a number of outlets, the grades (limited to major outlets where I read a critical mass of coverage). ((Thanks [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"jetpack_post_was_ever_published":false},"categories":[16,11],"tags":[],"class_list":["post-4635","post","type-post","status-publish","format-standard","hentry","category-accountability-frankenstein","category-education-policy"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/pag0MB-1cL","_links":{"self":[{"href":"https:\/\/shermandorn.com\/index.php?rest_route=\/wp\/v2\/posts\/4635","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/shermandorn.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/shermandorn.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/shermandorn.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/shermandorn.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=4635"}],"version-history":[{"count":29,"href":"https:\/\/shermandorn.com\/index.php?rest_route=\/wp\/v2\/posts\/4635\/revisions"}],"predecessor-version":[{"id":4671,"href":"https:\/\/shermandorn.com\/index.php?rest_route=\/wp\/v2\/posts\/4635\/revisions\/4671"}],"wp:attachment":[{"href":"https:\/\/shermandorn.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=4635"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/shermandorn.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=4635"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/shermandorn.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=4635"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}