{"id":2363,"date":"2010-08-29T11:56:39","date_gmt":"2010-08-29T15:56:39","guid":{"rendered":"http:\/\/shermandorn.com\/wordpress\/?p=2363"},"modified":"2010-10-13T11:08:21","modified_gmt":"2010-10-13T15:08:21","slug":"l-a-times-crash-and-burn-report","status":"publish","type":"post","link":"https:\/\/shermandorn.com\/?p=2363","title":{"rendered":"L.A. Times crash-and-burn report"},"content":{"rendered":"

Now that the L.A. Times<\/em> has made its database on teachers and its VAM scores public, we can and should hold the Times<\/em> accountable for whether it has responded to concerns professionally. Here is what Jason Felch, Jason Strong, and Doug Smith wrote in part to Jay Mathews<\/a> in response to his concerns:<\/p>\n

\n

… teachers and schools will not be receiving numbered percentile rankings, which imply a level of precision that is not possible with these estimates. Rather, their relative position will be shown along a spectrum of effectiveness in a way that visually indicates the inherent imprecision of these estimates.<\/p>\n<\/blockquote>\n

So far, so good at least in terms of words. What does this mean? When researchers show distributions of scores, they often show error bands to indicate "the inherent imprecision," as Felch, Strong, and Smith wrote. For example, see the following figure from a 2000 paper by Kenneth Rowe<\/a> on value-added measures:<\/p>\n\"Graph<\/a>\n

The point here is that showing imprecision is easy to do in a way that is professionally competent.  Is that what the L.A. Times shows in its database? Here's the chart for one teacher:<\/p>\n\"Sample<\/a>\n

There are two graphing sins here: dequantification and an implication that the estimate for the teacher is infinitely accurate (or at least as accurate as the center of the diamond images). I don't know what the Times editors and reporters thought they were doing by eliminating a scale, but this doesn't remove the central problem of visually implying that the estimate of effectiveness is precise. Instead, it commits the sin of dequantification<\/a>. To borrow from Edward Tufte, is the L.A. Times' publication of these figures an act of reporting or finger-painting?<\/p>\n

It also raises significant questions about the response to Jay Matthews. Was the Times deliberately trying to fudge what they were intending to do with the graphs, or are they really so incompetent an organization that they don't have people who know how to design statistical figures and also didn't check such a high-stakes display with people who do this professionally?<\/p>\n

For those who wonder if there are written ethical guidelines for journalists, you'd want to look at the Society of Professional Journalists' Code of Ethics<\/a>, specifically the section on minimizing harm, including the following:<\/p>\n

\n

Journalists should:<\/p>\n

— Show compassion for those who may be affected adversely by news coverage. Use special sensitivity when dealing with children and inexperienced sources or subjects….
\n\t\t— Recognize that gathering and reporting information may cause harm or discomfort. Pursuit of the news is not a license for arrogance.<\/p>\n<\/blockquote>\n

In the charts it published today, the Times blatantly misrepresents the accuracy of its statistical model results for people who did not choose to have their names in public. Those who read Brad DeLong's blog<\/a> will recognize the style of the title. I am not nearly as dyspeptic as he is regarding newspapers, but this is sheer arrogance about the consequences of publication without regard for competent representation of information.  <\/p>\n

Update: see Bill Boyarsky's comments<\/a> from September 2 for another journalist's take.<\/p>\n","protected":false},"excerpt":{"rendered":"

Now that the L.A. Times has made its database on teachers and its VAM scores public, we can and should hold the Times accountable for whether it has responded to concerns professionally. Here is what Jason Felch, Jason Strong, and Doug Smith wrote in part to Jay Mathews in response to his concerns: … teachers […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"jetpack_post_was_ever_published":false},"categories":[11],"tags":[],"class_list":["post-2363","post","type-post","status-publish","format-standard","hentry","category-education-policy"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/pag0MB-C7","_links":{"self":[{"href":"https:\/\/shermandorn.com\/index.php?rest_route=\/wp\/v2\/posts\/2363","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/shermandorn.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/shermandorn.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/shermandorn.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/shermandorn.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=2363"}],"version-history":[{"count":12,"href":"https:\/\/shermandorn.com\/index.php?rest_route=\/wp\/v2\/posts\/2363\/revisions"}],"predecessor-version":[{"id":2547,"href":"https:\/\/shermandorn.com\/index.php?rest_route=\/wp\/v2\/posts\/2363\/revisions\/2547"}],"wp:attachment":[{"href":"https:\/\/shermandorn.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=2363"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/shermandorn.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=2363"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/shermandorn.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=2363"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}