Now that the L.A. Times has made its database on teachers and its VAM scores public, we can and should hold the Times accountable for whether it has responded to concerns professionally. Here is what Jason Felch, Jason Strong, and Doug Smith wrote in part to Jay Mathews in response to his concerns:
… teachers and schools will not be receiving numbered percentile rankings, which imply a level of precision that is not possible with these estimates. Rather, their relative position will be shown along a spectrum of effectiveness in a way that visually indicates the inherent imprecision of these estimates.
So far, so good at least in terms of words. What does this mean? When researchers show distributions of scores, they often show error bands to indicate "the inherent imprecision," as Felch, Strong, and Smith wrote. For example, see the following figure from a 2000 paper by Kenneth Rowe on value-added measures:
The point here is that showing imprecision is easy to do in a way that is professionally competent. Is that what the L.A. Times shows in its database? Here's the chart for one teacher:
There are two graphing sins here: dequantification and an implication that the estimate for the teacher is infinitely accurate (or at least as accurate as the center of the diamond images). I don't know what the Times editors and reporters thought they were doing by eliminating a scale, but this doesn't remove the central problem of visually implying that the estimate of effectiveness is precise. Instead, it commits the sin of dequantification. To borrow from Edward Tufte, is the L.A. Times' publication of these figures an act of reporting or finger-painting?
It also raises significant questions about the response to Jay Matthews. Was the Times deliberately trying to fudge what they were intending to do with the graphs, or are they really so incompetent an organization that they don't have people who know how to design statistical figures and also didn't check such a high-stakes display with people who do this professionally?
For those who wonder if there are written ethical guidelines for journalists, you'd want to look at the Society of Professional Journalists' Code of Ethics, specifically the section on minimizing harm, including the following:
Journalists should:
— Show compassion for those who may be affected adversely by news coverage. Use special sensitivity when dealing with children and inexperienced sources or subjects….
— Recognize that gathering and reporting information may cause harm or discomfort. Pursuit of the news is not a license for arrogance.
In the charts it published today, the Times blatantly misrepresents the accuracy of its statistical model results for people who did not choose to have their names in public. Those who read Brad DeLong's blog will recognize the style of the title. I am not nearly as dyspeptic as he is regarding newspapers, but this is sheer arrogance about the consequences of publication without regard for competent representation of information.
Update: see Bill Boyarsky's comments from September 2 for another journalist's take.
May I suggest the subtle approach? Whack him over the head with a Tufte book.
Just think of is as part of the teacher professionalization project.
That should make it easier to swallow.
Teacher professionalization is ongoing, right? (Some would say that it hasn’t even begun.)
In this case, you have an *attempt* at constructing a measure of teacher effectiveness — seen as just another brick into the road toward professionalization, a road which, eventually will lead to the promised land. Regardless of how flawed it may be, it IS a step along the path toward teacher professionalization.
Now, suppose that, miracle of miracles, someone comes up with the perfect measure of teacher effectiveness. What would the response be? It would be universally applied, and — shazam! — the teaching profession would be professionalized with no more excuses by unions, admins, parents. End of all the controversy. Why, because now you have the perfect justification for hiring, firing, and promotions.