Teachers College sociologist Aaron Pallas takes a New York judge to task for miscategorizing what is Fact Or Opinion in the ruling to let the internal NYC "teacher ratings" become public. (The issue is on appeal, and the judge's ruling is stayed while the appeal is pending.) Essentially, Pallas's point is that with such uncertainty about the accuracy of the value-added scores (including concerns about misattribution of students to teachers), the numbers assigned to teachers cannot be described as facts.
In an abstract and pure-research way, he is correct. But in another (very cynical) way, it is precisely the job of technocratic apparatchiks to create facts that can be used in political judgments, accuracy be damned. My colleague at USF Stephen Turner makes this point in Liberal Democracy 3.0, that one function of courts and commissions is to allow fact-finding (or the creation of social facts) to be displaced away from the political arena, so that political discourse can focus on what to do about the social facts rather than the messy thing of what is reality in the first place. I leaned heavily on Turner's argument for one chapter in Accountability Frankenstein, and the hard-edged interpretation of test scores, school "grades," and AYP designations is this creation of facts or at least the veneer of factuality. No wonder the judge bought into this framework, because that is a recognized role of juries, initial hearings, expert witness testimony, and so forth.
So how could one burst through this veneer? I can imagine the following line of questions at hearing, at least up to a certain point:
UFT Attorney. Isn't there a great deal of uncertainty in the numbers you describe as the value added by teachers?
BOE Official. Well, yes, but all measures have uncertainty, and that doesn't mean that the measures are meaningless in and of themselves, and that's why we have an obligation to release them to the press. They're factual information.
[In a nutshell, that's the argument of the NYC BOE.]
Q. Let's imagine a slightly different scenario. Suppose you had a school official who just made up numbers. Totally false numbers that implied 500 teachers were horrible when they were pretty good. Would that be wrong?
A. Yes. But that's not the situation here. No one–no principal or other administrator in New York would ever do that.
Q. Let me follow this line of questions a little, okay? This school official makes up a list of 500 teachers with completely false information. He does it on a Board of Education computer. So that's a school district file, right?
A. Uh, …
Q. The district owns the computer and all the data on it created for official purposes, right?
A. Uh, yes.
Q. So the school district now owns that false information. Would you agree that if such a file existed, it would be defamatory to those teachers?
A. I'm not sure about the legal term…
Q. Would it be false information that could affect their professional reputations?
A. Yes, absolutely.
Q. So if the press asks for that file, what would the Board of Education do?
Q. Could the Board withhold the file on the basis that the information is false?
Q. I bet you'd find a way to withhold it with that explanation, that you're not obligated to release completely false information.
Q. I bet if you did release the list that was defamatory, you'd be slammed by the press for knowingly participating in defamation of public employees.
Q. I bet you'd withhold the list.
Q. Let's suppose that the list isn't completely false. Let's suppose in that list of 500 teachers, there is a single teacher with information that correctly identifies him as a really good teacher. A list that falsely portrays 499 teachers as awful and accurately portrays one teacher as good.
A. I don't quite see the point of–
Q. Let's say this one teacher portrayed as a good teacher has a name. Mr. Lot.
Judge: Oh, good grief…
Q. Your honor, am I allowed a small joke?
Judge: Go on…
Q. So we have 499 bad numbers and 1 good number. Does the one good number save the entire list?
A. I don't understand.
Q. You just said a few seconds ago that if the list is completely false and defamatory, the Board of Education doesn't have to release it to the press.
Q. Now we have a situation where the list only defames 499 teachers, but it's very nice to Mr. Lot. Does Mr. Lot make it possible to release the list?
A. I don't see how that helps. As a whole, the list is inaccurate.
Q. I agree. Now let's change the situation. Suppose you know only that the list is defamatory to 499 teachers but is accurate about 1. But you don't know which teacher has the accurate number. It's accurate by random Lot but false for all but one teacher.
BOE Attorney: Your honor, I object.
UFT Attorney: How is a bad pun a violation of procedure?
Judge: No more puns, okay?
UFT Attorney: Okay. My question remains. Would the fact that the list is accurate for some teacher you don't know about make the list any better?
A. No. It's still awful on the whole.
Q. Good. We're agreed up to that point. Now let's switch the numbers and the situation a bit. Again, it's a 500-teacher list, but instead of being false and defamatory about all or almost all, the list is accurate for 400 teachers and completely defamatory about 100. And the school district knows the identity of the 100 teachers with false information attributed to them.
A. I don't see the–
Q. Please follow me for just a little longer. If the Daily News or Post asked for this list, what would you do? We've agreed that the Board can withhold the list when it's completely defamatory or almost completely defamatory, but what about when 80% of it is correct and you know what part of the list is correct?
A. Uh… I think we'd have to give out the part of the list with the 400 teachers with accurate information and work on the other 100.
Q. And you can do that because, in this hypothetical situation, you know which teachers have accurate information, right?
A. Yes. We can identify what's correct and what's not and withhold the false information.
Q. I agree. What if you don't know which teachers have the accurate scores, though? Suppose you know that of the 500 teachers on the list, 100 will have false, damaging information, but you don't know which teachers have false information attributed to them?
Q. That's the situation with value-added measures, isn't it?
Q. You have a list of teachers, but the list is inaccurate and professionally damning for a certain proportion?
A. No, no…
Q. But you've acknowledged earlier the uncertainty in the data.
A. Yes, but that's not…
Q. It means that some teachers are going to be falsely judged as bad teachers, right?
A. Uh… yes. Okay.
Q. So let's go back to that 500-teacher list. If you release the entire list, you know, in advance, that 100 teachers are going to be publicly labeled, and falsely labeled, as bad teachers. If you knew their identity, you've already told me the Board of Education can withhold that part of the list. Why does the legal and ethical obligation of the district change when you don't know which teachers have false labels?
And here, I'd love to see how someone who thinks disclosure of value-added scores is good policy would answer the question at the end.