{"id":8012,"date":"2015-08-07T15:39:53","date_gmt":"2015-08-07T22:39:53","guid":{"rendered":"http:\/\/shermandorn.com\/wordpress\/?p=8012"},"modified":"2015-08-07T15:39:53","modified_gmt":"2015-08-07T22:39:53","slug":"validitys-likely-irrelevance-in-personnel-processes","status":"publish","type":"post","link":"https:\/\/shermandorn.com\/?p=8012","title":{"rendered":"Validity’s likely irrelevance in personnel processes"},"content":{"rendered":"

My colleague Audrey Amrein-Beardsley is starting a series of blog entries glossing the March special issue of\u00a0Educational Researcher<\/em>\u00a0devoted to value-added or growth measures, specifically the technical qualities vis-a-vis teacher evaluations. In her first entry<\/a>\u00a0on the issue today, she argues\u00a0the following:<\/p>\n

[Quoting Doug Harris and Carolyn Herrington:]\u00a0\u201cThe issue is not whether value-added measures are valid but whether they can be used in a way that improves teaching and learning.\u201d [Amrein-Beardsley:] I would strongly argue that validity is a pre-condition to use, as we do not want educators using invalid data to even attempt to improve teaching and learning. I\u2019m actually surprised this statement was published, as so scientifically and pragmatically off-based.<\/p><\/blockquote>\n

In psychological and educational research, the term\u00a0validity<\/em> refers to a basket of substantive warrants for the relevance of research: is a construct or measure predictive of something important, is it highly associated with another important and independently verified construct\/measure, does it have merits for particular uses, etc. While researchers sometimes fetishize reliability and validity as terms of art, it is important to understand that the terms simply refer to procedural and substantive warrants for the concept or tool in question.<\/p>\n

Here is the broader context of the quotation (with the quoted sentence emphasized and Harris and Herrington’s references deleted):<\/p>\n

Since [Race to the Top’s]\u00a0inception in 2009, our academic journals have been filled with articles about the validity and reliability of value-added measures. Books and reports have been written that describe the pros and cons of value added. Others have expressed skepticism, whereas prominent foundations and some think tank reports have been more positive.\u00a0The various sides look at the same evidence and see a different picture partly because the evidence we have is so far removed from the decisions that have to be made. The issue is not whether value-added measures are valid but whether they can be used in a way that improves teaching and learning<\/em>. How do educators actually respond to policies that use value-added measures? On this question, we know very little. Therefore, in the debate about these policies, perspective has taken over where the evidence trail ends.<\/p><\/blockquote>\n

In essence, Harris and Herrington are saying that value-added measures do not have to have other<\/em>\u00a0clearly-accepted substantive warrants for use if they are useful for helping teachers… which is usually something that falls under the term\u00a0consequential validity<\/em>. Amrein-Beardsley is saying that an assertion of consequential validity\u00a0cannot stand by itself without evidence of some other form of validity. She does not flesh out that argument, but let’s try substituting something ridiculous or irrelevant for the term value-added measures<\/em> to see how the reasoning\u00a0plays out in other areas where “we know little” is a reasonable conclusion:<\/p>\n

The issue is not whether a classroom’s average cholesterol score\u00a0is\u00a0valid but whether it\u00a0can be used in a way that improves teaching and learning. How do educators actually respond to policies that use a classroom’s average cholesterol score? On this question, we know very little.<\/p>\n

The issue is not whether a Rorschach\u00a0test is\u00a0valid for teachers but whether it\u00a0can be used in a way that improves teaching and learning. How do educators actually respond to policies that use Rorschach\u00a0tests? On this question, we know very little.<\/p><\/blockquote>\n

My colleague wins on the logic front:\u00a0it is reasonable to ask if there is any\u00a0there<\/em> there before using\u00a0a weak evaluative mechanism as a projective test for professional changes.<\/p>\n

On the other hand, there is neither a legal nor practical reason to set the bar high for\u00a0some<\/em> use of weak evaluative mechanisms, as long as they do not determine outcomes\u00a0by themselves. I have absolutely no reason to think\u00a0that value-added residuals tied to teachers will give better practical guidance for professional change\u00a0than reasonably-constructed classroom observations. Classroom observation tools are also flawed, sometimes very flawed. As I have written before, the fundamental issue is that schools and school systems cannot\u00a0wait for the perfect evaluation system, and given my cynical nature I think most of them are so far below the Lee Shulman idea of a “marriage of insufficiencies” that most of them are best characterized as marriages of incompetencies.<\/p>\n

Without getting into the details of designs, any good system of evaluation should have the following political and human\u00a0<\/em>attributes:<\/p>\n