My friend Tom Smith wrote the official Talk like a Pirate Day anthem:
The closest attitudes I’ve seen in education research ARRRRRRRRRR from econometric specialists, such as those trained by MIT’s Joshua Angrist. If you have Angrist and Jörn-Steffen Pischke’s Mostly Harmless Econometrics, like me, matey, you may have read a certain swashbuckling tone between the lines: Let’s grab that data, make bold assumptions that are pretty reasonable, and look at
our booty the conclusions we can draw!
As at least one colleague has pointed out to me, thar be a certain boldness in ’tis style ‘o reasonin’, ‘n them ‘o us trained in skepticism can find it bleeding uncomfortable.1 In An Enquiry Concerning Human Understanding (1777/1902), David Hume gave the skeptic’s view of empirical induction: It works, but thar be no reason why it works. The paragraph below is pretty typical of Hume’s style:
In reality, all arguments from experience are founded on the similarity which we discover among natural objects, and by which we are induced to expect effects similar to those which we have found to follow from such objects. And though none but a fool or madman will ever pretend to dispute the authority of experience, or to reject that great guide of human life, it may surely be allowed a philosopher to have so much curiosity at least as to examine the principle of human nature, which gives this mighty authority to experience, and makes us draw advantage from that similarity which nature has placed among different objects. From causes which appear similar we expect similar effects. This is the sum of all our experimental conclusions. Now it seems evident that, if this conclusion were formed by reason, it would be as perfect at first, and upon one instance, as after ever so long a course of experience. But the case is far otherwise. Nothing so like as eggs; yet no one, on account of this appearing similarity, expects the same taste and relish in all of them. It is only after a long course of uniform experiments in any kind, that we attain a firm reliance and security with regard to a particular event. Now where is that process of reasoning which, from one instance, draws a conclusion, so different from that which it infers from a hundred instances that are nowise different from that single one? This question I propose as much for the sake of information, as with an intention of raising difficulties. I cannot find, I cannot imagine any such reasoning. But I keep my mind still open to instruction, if any one will vouchsafe to bestow it on me.
Those of us trained in disciplines that encouraged caution and suspension of judgment can end up feeling
stabbed by the cutlass taken aback by the swashbuckling style and confident (ay, what some call the bloody) tone of econometricians, political scientists, and some others “of that ilk” (to steal from Tom Lehrer).2
How much of that is style?3 In a now-classic article on historical reasoning, Sam Wineburg summarizes how the professional historian’s “talk-aloud” through a primary source contains both shrewd awareness of subtext and a great deal of uncertainty:
Texts come not to convey information, to tell stories, or even to set the record straight. Instead, they are slippery, cagey, and protean, and reflect the uncertainty and disingenuity of the real world. (p. 500)
A number of fields value this suspension of judgment from history to cultural anthropology, in practices such as ethnography and more general qualitative research and even the medical practice of diagnosis. So there is a substantial difference in the rhetorical aesthetics between econometrics and my training as an historian. But I think this goes beyond rhetoric, and we can learn a great deal about disciplinary assumptions from looking at disciplinary attitudes in a subtly different way.
Here’s the gist of the argument: all fields have at least one Necessary Fallacy. A Necessary Fallacy is a slippery assumption that the field requires to do its work. Sorry, forgot to translate: A Necessary Fallacy be a slippery dog that th’ field requires to set the sails. Much like the empirical inferences that Hume attacks, these are unjustifiable if you dig far enough down, and they are engines of both the social process and the substantive warrants in a field. An example of a whole subfield devoted to Necessary Fallacy Studies is critical legal studies: judges have to make decisions, and the bases on which they do so is not justifiable from any abstract, absolute sense.
Do not assume that ye field has no Necessary Fallacy: o’ course it does.
As far as I can tell, econometrics has two Necessary Fallacies:
- What is measurable is often meaningful. This is where I have seen people in other fields tilt most vigorously at econometric studies, and it often is where economists without much background knowledge in an area can make massive category errors. But working economists know this criticism, focus in a limited set of areas if smart, and generally address things on the front end.4
- What is allocable is attributable. Because econometrics focuses on careful methods of allocating measurements, they are generally criticized on this point less frequently than on N.F.#1.5 This is the warrant slippage that is the deeper Necessary Fallacy: there is no way to have an econometric analysis without this assumption,. Yet it is just an assumption: what we can quantitatively allocate to a hypothetical cause is the attributable effect. As with everything David Hume discussed, there is no provable link. It just is.
What we learn from a field’s Necessary Fallacy (or Fallacies) is the type of academic values that can only be spread by personal connections, commonly in graduate school, often accompanied by good (or bad) grog. This is not abstract, nor is it just culture. ’tis th’ often-weird intersection between how human bein’s construct academic fields ‘o study ‘n th’ reasonin’ that be presented in publications–or be absent. It is inevitable. Like International Talk Like a Pirate Day. Avast, ye readers!
- Translations courtesy of Post Like a Pirate. [↩]
- I have a masters in demography, so my graduate degrees allow me to pick and choose what offends me. [↩]
- I think lawyers try to exercise that “voice of God” trick more than social scientists, but that may be my idiosyncratic experience. [↩]
- The most (in)famous such category error that I learned in grad school was the analysis of recorded beatings in a plantation owner’s diary in Fogel and Engerman’s Time on the Cross (1974). As Herbert Gutmann pointed out, the relevant social measure for the lives of those held as slaves on a plantation was not just how often you felt a whip but how vulnerable you felt from the general frequency of violence. The latter was of course more common and the relevant issue for a community of people held as slaves. [↩]
- Andrew Gelman is an exception, and that’s on technical grounds. [↩]