What do we want?
Evidence-based change!
When do we want it?
After peer review!
– sign at 2010’s Rally to Restore Sanity and/or Fear
Ten years after1 his most famous article, Why Most Published Research Findings are False, John Ioannidis has been interviewed by Vox. The interview does not explore Ioannidis’s underlying reasoning, which presumes that in any individual field, there is a general rate of discoverable (and true) relationships, and then infers a general method of estimating false published research findings. The reasoning in the 2005 article is accessible, and even if you do not agree with the particular models that Ioannidis used, Ioannidis crafts a reasonably persuasive case that having been published — published after peer review — is far from a sufficient claim to truth for a paper.
I have been spending part of my last month’s commuting thinking about research synthesis, both quantitative meta-analysis that has developed over the past four decades and attempts to develop methodical ways to synthesize qualitative research, such as Noblit and Hare’s meta-ethnography. My brief impression on the qualitative synthesis writings: as happens occasionally, there is too much effort to wave huge rhetorical signs that say, “See, we are just like the quantoids, but in our own way!” and too little effort to think carefully about what research synthesis is for, and what it requires.
The interview with Ioannidis came at just the right time today, reminding me of his argument and how there is an underlying principle of research synthesis we usually forget. I am in the “need to [TMI]” stage of a cold, so this will be a short listing of my current thoughts on research synthesis (generic). At some point I need to return to this in more depth, so this entry is a bit of a placeholder and likely to be cryptic.
- Research synthesis allows one to explore a (manufactured) consensus on a research question, taking advantage of multiple studies that address said question (or a set of related questions).
- The set of studies that address a question or collection of questions is commensurable by definition. Commensurable does not mean that the conclusions are identical; it means that the studies all address the question or collection of questions in a comparable way.2
- A common assumption of research synthesis is that all commensurable studies can contribute to generalized knowledge.
- In contrast,3 unless one is in the unusual position of looking at a set of studies that entirely duplicate each others’ conclusions, one must be willing to throw out some part of the conclusions of most or all of the studies. (This is where seeing the Ioannidis interview and reviewing the original article helped me.)
- A key question is how one discards information from individual studies. If you look for something like a “central” conclusion, you toss information that was used in one study but appears to be an exception in the context of multiple studies.4 This evening, my attention is too distracted to describe other ways to do so, but the central issue is the need to discard information in a principled fashion, based on what you are attempting to do in the synthesis.
- While a common goal of synthesis is to provide practical, applicable guidance from a body of research — is X something we should do based on the research? — there are other ways to use synthesis findings. Two examples: find ways to test the current understanding of the question (find good areas for future research), and model the synthesis to test it against other information (formally model the research to test it against reality).
- If the existing research in an area includes studies that are not commensurable with a specific research question or questions, you can sometimes include those studies by changing the question… but often, you do so by learning more about the human behavior we call scholarship than about the subject at hand.
Notes