Single studies are good; literature is amazing

We can learn quite a bit from the surge of amateur epidemiology: It’s hard to be a good reader of a single study, and you don’t have to do that to learn from research. For almost half a year, I’ve repeatedly seen many well-educated, well-read people try to learn The Secret of Covid from individual studies in fields they have no training in.

This is understandable, but not generally a good use of someone’s time. Nor is it necessary.

My relevant expertise is epidemiology-adjacent: a masters of arts in demography, taught by some of the leading researchers in mortality and fertility. Graduate degree in counting death, in part. This is not the same as epidemiology, and I earned the degree 28 years ago. In terms of understanding papers coming out today, I have a fighting chance of understanding the gist of an article on epidemiological modeling but no experience in original research in the area.

This matters. It means I can follow an argument, but not evaluate it anywhere near an expert in the field.

So when multiple relatives/friends started citing a certain PNAS paper to say it “proved” the value of face masks, I said something like, “Let’s pause and wait a few days for epidemiologists to weigh in.” That process took less than a day — most prominently, in this Twitter thread from Kate Grabowski:

I’m claiming no epidemiological spidey-sense here. Instead, it’s considerable experience in not putting too much weight on a single study.

At my college, we teach masters students how to read individual studies in empirical social-science traditions. It’s a good course! And it’s a more formalized version of what I had to learn when I moved into colleges of education – I mean, what’s the fun in trying to silo oneself off as an historian of education when there’s all this brain candy going on around you? In my postdoc, it was a bit of a drinking from a firehose experience.

At this point I have more experience and facility with reading empirical education research than I ever had in demography. I’m pretty good at spotting strengths/weaknesses, and will probably catch a majority of major issues in manuscripts regardless of field. Probably.

And unless you have more experience than I do in a specialized field, or are sharper than me (and you probably are! But still), you’d need to spend a lot of time with each article or white paper to evaluate it.

There are hundreds of papers on Covid coming out each day, and many of them are in languages you don’t read or speak.

So it may be intellectually stimulating to ask, “What can I learn about X (Covid, whatever) from this individual paper?” But outside your expertise, you’re going to drown before too long. A question you can answer, with a little more efficiency, is, “What can I learn about X from a chunk of studies?”

It is one thing for a single study to claim that putting on sunblock will help prevent the next pandemic. It’s when a barrel of studies does so that it matters. No, don’t put on sunblock to stop a disease! That was a joke! Yeesh…. I hope you get the point.

Even when you see a review or synthesis of research, there is still the question of study quality. But the nice thing about trying to look for multiple studies is that you don’t have to be an expert, in general. In an area of active study, the researchers in the field critique each other. And you can see much of it. Often that debate takes time. Many years ago, John Hattie started to gather every single meta-analysis (systematic synthesis) in a whole bunch of areas of education research: the visible learning project. What we know now is that he collected everything he could, good and bad.

The collection of every study you can find gives one the rough average of hundreds of hundreds of studies. It’s like the average taste you get when you mix champagne with swamp water.

We know now that you probably want to filter out the swamp water (or its equivalent in research) to keep the champagne (or at least decent wine) — that is, set up some good inclusion/exclusion rules in a meta-analysis.

That internal discussion takes time, and I know it’s frustrating. But with Covid, we’re seeing all of it compressed. The public critique of a PNAS article in less than 24 hours? Amazing.

So… hold off helping that single study go viral outside your area of expertise. Instead, follow people who explain what they learn from multiple studies. Right now, in Covid, it’s going to be pretty rough, and people are going to make errors. As long as you can see the discussion as people wrestle with it, you’re at least following what experts say.

Will the cultural script of “school” change?

A colleague asked me over the weekend if there is any guidance from the history of education on what may change permanently in elementary and secondary schools as a result of the pandemic. There is now a little industry devoted to hot-takes about how this is the “end of ____ as we know it,” and there are plenty of entries in education as well as in other areas of life, from Steven Mintz and Bonnie Kristian in higher education to Conor Williams, David Mansouri, and Diane Ravitch for K-12. So I was not surprised by the question.

A bunch of us (historians of ed) have struggled with the question of what the history of pandemics and school closures may suggest. There are some things we know (about school closures in 1918, the use of “radio schooling” in polio waves in the U.S., radio schooling again in South Korea in the 1950s, etc.), but there isn’t a ready monograph on the history of schooling in pandemics.

There are a few issues that I think come out of that history, and this is primarily to explain how I place the question my colleague raised — and this is focused on elementary and secondary education:1

  • How did these (and other major disruptions – the Revolutionary War and Civil War are big ones) shape children’s lives in a concrete sense? Glen Elder is probably the scholar best known for this type of work with his Children of the Great Depression. But probably the biggest change was the Civil War and the end of slavery – that opened up educational opportunities for formerly enslaved people of all ages.
  • How did these disruptions change Americans’ way of thinking about themselves, and the future? That is intimately tied to how we talk about children and education. The 1918 pandemic was not isolated from the deep unpopularity of WW1, and was quickly followed by a sharp depression in 1919 and the Palmer Raids on alleged anarchist conspiracies (really, an excuse for the federal government to go after a broad range of leftist political organizations) – so, it was a major part of several years of great turmoil. Nicole Hemmer thinks the pandemic is going to dramatically increase our sense of uncertainty (see towards the end of her recent CNN column), and while I think it’s plausible, I’m not so sure what will last. I tend to see more debates coming out of great historical changes than Hegelian-like shifts of uberconsciousness – I wrote about that a little 19 years ago.
  • How did people see the responses to great disruptions – as innovations or as emergency measures? This is the one most clearly tied to the present moment. We know that radio-school programs did not last, and though I have some guesses as to why, I don’t think anyone’s written on that question … yet! For what it’s worth, K-12 schooling was more disrupted by the 1918 pandemic than colleges, because so many colleges had already been disrupted by the war. 

I know that we’ll learn a great deal about emergency responses and related operations – that piece is already clear at Arizona State University, at least from my perspective. How much of that translates into persistent practice, or changed expectations about what education might mean? A few thoughts:

  • I suspect that many parents who have at least some connection with schooling will have one of two impressions, and often both at the same time:
    • Deep sympathy for teachers and other school officials engaging in emergency instruction
    • Deep distrust that K-12 schools can do anything at the moment that is meaningful and educational
  • My impression is that schools have made “modal” choices for how to continue instruction – that is, they choose a mode, and their expectations of teachers, and are just trucking ahead the best they can along the path they chose. What would be a telling sign of district leadership is how they have been fine-tuning their response over the past several weeks – what are they modifying, and how? 
  • The upshot of the items above: it is highly unlikely that the pandemic has upended the cultural scripts of what schools are. Where schools are having any success in continued operations, it is surely tied to maintenance of the most common understandings of “school” as formal academics.
  • Other reasons to suspect that the cultural script for P-12 schooling is going to remain stable, and most of these are about the non-academic roles of schools:
    • Many children have no education at the moment. This is unlike the viciousness of the Prince Edward County School Board, which closed all of its schools from 1959-1964 out of spite at desegregation – but it’s still a huge gap in their academic and other educational experiences, let alone meals, health screening, and related services that piggyback onto formal schooling.
    • Parents who listen to their children are hearing a great deal about the absence in their children’s lives of the non-academic role of schools in their children’s lives – connections with friends, sports and clubs, and even just the opportunity to run around a schoolyard.
    • Parents who are employed are missing child care that schools provide.
    • Parents who have just lost their jobs are desperately fearful now, and likely desperate over the summer. Formal schooling is going to compete in parents’ minds, and their children’s, with nutrition, health care, and shelter. And two of those are partially provided by schools.
    • The recession we are now in is wreaking havoc with state budgets for the next fiscal year. There will be layoffs, and possibly very little wiggle room for rethinking organizational and cultural scripts absent an explicit push from local leaders such as district superintendents or unusual school boards.

School districts may learn (and states may give) more flexibility on some important issues. But millions of parents will be grateful if there can be anything like a normal school year next year. Not that “normal” is what we want it to be.

I do have a suggestion for everyone ready to gear up their chosen bandwagon under the “this absolutely proves that schools must change!” banner: this was the high-pressure approach under the Obama administration in the last recession. That last wave of school reform brought the Common Core State Standards, attacks on teachers and their unions, two giant state testing consortiums, and attempts to tie teacher careers to student tests. How exactly did that prior wave help schools prepare for a pandemic?

Instead of making that argument, I recommend looking to our current period of emergency education as a basis for asking what if? My colleague Ron Beghetto writes about this in Beautiful Risks (publisher; library), and having observed his use of what he calls a possibilities-thinking protocol, I think it’s a useful framework now. A few examples:

  • What if we acknowledged that no educator today has a good handle on how to educate children in a pandemic?
  • What if we then asked families their suggestions for how to set the right expectations when schools are closed?
  • What if we applied those ideas beyond the current emergency — to snow days, to summers?
  • What if schools and families collaborated on the basis of those experiences to look at expectations more broadly?2

People do this type of work in communities, a lot. But because the work is inherently local, it doesn’t attract all the bling that more aggressive efforts take. And there is a place for forceful edicts in terms of civil rights. But we don’t need to delve back to 1918 to see where we were 12 years ago — less disastrous, it turned out, but deeply painful as a recession, and used as a lever for a number of education policy changes. Some stuck, some didn’t, and it all had an argument with the flavor of “schools must change, and now,” what I think of as the Borg fallacy. That was the universal tone, and its use did not predict whether a proposed change in policy and practice lived or died. Time for a different approach.


  1. The classic pieces on cultural expectations of schools are Metz (1989), who uses the term script to refer to those expectations, and Tyack and Cuban (1995), who coined the term grammar of schooling. []
  2. Ann Ishimaru talks about this type of collaboration from a different angle in her new book Just Schools, available from her publisher, independent bookstores, and libraries. []

More candidate education platforming

I have some thoughts about the K-12 policy position released by Elizabeth Warren’s campaign on Monday. The big story told in most reporting is about the quadrupling of federal aid to local schools under Title I, but I have some other observations:

  1. The plan firmly sees education policy as embedded in broader social policies. Here are some of the policy initiatives mentioned in Warren’s plan that would come from outside the U.S. Department of Education:
    • Energy efficiency building upgrades
    • Housing plan grants for communities, tied to incentives to slow/block restrictive zoning
    • Lead abatement
    • Rural broadband expansion
    • Excluding schools from immigration enforcement (by policy, not by practice as happened before the Trump administration)
    • More investigation of potential sham non-profit structures, by the IRS
    • Lobbying reforms that would also apply not just to the federal government but at any level of government receiving grants-in-aid.
    • Aggressive blocking of data mining by the Federal Trade Commission
  2. Much of the plan is regulatory and thus does not require legislative action. A substantial part of the regulatory part of the plan would reverse Office of Civil Rights actions under Betsy DeVos, but there are other regulatory actions embedded in the plan — most notably (to me), the declaration that integration would be an evidence-based intervention under ESSA.
  3. Charter schools serving poor children would probably receive more funding under this plan than currently, because of the quadrupling of Title I aid.1
  4. There appear to be three prominent targets of potential civil lawsuit and regulatory action by the Office of Civil Rights and/or the Justice Department: violations of LGBTQ rights, close attention paid to academic expectations of students with disabilities, and attempts by wealthy, predominantly white communities to advance segregation by creating breakaway local districts.
  5. The plan does not use school to prison pipeline as a term, but the evidence-based school safety paragraph is a direct result of the efforts of activists to tie disciplinary/law-enforcement actions at schools to mass incarceration. Arguably, this is a bigger activist accomplishment than anything in the plan that teachers unions like.
  6. The specific legislative items can be divided into two pieces: large spending/reallocation promises (the expansion of funding for Title I and the Individuals with Disabilities Education Act, most prominently); and some specific targeted legislation around public-employee collective bargaining (again, something outside the U.S. Department of Education), and expanding the right of private individuals to sue under Title VI of the Civil Rights Act.
  7. In the large spending proposals, there is somewhat squishy language about negotiation with schools and educators around rewriting the Title I aid formula and how $100 billion in Excellence Grants would be distributed.
  8. The biggest lightning-rod issues written about thus far, and likely in the near future, are areas where the role of a Warren administration would be most circumscribed.
    • I do not see Congress eliminating the current role of testing in ESSA/ESEA — and the rest of standardized testing use is a matter of state and local policy.
    • Almost all charter-school policy is determined at the state level.
    • ESSA and politics would make it virtually impossible for the federal executive branch to require a culturally-responsive curriculum in states.

This is the longest education proposal thus far in the 2020 campaign, and like all comments I’ve seen thus far, this is incomplete, and I’ve found it hard to address the scope of the Warren plan.2


  1. This will be a big surprise to everyone who looked primarily at the proposed zeroing out of federal aid for new charter schools and the plan’s discussion of for-profit charter schools and for-profit servicers with a cozy relationship to the schools. []
  2. I have also not made any evaluative judgments, as I didn’t when discussing Bernie Sanders’s plan. []

Yes, we can make historical analogies

Yesterday, the U.S. Holocaust Memorial Museum released a statement whose first sentence is stunning in its historical ignorance: The United States Holocaust Memorial Museum unequivocally rejects efforts to create analogies between the Holocaust and other events, whether historical or contemporary. The museum’s leadership is wrong. To put it bluntly, if we are never allowed to draw historical analogies, we remove one of the most important reasons to learn history. There are no exact parallels or repeats in history, but there are patterns, and a significant part of our job as people (let alone my job as an historian) is to learn those patterns and watch for them where we know they’re dangerous.

There is a defensible statement within the misguided claim of the museum’s leadership, but it’s restricted: there is no useful point in comparing the horribleness of genocides. The Shoah is incomparable. So was the slave trade. So was the genocide of indigenous peoples on multiple continents.

But we don’t have to rank-order the horrors of history to learn from them. One can learn from them as singular events, as you would in a seminar class — and Peter Hayes taught such a class on the Holocaust at Northwestern for years (his new book Why: Explaining the Holocaust reflects that focus). One can also learn from putting them into broader context, learning from or teaching about genocides and intolerance more broadly. You learn different things from studying an isolated event from an event in context, and both are valuable.

But you don’t go around telling people not to learn broader lessons from the Holocaust, and that’s what the museum leadership is attempting to do. And that’s just plain wrong for history and for citizenship.

Democratic politics and charter schools, brief gloss

Tl;dr version: don’t waste your energy on trying to suss out The Position on National Charter School Politics. But why it doesn’t make much difference is different from the details of charter-school debates.

Continue reading “Democratic politics and charter schools, brief gloss”