Challenges to internal validity. To understand why many studies of pandemic-induced policy changes could suffer from serious threats to internal validity, first consider a common research design with notably strong internal validity: a randomized experiment. If researchers were to use a randomized experiment to estimate the impact of remote instruction on student learning, they would need to assign students at random to the treatment group (students who receive remote instruction) or the control group (students who receive in-person instruction). Assigning the treatment at random ensures there are no systematic differences between the two groups other than the treatment itself. A randomized experiment has high internal validity because it clearly isolates the impact of a particular treatment.
Quasi-experimental research designs, or “natural experiments,” take advantage of variations in treatment status that occur as a result of policy changes or other “natural” phenomena outside of a researcher’s control. While natural experiments eliminate the need to actively assign individuals to treatment and control groups, they typically face greater threats to internal validity than do randomized experiments. One common type of natural experiment uses an econometric instrument to estimate the effects of a particular policy change. The onset of the pandemic holds obvious appeal as an econometric instrument for studying the effects of online instruction, since the crisis caused a sudden shift to remote teaching. A key assumption of this research design, however, is that the policy change of interest (the shift to online learning) does not coincide with other relevant changes. In econometrics, this is known as the “exclusion restriction,” which requires that the econometric instrument (the pandemic) affect the outcome of interest (student learning) only through the policy change of interest (the shift to online learning) and not through other channels.
While the Covid crisis did spur the shift to online instruction, it fails the exclusion restriction because of the many contemporaneous changes that likely also affected student learning. Students shifted to remote learning as their parents lost jobs, as their family members suffered Covid’s health effects, and as they lost the ability to leave the house and see friends, among other significant changes to their lives. If Covid-19 did affect student learning, it would be difficult to attribute the changes in outcomes to remote instruction rather than any of these other contemporaneous factors. We illustrate this in Figure 1, which shows the potential use of Covid-19 as an econometric instrument for remote instruction. The exclusion restriction requires that there be no “causal arrow” between the other channels affected by Covid-19 and student learning, an assumption that is certainly violated based on both common sense and prior empirical evidence. It may be possible to estimate the overall effect of Covid-19 on student outcomes, but attributing that effect to any one channel is likely impossible.
We can illustrate the violation of the exclusion restriction with an example from our own recent research into Covid’s impacts on household Internet-search behavior. In that work, we show that Covid-induced school closings caused parents to seek out online learning resources that might compensate for lost in-school instructional time. An example of this can be seen in the top panel of Figure 2, which shows a large increase in Google searches for “online learning” that corresponded precisely with the timing of the pandemic outbreak in the United States. This provides evidence that the Covid-19 crisis indeed represents a sudden shock to the demand for online learning resources.
At the same time, however, there were many other changes in students’ lives that are reflected in Internet search behavior. Data show, for example, that there were sudden and contemporaneous increases in Google searches for terms relating to the economic condition of households, such as “unemployment insurance” and “food stamps.” The pandemic changed students’ educational experiences but also generated a considerable economic shock to many households. These large, simultaneous changes make it difficult to separate the effects of one shock from another.
Challenges to external validity. The unprecedented circumstances surrounding the Covid-19 crisis also present serious challenges to external validity. Researchers typically examine whether the context and implementation of a policy shift are representative of potential future enactments of the same policy. Neither the context nor the implementation is likely to be representative in this case.
First, the learning environment during the pandemic is unlikely to generalize to typical school years because of the many changes in students’ lives that likely put a strain on their capacity to learn. For instance, students and young adults have reported substantial increases in anxiety and depression during the pandemic. Second, pandemic-induced policy changes were implemented in a way that is unlikely to resemble a more carefully planned implementation of the same policy in a typical year. For example, the pandemic-induced shift to online learning required teachers, with no advance warning, to quickly redesign lessons originally intended for in-person instruction. Under normal circumstances, teachers would have been afforded time to prepare lessons specifically for online instruction. Even in the fall of 2020, there was still substantial uncertainty around schooling logistics and instructional modality, making it difficult for educators to plan instruction effectively in advance. The unprecedented circumstances of the pandemic and the corresponding ad hoc policy shifts are therefore unlikely to generalize to well-planned policy changes in a typical school year.
You must be logged in to post a comment.