Asking About Causes Without an Experiment — Natural Experiments and Four Designs

In summer, ice cream sales go up, and in the same season, drowning accidents also increase. Even so, no one says that ice cream causes drownings. This is simply because both are pushed up together by a third cause called "heat." In this example, it is easy to distinguish between correlation (a relationship where two values go up and down together) and causation (a relationship where one actually causes the other). However, most real-world questions are not this friendly. Does raising the minimum wage reduce jobs? How much does going to college raise income? Does a certain medicine really extend life span? In questions like these, hidden "heat" is everywhere.

The traps that block causation are broadly three. Below, treatment refers to the intervention whose effect we want to know (such as a policy, a medicine, or educational attainment), and outcome refers to the value we want to see changed after that intervention (such as employment, income, or life span).

• Confounding is a hidden factor that moves the treatment and the outcome at the same time. The heat between ice cream and drownings is like this.
• Selection bias is the problem that people who received the treatment and those who did not were different people to begin with. People who go to college may have higher income potential originally.
• Reverse causation is the possibility that the direction of cause and effect is reversed. This is the case when it is impossible to know whether people drink more coffee because they are healthy, or whether coffee protected their health.

Unless these three are cleared away, no matter how much data there is, correlation cannot become causation. How confounding and adjustment are used in papers was covered in What "Adjusted for Age, Sex and BMI" Actually Did — How to Read Confounding and Adjustment.

The cleanest solution is the randomized controlled trial (RCT). If you flip a coin to divide a treatment group and a control group, the two groups are on average the same in every aspect, whether observed or not, and differ only in whether they received the treatment. Therefore, you can attribute the difference in results to the treatment. Whether the RCT is truly the 'Gold Standard' was examined separately in Is the Randomized Controlled Trial Really the 'Gold Standard'?. This article deals with cases where you get blocked before that point. You cannot randomly decide to raise the minimum wage in some cities and not in others. You cannot draw lots among citizens to send some to college and block others. Large questions dealt with by economics, sociology, and epidemiology are usually impossible or unethical to experiment on.

So, over the past half-century, social science has found another path. If you cannot run an experiment, you find and use 'near-random' situations that the world accidentally created. This article opens that toolbox. The goal is to enable you, when you open an empirical economics or epidemiology paper, to read what the authors are comparing with what to claim causation (identification strategy). For that comparison to be accepted as pointing to a causal effect, researchers must believe conditions that the data does not tell you. These conditions are called identifying assumptions, and the whole article follows these assumptions. Since most of the evidence consists of arXiv preprints which are manuscripts before peer review, please read the notice at the end of the article together.

Comparing With a World We Never Saw — The Mental Framework of Counterfactuals

The starting point of modern causal inference is a simple and uncomfortable observation. There is the income of a person when they go to college, and the income of that exact same person when they do not go to college. The difference between these two is the real effect that college had on that person, and together they are called potential outcomes. The problem is that only one of the two can be observed. The income that a person who went to college would have received "if they had not gone" can never be seen. This invisible world on the side that did not happen is called the counterfactual. The fundamental problem of causal inference is how to fill in the counterfactual.

RCTs solve this problem through random assignment. This is because the control group, which has become statistically identical, takes the place of the treatment group's counterfactual. Causal inference without experiments does the same thing. However, instead of the researcher creating random assignment, it finds "nearly random" assignment that already exists in the world. The methodology of modern social science causal inference is a body of literature that has systematized these identification assumptions and countermeasures for when those assumptions are broken (arXiv:2408.00032).

Natural Experiments — The Coin Toss Tossed by the World

Most of the foundational studies that opened this toolbox predate 2017, around the time economics began uploading papers to preprints (arXiv). This is true for Card and Krueger's 1994 minimum wage study (difference-in-differences comparing fast-food employment across the New Jersey-Pennsylvania border), Angrist's instrumental variables study using the Vietnam War draft lottery, the formalization of the Local Average Treatment Effect (LATE) by Angrist and Imbens, and Card's 1990 Mariel boatlift study. Therefore, this article cannot directly cite the original papers, and instead relies on modern methodological papers that summarize and expand these methods today. The events of the founding era (the 2021 Nobel Prize in Economics to Card, Angrist, and Imbens, and the name given to that movement, the "credibility revolution") are described as widely known facts, while the explanations of the methods themselves lean on modern papers.

Until the 1980s, empirical economics relied heavily on complex regression models (statistical methods that fit relationships between variables into equations) stuffed with all sorts of variables, mimicking "all else being equal" statistically. However, hidden variables that failed to make it in always remained, and conclusions tended to sway depending on how the model was built. By the 1990s, the tide had turned. Researchers began looking for clean comparison situations instead of elaborate models. These are moments where a policy applied to one region but not a neighboring region, a law was split based on a specific date of birth, or something was assigned by a lottery. Such situations are called natural experiments. This methodological shift was later dubbed the "credibility revolution," and the 2021 Nobel Prize in Economics honored that contribution (this naming and award are widely known facts).

The term "natural experiment" is actually used loosely. There is a usage that refers only to cases where someone other than the researcher (the government, nature, an institution) truly assigns the treatment randomly, and there is also a usage that refers broadly to observational studies lacking controlled random assignment. One methodological paper attempts to rigorously define a natural experiment. In this definition, the assignment mechanism must be (i) not designed or executed by the researcher, (ii) unknown to the researcher, and (iii) probabilistic depending on external factors. This paper adds a warning here. Because the difference between a natural experiment and a randomized experiment is one of "essence" rather than "degree," treating a natural experiment identically to a randomized experiment is neither rigorous nor useful (arXiv:2002.00202). Keeping this warning in mind, we examine four representative designs one by one.

Design 1 — Difference-in-Differences (DiD): Looking at the 'Difference in Changes'

Question it answers: When a policy is implemented for one group and not another, what is the effect of that policy?

The intuition is this. Looking only at how much the outcome in the treatment group (the group that received the policy) changed before and after the policy is not enough. That is because the economy, the seasons, and the whole world would have changed together in the meantime. So, we subtract the change over the same period in a control group that did not receive the policy. The change in the treatment group minus the change in the control group—in other words, the "difference in differences"—is the difference-in-differences estimate of the policy's effect. Card and Krueger's classic study is known for subtracting the employment change in neighboring Pennsylvania, which did not raise the minimum wage, from the employment change in New Jersey, which did.

In its simplest form, the procedure is a 2×2 table multiplying two groups (treatment and control) and two time points (before and after). You calculate the mean of each cell and compute (treatment after − treatment before) − (control after − control before). This canonical form is well understood. Real-world research goes far beyond this. It handles multiple time points, adjusts for other characteristics of the groups (covariates), applies weights, and even deals with "staggered adoption," where the policy is introduced at different times in different groups. Because practice easily gets messy in these extensions, a recent practical guidance paper presented a framework organizing the types of DiD designs and their corresponding estimators (formulas for calculating the effect) (arXiv:2503.13323).

The results are usually read through an "event study" plot. This plot is considered central to DiD analysis (arXiv:2512.06804). The horizontal axis is relative time with the policy implementation set to 0, and the vertical axis is the gap in outcomes between the treatment and control groups. The points before time point 0 should lie flat near 0. This is because it means the two groups moved side by side during the time before the policy. If the gap widens after time point 0, that widening is the policy effect changing over time.

The core assumption of DiD is parallel trends. It is the assumption that, had there been no policy, the outcomes of the treatment and control groups would have moved side by side (in parallel). The most important trap, which we will cover later, lies right here. The fact that the gap in the pre-policy period is flat serves only as circumstantial evidence for this assumption, not proof.

Design 2 — Instrumental Variable (IV): A Lever That Reaches Only Treatment

Question it answers: When a simple comparison goes wrong because people who chose the treatment and those who didn't were different to begin with (selection bias and reverse causality), how do we extract the causal effect of the treatment?

The intuition is this. We find an external lever that affects the treatment (smoking, going to college, drinking) but reaches the outcome exclusively through the treatment. This lever is the instrumental variable (IV). If the lever pushes and pulls the treatment "almost at random," we can work backward to calculate the effect of the treatment through the ripple effect of that push and pull on the outcome. Angrist using Vietnam War draft lottery numbers as an instrument to estimate the effect of military service on later earnings is a textbook example. Because the lottery numbers are determined at random, they are not entangled with individual tendencies. However, the exclusion restriction—that the numbers do not affect earnings through any path other than military service—must still be argued.

Two conditions are required to be an instrument. The first is relevance, meaning the instrument must actually move the treatment. The second is exclusion restriction, meaning the effect of the instrument on the outcome must occur strictly through the treatment and must not leak out through other paths. For instance, there are analyses that use genetic variants as an instrument for alcohol consumption. This is a design relying on the assumption that genes are determined at birth and are less entangled with lifestyle habits, which epidemiology calls 'Mendelian randomization.' However, if that gene affects the outcome through another path, the exclusion restriction breaks down, so this assumption also requires argumentation. A paper dealing with testing methods for handling multiple weak instruments (arXiv:2604.15437) applied this design to UK Biobank data, showing an example of studying the effect of alcohol consumption on body mass index (BMI).

The standard method for the procedure is two-stage least squares (2SLS). In the first stage, the instrument predicts the treatment, and in the second stage, that "predicted treatment" explains the outcome. Because it selectively uses only the variation pushed by the instrument, it bypasses the part where the treatment was self-selected (the contaminated part).

There is something easily missed when reading the results. The value estimated by IV is different from the average effect for everyone. It is the effect on the people who changed their treatment in response to the instrument—in other words, 'compliers.' In the draft lottery example, only the effect on people who enlisted because they got a bad number (and wouldn't have enlisted if they got a good number) is identified. This value is called the Local Average Treatment Effect (LATE), and its formulation is known to date back to a 1994 study by Imbens and Angrist. There is also research that has extended that assumption to a dynamic setting (arXiv:2405.01463). LATE is powerful, but it is a restrictive figure that requires us to always ask "whose" effect it is.

Design 3 — Regression Discontinuity (RDD): Right on Either Side of the Threshold

Question answered: When a qualification or treatment is split strictly by the cutoff of a continuous score, what is the effect of that treatment?

The intuition is like this. Let's say 60 points is the passing line on a test. Two students who scored 59 and 61 are practically the same in ability, but one fails and the other passes. People right on either side of the threshold are 'almost identical' subjects, yet only their treatment is split, so within this narrow window we can view it almost like random assignment. The size of the discontinuous jump in outcomes right above and below the threshold is the effect of the treatment, and this method is the regression discontinuity design.

The procedure goes like this. We put the score on the horizontal axis (called the 'running variable' because it's the value that splits the treatment) and the outcome on the vertical axis. After estimating the trend of the outcome on the left and right of the threshold respectively, we measure the 'jump' that opens up between the two trends at the threshold. The core identifying assumption is continuity. It is that in the absence of treatment, the expected value of the outcome would have continued smoothly across the threshold (arXiv:2507.12693).

An introduction to regression discontinuity in medical and health sciences is organized in arXiv:2508.03878. When reading the results, we look at the size of the jump at the threshold and how stably the trend lines on either side were estimated. Sweden provides an impressive case. Sweden stopped neonatal BCG tuberculosis vaccination in 1975, and the vaccination rate plummeted from 92% to 2% in a freefall. One preprint study compared cohorts of about 1 million people each (groups born in the same period) born right around that boundary using regression discontinuity to test whether BCG vaccination received as a newborn prevents COVID-19 (arXiv:2006.05504). The authors conclude that they obtained strong evidence that vaccination received as a newborn provides no protective effect, adding that the effect of recent vaccination must be evaluated separately. It was precise enough to rule out at the 95% confidence level even the possibility of reducing confirmed cases by more than 20%. Without designing an experiment, a single date drawn by a system effectively became a massive natural experiment.

Design 4 — Synthetic Control: Creating a Virtual Twin

Question answered: What do you do if there is only a single treated subject (one country, one city, one state)? How do you draw the counterfactual when there is no control group to compare against?

The intuition is this. If there is no suitable single control, you take a weighted combination of several candidate controls to synthesize a "virtual twin" that closely resembles the treated subject. This is synthetic control. For instance, the counterfactual for a state that implemented a specific policy is replaced by a "synthetic state" constructed by weighting several other states so that their pre-policy trajectory matches that state most closely. The gap that opens up between the actual subject and this synthetic twin after the treatment is the estimate of the policy effect.

The weights are determined to match the pre-treatment period outcomes (and predictors) as closely as possible. While this method is intuitive, it carries the risk of overfitting. If you try too hard to match the previous period perfectly and end up mimicking even the noise, subsequent predictions become distorted instead. Therefore, practical guidelines for synthetic control recommend guarding against overfitting bias and establishing interpretability and validation (placebo) procedures for the results (arXiv:2203.06279). Recently, it has also been extended to high-dimensional settings with multiple treated subjects, and was used to estimate the effect of COVID-19 stay-at-home orders on unemployment rates across US counties (arXiv:2510.22828).

Common Misunderstandings and Limitations — Tools Rest on Assumptions

Up to this point, it might sound as though these designs are a magical substitute for experiments. The true virtue of these tools does not lie in "proof." Each design rests on assumptions that cannot be confirmed by data alone, and its honesty lies in making those assumptions transparent. Below are traps that practitioners frequently stumble into.

① Parallel trends is an assumption, not a test. The parallel trends assumption is a statement about a counterfactual: that the two groups would have moved in parallel if the policy had not existed. Because that counterfactual is never observed, it cannot be directly tested in principle (arXiv:2510.26470). Researchers often feel relieved after seeing that trends in the pre-policy period were parallel, but this is merely circumstantial evidence, not a guarantee. Parallel trends can also depend on the scale. Trends that are parallel when looking at the original values may become non-parallel when converted to logs (values transformed into multiplicative units). This means that the parallel trends assumption may or may not hold for the exact same data depending on the scale from which it is viewed (arXiv:2010.04814). Some research outlines conditions using causal diagrams (drawings of cause-and-effect arrows between variables) that can rule out parallel trends in advance. One example is when the treatment is affected by past outcomes (arXiv:2505.03526). Ultimately, the core of justifying parallel trends is an argument about how the treatment was assigned (the selection mechanism), and strategies for benchmarking that argument with data have also been proposed (arXiv:2203.09001).

② There is a trap in pre-trend testing for event studies. There is a widespread practice of statistically testing whether "pre-policy points are flat," and proceeding with the analysis with peace of mind if they pass. However, this pre-test itself can distort subsequent estimation and inference. Recent studies point out that such preliminary tests lead to low statistical power (weakness in catching existing violations), bias, and confidence interval undercoverage (where the proportion of confidence intervals capturing the true value is lower than the indicated level), and have proposed procedures to overcome this (arXiv:2510.26470). More fundamentally, it has been pointed out that standard event study plots cannot yield honest causal inference if parallel trends or "no-anticipation" (the assumption that people do not react in advance before a policy is implemented) are violated. Methods to remedy this and make the plot itself a rigorous inference tool have also been proposed (arXiv:2512.06804).

③ Difference-in-differences with Two-Way Fixed Effects (TWFE) regressions can be biased in staggered adoption. TWFE regressions control for baseline differences across groups and time periods, respectively. They have long been used as the standard in common situations where policies are adopted at different times across groups, but it has recently come to light that these regressions can distort causal effects. Regression coefficients are a weighted sum of the average treatment effects for each group and time period, and some of those weights can be negative. This can lead to the absurd outcome where the regression coefficient turns out negative (−) even though the true effect is positive (+) in all groups (arXiv:1803.08807). One survey summarizes the scale of the problem like this: 26 out of the 100 most-cited papers in the American Economic Review (AER) between 2015 and 2019 used this regression, and when effects vary across groups and time periods, this regression can produce misleading estimates (arXiv:2112.04565). Since then, alternative estimators robust to heterogeneous effects (effects that differ by group and time period) have flooded in. One example is an alternative estimator that views event studies as "filling in missing data" (arXiv:2108.12419). This alternative is not free either. It often has high variance, leading to low statistical power in many applications, leaving a trade-off between bias and variance (arXiv:2503.05125). Because much of this discussion involves methodologies at the preprint stage that are still actively debated, it is more honest to read them as ongoing improvements rather than settled dogma.

④ The Achilles' heel of IVs is weak instruments and exclusion violations. Both assumptions of instrumental variables are fragile. It is known that standard inference becomes difficult if an instrument moves the treatment only weakly (weak instruments), and methods for testing when there are multiple weak instruments have been separately studied (arXiv:2604.15437). A more fundamental risk is the exclusion restriction. If an instrument leaks into the outcome through a path other than the treatment, the estimation is contaminated, and this assumption usually cannot be directly verified with data. It has been proven that the validity of an instrument cannot be tested in principle when the treatment is a continuous variable. However, testability can theoretically be recovered by imposing weak structural assumptions such as continuity between observed variables (arXiv:1806.09517). Commonly used falsification tests (tests that deliberately search for traces that would appear if the assumptions were wrong) are also imperfect, and can mistakenly flag problems even when a valid instrument is used (arXiv:2312.15624). Ultimately, the exclusion restriction must be defended with arguments, not statistics.

⑤ LATE is the effect for "compliers." As seen earlier, the value provided by an IV is the effect on compliers who responded to the instrument (LATE), and there is no guarantee that they represent the entire population. The interpretation becomes more subtle if the data has a cluster (units bundled together like the same school or region) structure and the effects differ across clusters. In this case, two-stage estimation with cluster indicators gives a weighted average of cluster-specific LATEs, but standard 2SLS generally fails to identify even such a weighted average (arXiv:2601.13507). A study re-examining the effects of a job training program using sensitivity analysis (an analysis examining how much omitted variables could shake the results) found that the program's effects were robust for women but fragile for men (arXiv:2604.03544). "An effect on whom, and how robust is it?" is not a peripheral question.

⑥ RDD is local and vulnerable to manipulation. What RDD identifies is the effect right around the threshold. When the cutoff score is 60, there is no guarantee that the result applies to someone with a score of 30 or 90 (locality). Another risk is manipulation. If the cutoff is widely known and the treatment is valuable (such as government benefits), people may push the running variable past the threshold. This breaks the premise that "the two sides are nearly identical." This is a real enough threat that methods to counter such strategic behavior are studied separately (arXiv:2507.12693).

⑦ External validity: Is what is true here also true there? Because natural experiments yield results obtained in a specific time, place, and group, whether they generalize to other contexts (external validity) is another issue. One study replicated Angrist and Evans's famous natural experiment (the effect of sibling sex composition on fertility and labor supply) over 100 times using census data from various countries around the world. Using this replicated data, they decomposed the sources of error that arise when predicting treatment effects in different contexts. In this replicated data (effects on fertility and labor supply), macro (country and era) conditions were more important than micro conditions in reducing errors (arXiv:1906.08096). Even an effect precisely estimated in one place must be transferred to another with caution.

Conclusion — The Discipline of Asking

The title of this article is not how to 'prove' causality without experiments, but how to 'ask'. This is because if the assumptions hold, the causal effect is identified, but those assumptions cannot be verified with data. Therefore, it is difficult to establish causality definitively with any of Difference-in-Differences, Instrumental Variables, Regression Discontinuity, or Synthetic Control. Indirect checks like parallel trends or placebo tests can be done, but they do not prove the assumption itself. Difference-in-Differences relies on parallel trends, Instrumental Variables on exclusion restrictions, and Regression Discontinuity on continuity, and when these assumptions break down, the conclusions collapse. Synthetic Control has no guarantee that a good pre-treatment fit will continue after the treatment, so overfitting and validation procedures must be considered.

Right there lies the real power of these tools. They force us to reveal 'what is being assumed' explicitly and falsifiably. A good causal inference paper puts its identification assumptions front and center without hiding them, argues why those assumptions are plausible, and reports together how well the conclusion holds up when the assumptions are shaken. Instead of insisting that correlation is causation, it honestly states, "Under these assumptions, this is causation." Causal inference is not a technique of perfect proof, but a discipline of making assumptions transparent.

Now, whenever you open any empirical paper, the first question converges to one. "What are the authors comparing to what, and on what grounds do they believe that comparison is 'almost random'?" If you can ask this question, you have already started reading data with the eyes of causality.

---

This article covers a rapidly evolving methodology field where some parts are still debated. A significant portion of the cited evidence consists of preprints prior to peer review (arXiv), and the discussions surrounding staggered adoption DiD and two-way fixed effects in particular are ongoing methodological improvements rather than established dogma. Conclusions may change.

Source Papers (arXiv)

• arXiv:2002.00202 — Definition of natural experiments: The essential difference between 'as-if random' assignment and randomized experiments
• arXiv:2408.00032 — Overview of modern causal inference methodologies in social sciences (identification assumptions, doubly robust ATE)
• arXiv:2503.13323 — Practical guidelines for difference-in-differences (DiD): Design typologies and estimation framework
• arXiv:2512.06804 — Honest event study plots: A functional data approach
• arXiv:2510.26470 — Valid inference for tests of parallel trends violations: Low power, bias, and undercoverage of pre-tests
• arXiv:2010.04814 — When are parallel trends sensitive to functional form (scale dependence)?
• arXiv:2505.03526 — Ex-ante evaluation of parallel trends using causal diagrams
• arXiv:2203.09001 — Selection and parallel trends: Selection mechanisms and DiD bias
• arXiv:1803.08807 — Two-way fixed effects estimators under heterogeneous treatment effects (the negative weight problem)
• arXiv:2112.04565 — Survey on TWFE and DiD with heterogeneous treatment effects (AER most cited 26/100 papers)
• arXiv:2108.12419 — Rethinking event study designs: Robust and efficient estimation ('imputation' alternative estimators)
• arXiv:2503.05125 — When is two-way fixed effects regression acceptable (bias-variance trade-off testing)?
• arXiv:2405.01463 — Dynamic local average treatment effects (LATE) and the Imbens-Angrist (1994) assumptions
• arXiv:2601.13507 — 2SLS and LATE in clustered data: Weighted average LATE under cluster heterogeneity
• arXiv:2604.15437 — Jackknife instrumental variable inference (many weak instruments; genetic variants for alcohol consumption → BMI)
• arXiv:2312.15624 — Negative control falsification tests in instrumental variable designs (exclusion restriction, valid instrument false positives)
• arXiv:1806.09517 — The unfalsifiability of instrument validity under continuous endogenous variables
• arXiv:2604.03544 — Quantifying omitted variable bias in nonlinear IV (LATE, JTPA, robust for women / vulnerable for men)
• arXiv:2508.03878 — Introduction to regression discontinuity (RD) designs in medical and life sciences (continuity assumption)
• arXiv:2006.05504 — Regression discontinuity of the Swedish BCG vaccination suspension natural experiment (absence of COVID-19 protective effects)
• arXiv:2507.12693 — Placebo discontinuity designs: Running variable manipulation and strategic behavioral responses
• arXiv:2203.06279 — Synthetic control in practice: Overfitting bias, interpretability, and validation principles
• arXiv:2510.22828 — Synthetic control for high-dimensional disaggregated data (COVID-19 stay-at-home orders → county unemployment)
• arXiv:1906.08096 — External validity of natural experiments (Angrist & Evans 100+ replications, macroeconomic conditions dominate)