Psychology's Replication Crisis — Why the Findings We Believed In Wobbled

There were findings that filled psychology textbooks and TED stages. I will pick just three. Ego depletion is the theory that willpower is like a muscle that runs down the more you use it. Power posing is the claim that holding a confident posture for 2 minutes changes your hormones and confidence, and social priming is the claim that a subtle word or picture changes a person's subsequent behavior. All were intuitive stories that could be summed up in a single line. Many people accepted these as "facts proven by science."

Yet over the past decade or so, a large part of that belief has shaken. This is because when other laboratories ran the same experiments again, the same results often did not come out. The property of getting the same results when you run the same experiment again is called replicability, and the situation where non-replicable findings piled up and shook the trust of the entire field is called the replication crisis. This article follows what was not replicated, why that happened, and how psychology changed its research methods afterward. I will also point out along the way that saying something was not replicated does not mean there is no effect.

The Outbreak of the Crisis — Findings That Disappeared When Tested Again

The trigger was 2015. A large-scale project (Open Science Collaboration) uniting dozens of laboratories worldwide redid 100 studies published in major psychology journals and announced the results. 97% of the original studies reported statistically significant results. Statistically significant means passing a statistical threshold stating that a difference this large is hard to get by chance alone ("What Does "Statistically Significant" Actually Say? — The p-Value, Explained with a Coin Toss"). Yet only 36% yielded significant results in the replication. The size of the replicated effect, meaning the measured magnitude of the difference, was also about half of the original (PMID 26315443). Resulting from redoing 100 studies across 3 psychology journals, this was a figure that shook trust across the entire field.

Rebuttals to this result arrived immediately. One critique pointed out statistical errors in the original papers and argued that the reproducibility of psychology is actually high (DOI: 10.1126/science.aad7243), and we will not cover the counter-rebuttals by the original authors here. Yet it is worth remembering that the starting point of the crisis was a debate itself.

A more fundamental problem also existed. While people say replication is the cornerstone of science, studies redoing the original experiment as closely as possible—meaning direct replications—were rarely published as papers. Among 84,834 papers published between 2010 and 2021 by the top 100 psychology journals by impact, only 0.2%, or 169 papers, were direct replications. Out of 88 journals, 54 did not publish a single one during those 11 years (PMID 39418470). However, the authors draw a line, noting that this count did not tally replication results released through other channels outside of journals.

When multiple laboratories simultaneously re-verify the same effect using the same procedure, it is called a multi-site replication project. A review examining 36 multi-site replication projects in social psychology found that only 4 (11%) were clearly successful, 5 had mixed results, and the remaining 27 (75%) failed (PMID 36442681). However, the same review considers that the cause of failure was not solely errors in the original hypotheses. There were also problems on the side of the replication studies, notably the widespread low engagement of participants. The rare real interpersonal interaction procedures performed much better.

Ego Depletion — Does Willpower Really Run Out?

The most iconic case was ego depletion. It is the theory that self-control relies on a limited mental resource, so once you use self-restraint, your self-restraint drops on the next task, and hundreds of studies seemed to back this up.

However, in 2016, when 23 labs tested ego depletion using the same procedure with their plans published in advance (preregistration), the effect size ($d$) was nearly zero. $d$ is the difference between two group averages divided by how scattered the data is, and 0 means there is no difference. The result of this test was $d=0.04$, and the confidence interval—the range where a real effect could reasonably lie—included 0 (PMID 27474142). Another large-scale test involving 3,531 participants across 36 labs was not significant in the pre-planned analysis ($d=0.06$). However, in an exploratory analysis from the same study using the full sample without exclusion criteria, it was significant at $d=0.08$ (PMID 34520296).

The results do not point in just one direction. A multi-site study of 12 labs reported a small but significant effect ($d=0.10$) (PMID 34113424). One online experiment (a single study) failed to successfully induce depletion, failing to confirm the prediction that people grow more depleted as time goes on (PMID 40452478). Rather than discarding ego depletion, a preprint proposal suggested reinterpreting it as "temporary cognitive fatigue" (DOI: 10.31234/osf.io/sy2mw). A preprint analysis also notes that the size of the effect has been steadily shrinking since roughly a decade before the replication projects. The authors suggest this decline runs parallel, at least in part, to a trend where more rigorous and less biased studies are published (DOI: 10.31234/osf.io/esjhp). In an anonymous survey inviting 1,721 ego depletion researchers, 37.7% of the 277 respondents reported using questionable research practices, while over 60% reported conducting a priori power analyses (PMID 29940020).

Why findings get smaller when you measure them again, and how to read these numbers, are covered in detail in The Shrinking Effect — Why Textbook Findings Get Smaller When You Measure Them Again.

Power Posing and Social Priming — Stories That Spread on Stage

Power posing is the claim that two minutes of a confident posture raise testosterone, lower the stress hormone cortisol, and increase the tendency to take risks. Starting from a 2010 study, it became a global sensation (DOI: 10.1177/0956797610383437). Subsequent replication attempts (preprints, single studies) failed to replicate the testosterone effect and the "feeling of power" effect. However, the same study supported the original results regarding the reduction of cortisol and risk-taking (DOI: 10.21203/rs.3.rs-3970076/v1). Only parts were replicated in this single study. Other replication attempts were not covered in this article, so this is not a verdict on power posing as a whole.

Money priming, a branch of social priming—the claim that subtle cues reminding people of money change their political views—also failed to show evidence of a priming effect in a large-sample re-examination (single study) (d=0.03). A multi-site project covering this effect also saw only one out of 36 laboratories find the original effect (PMID 26214168). Some papers have presented preliminary evidence that research built on the premise that "the self is an empty vessel filled by the situation," such as social priming or embodied cognition (the research stream that the state of the body influences thoughts and emotions), is particularly unlikely to replicate (PMID 41493875).

Why Did It Shake — Structural Causes

It was not just the dishonesty of individual researchers. The background of the crisis most frequently pointed to is the incentive structure that determines which research gets rewarded. However, explanations such as low statistical power, heterogeneity of effects, and base rates are also presented alongside it.

Low statistical power. Statistical power is the probability of actually catching a true effect when it exists, and it gets lower when the number of people participating in the study (the sample) is small ("Where Did 0.05 Come From? — Counting How Many Patients a Finding Rests On"). An analysis that gathers multiple study results together and recalculates them is called a meta-analysis. In a meta-analysis pooling 12,065 effect sizes across 200 research fields, the median statistical power of psychology studies was only about 36%, and studies with sufficient power (80%) were only about 8% (PMID 30321017). The same analysis states that the residual reporting bias itself was small, and traces the prime suspects of replication failure to low statistical power and high heterogeneity (the scatter of results among studies measuring the same effect) rather than bias. A study analyzing 35,515 psychology papers from 1975 to 2017 also reported that statistical power fell short of recommended lines and that publication bias and p-hacking were substantial (large effects are the exception, and this estimate relies on several untestable assumptions). However, those authors' conclusions were on the somewhat more optimistic side that, despite this, most results reported as significant are likely substantial discoveries rather than statistical artifacts (PMID 37847689).

Publication bias. Academic journals prefer "significant" results. Results with no effect (null) often fail to be published (the degree varies by field and study type), leaving a relatively large number of significant results behind. When this bias piles up, the entire literature can look inflated ("Same Trials, Different Conclusions — Publication Bias and What Went Into the Pool"). One consensus report emphasized that to solve this long-standing problem, a way must be created to publish negative and null results (PMID 40991668).

p-Hacking and researcher degrees of freedom. The p-value is a number measuring the probability that a difference this large would show up by chance even though there is actually no difference, and it is judged as significant when it is smaller ("What Is p = 0.03 the Probability Of? — The One Question a p-Value Answers"). Researcher degrees of freedom are the arbitrary choices researchers can make, such as analysis methods, variable processing, and when to stop collecting samples. If there are many choices, countless results come out of a single dataset, and p-hacked results are those where only the ones that came out significant are cherry-picked and reported. How it works is covered separately in "Researcher Degrees of Freedom and p-Hacking — How You Get the Result You Wanted Without Faking Anything". One study took data from registered replication reports where multiple labs repeated the same experiment through preregistration procedures, and simulated what bias would have occurred if there had been no preregistration. Even with the same data, effect sizes swung wildly depending on analytical choices, and out of countless analytical pathways, only about 30% of studies yielded a significant effect in the direction of the hypothesis (PMID 37166859). There is simulation evidence that if you can choose a cutoff line dividing continuous values into two by trying it multiple times, you can create results that look like an effect exists even when it does not (PMID 41233729). p-hacking strategies are so diverse that they are organized into catalogs rather than just one or two (PMID 36778954).

The prevalence of questionable research practices (QRPs). QRPs are not outright data manipulation, but gray-area practices that depart from responsible research. p-hacking, selective reporting of not liking results, and HARKing (Hypothesizing After the Results are Known)—writing as if hypotheses were formed after seeing the results—fall under this. A classic study surveying over 2,000 psychologists viewed that some QRPs may be "dominant research norms" (PMID 22508865). In a recent study surveying about 4,700 researchers from 5 countries including Denmark regardless of field, 9 out of 10 respondents admitted to using at least one QRP. However, the same survey drew a line, noting that compared to responses of having used them even once, the actual frequency of use was much lower, which does not suggest the "systematic use" of specific practices (PMID 39133711). One perspective even argues that these practices do not merely reach a "questionable" level, but amount to violations of the American Psychological Association ethical code (PMID 39964511).

Reform — Science Fixing Itself

The crisis did not end in despair. Psychology has been diligently building tools to fix itself over the past decade.

Preregistration is locking in your hypothesis and analysis plan publicly before looking at the data. It separates 'prediction' from 'retroactive fitting' to restrict researcher degrees of freedom (preprint chapter before peer review) (DOI: 10.31234/osf.io/ud67s). But it is not a cure-all. While structured preregistration templates reduced degrees of freedom better, they did not eliminate them entirely (PMID 33296358). Preregistered studies did not report positive results any less often than unregistered studies, and the authors wrote that they found no definitive evidence that preregistration stops p-hacking and HARKing. However, preregistered studies performed power analyses more frequently and had larger samples, and the authors consider there to be gains in statistical power and impact (PMID 37950113).

Registered Reports are a publishing format where a study is reviewed and acceptance is guaranteed at the research design stage, before results are known. Because publication is guaranteed whether the results are significant or not, this reduces outcome-based selection at the publication stage. One essay (preprint before peer review) states that registered reports publish null results at a higher rate—54% versus 21%—though the denominator is not specified in the piece (DOI: 10.2139/ssrn.6436338).

Many-labs collaborations are a method where multiple labs together verify a single lab's fluke. One study reported that 86% of newly discovered social behavior effects were replicated when four labs applied rigor-enhancing practices, but this paper was subsequently retracted (PMID 37945809). This article does not use those results as evidence of reform.

Transparency and openness. Guidelines and checklists to standardize statistical reporting are also being proposed (PMID 41238797). However, reports note that data sharing alone is not enough. In a preprint observational study that reanalyzed 25 papers with open-data badges from a single journal (Psychological Science) using the same data, large numerical discrepancies initially appeared in 64%, and even after consulting with the authors, 40% could not be fully reproduced. The main cause was unclearly written analysis procedures, and the authors wrote that the original conclusions appear unaffected (DOI: 10.31222/osf.io/h35wt). Since this involves reanalyzing the same data, it is different from the reproducibility mentioned earlier (running a new experiment), but it means that open data alone does not guarantee analytical reproducibility.

The Honest Current State

So, is the crisis over? It is still too early to conclude that. The fact that direct replication papers remain rare is the basis for saying it is still too early to declare the crisis over (PMID 39418470). Furthermore, science’s self-correction shows limits. In a 2008 study analyzing 98 papers, the citation count of original papers that failed to replicate dropped by only 5% to 9% compared to those that succeeded, and this difference was statistically indistinguishable from 0. Among papers citing the original study, fewer than 3% cited the replication attempt (PMID 35713980).

We will honestly add two more things.

First, this is not just a psychology problem. In a survey of ecology and evolutionary biology researchers (preprint), 64% of respondents had failed to report results because they were not significant, 42% had collected more data after seeing whether it was significant (a form of p-hacking), and 51% had reported unexpected results as if they had been hypotheses from the beginning. These are rates similar to psychology (DOI: 10.31219/osf.io/ajyqg). In criminology, a majority of respondents (DOI: 10.31235/osf.io/bwm7s), and in communication a non-negligible proportion (DOI: 10.31234/osf.io/7uyn5), answered that they used QRPs, and it has been reported that many researchers use them in education as well (DOI: 10.35542/osf.io/f7srb). All of these are self-report survey preprints. In an analysis comparing publication bias across multiple fields, economics was the most severe, with environmental science and psychology following right behind (PMID 38327122). There is also an analysis suggesting that discussions surrounding the crisis began almost simultaneously in psychology and biomedicine. Discussions in both fields shared the same core: the limits of peer review, the transparency of methods and data, and the distorted incentives of academia (DOI: 10.1177/10892680211046508).

Second, the crisis is a signal of flaws and at the same time a signal of self-correction. One perspective paper calls this the "credibility revolution," stating that the replication crisis ultimately led to positive structural, procedural, and community changes (PMID 39242883). Counterarguments are also seriously alive. One selective review (preprint) reports that p-hacking, HARKing, and publication bias may not be the main culprits of the crisis as commonly thought (DOI: 10.31222/osf.io/6xzaw_v5). One simulation study argues that the biggest variable determining the replication rate is not QRPs, but the base rate of how rare truly true effects are to begin with (DOI: 10.7554/elife.58237). In fields where true effects are rare, the replication rate drops for purely statistical reasons. Others argue that the replication crisis should be viewed not just as a problem of methods, but as an epistemological problem where the observer becomes part of what is observed (PMID 39775616).

The uncovering of non-replicable claims is a part of the system working, but the pace at which things are actually fixed after being uncovered is slow. In ego depletion, pre-registered large-scale multi-site tests estimated the effect to be almost 0, and in power posing and money priming, re-examinations failed to replicate part or most of the original results. What psychology lost was confidence in a few famous claims, and what it gained were tools to check itself. Whether those tools work sufficiently remains open.

This article is for providing scientific information, and the cited evidence includes some preprints prior to peer review. Detailed contents may be updated by follow-up research.

Source Papers

The Outbreak of the Crisis · Large-Scale Replication Projects

• Estimation of psychological reproducibility (36% replicated out of 100 papers, 97% originally significant) (PMID 26315443)
• Critical comment on the above results ("Reproducibility is actually high") (DOI: 10.1126/science.aad7243)
• Proportion of direct replication papers in top journals (0.2%, 169/84,834) (PMID 39418470)
• Review of social psychology multi-site replication project (75% failed) (PMID 36442681)

Collapsed Representative Effects — Ego Depletion

• Registered replication report of ego depletion (d=0.04, including 0) (PMID 27474142)
• Verification across 36 laboratories of ego depletion (non-significant, supporting Bayes null) (PMID 34520296)
• Unverified prediction of "depletion over time" (PMID 40452478)
• 12-laboratory multi-site replication (small but significant d=0.10) (PMID 34113424)
• Reinterpretation of ego depletion as temporary cognitive fatigue (DOI: 10.31234/osf.io/sy2mw)
• Ego depletion effect has been decreasing already since 10 years ago (DOI: 10.31234/osf.io/esjhp)
• Self-reporting of QRPs in ego depletion research (37.7% admitted) (PMID 29940020)

Collapsed Representative Effects — Power Posing · Social Priming

• Original power posing study (2010) (DOI: 10.1177/0956797610383437)
• Partial replication failure of power posing (testosterone and confidence failed, cortisol and risk-taking supported) (DOI: 10.21203/rs.3.rs-3970076/v1)
• Money priming replication failure (d=0.03, 1 out of 36) (PMID 26214168)
• Low reproducibility of research premised on an 'empty self' (PMID 41493875)

Structural Causes

• Statistical power seen through meta-analysis (median 36%, adequate 8%) (PMID 30321017)
• Trends in psychology statistical power and publication bias (1975-2017) (PMID 37847689)
• Publication bias (absence of null results publication) (PMID 40991668)
• Multiverse and researcher degrees of freedom (~30% significant results) (PMID 37166859)
• Arbitrary dichotomization triggers false positives (PMID 41233729)
• Catalog of p-hacking strategies and simulations (PMID 36778954)
• QRP incentive truth-telling survey (possibility of dominant norms) (PMID 22508865)
• Multinational QRP survey (9 out of 10 admit) (PMID 39133711)
• QRPs violate the APA Ethical Principles (PMID 39964511)

Reforms

• Definition, pros and cons of preregistration, and dealing with QRPs (DOI: 10.31234/osf.io/ud67s)
• Constraint of degrees of freedom through structured preregistration (PMID 33296358)
• Preregistration vs non-preregistration comparison (not a panacea) (PMID 37950113)
• Registered report null publication rate (54% vs 21%) (DOI: 10.2139/ssrn.6436338)
• 86% replicable when rigor is strengthened (PMID 37945809) — retracted later (Nat Hum Behav 2024;8:2067)
• Improving statistical reporting (TSRP checklist) (PMID 41238797)
• Open data badge ≠ reproducibility (DOI: 10.31222/osf.io/h35wt)

Honest Current Address · Self-Correction · Cross-Field

• Weak self-correction (5-9% decrease in citations) (PMID 35713980)
• 'Credibility revolution' triggered by the crisis (PMID 39242883)
• Impact of QRPs may have been exaggerated (DOI: 10.31222/osf.io/6xzaw_v5)
• The real variable of replication rate is the base rate (DOI: 10.7554/elife.58237)
• The crisis is an epistemological problem (second-order cybernetics) (PMID 39775616)
• QRPs in ecology and evolutionary biology (64/42/51%) (DOI: 10.31219/osf.io/ajyqg)
• QRPs in criminology (DOI: 10.31235/osf.io/bwm7s)
• QRPs in communication studies (DOI: 10.31234/osf.io/7uyn5)
• QRPs in education (DOI: 10.35542/osf.io/f7srb)
• Cross-field publication bias (economics worst) (PMID 38327122)
• Simultaneous occurrence in psychology and biomedicine (DOI: 10.1177/10892680211046508)