The Shrinking Effect — Why Textbook Findings Get Smaller When You Measure Them Again

Why This Debate

In 2015, the widely known number from the Reproducibility Project (a large-scale attempt to re-run psychology experiments) was "only 36 out of 100 trials were significant." Statistically significant means passing the threshold where a difference this large is unlikely to be driven by chance alone ("What Does "Statistically Significant" Actually Say? — The p-Value, Explained with a Coin Toss"). That same paper contains one more sentence about magnitude: the effect size in the replication was half of the original effect size (PMID 26315443). Effect size is a value measuring how large a difference is. Significance is a matter of whether a threshold was crossed, and the half is a matter of magnitude.

The first piece, "Psychology's Replication Crisis — Why the Findings We Believed In Wobbled", covered that event, and the second piece, "Researcher Degrees of Freedom and p-Hacking — How You Get the Result You Wanted Without Faking Anything", dealt with the problem of analytical discretion. The question of this piece is this: if you measure again and the effect does appear, but it is much smaller, what does that mean?

The cases this piece follows are three. Ego depletion is the theory that self-control runs dry the more you use it, growth mindset is the claim that making people believe abilities grow through effort raises grades, and the Implicit Association Test (IAT) is a test measuring associations you hold without realizing it. In re-tests, ego depletion and mindset saw their effects shrink, and debates continued over the predictive power of the IAT. The easiest interpretation is "the original authors were wrong." This article does not use that interpretation. A machinery that operates even without dishonesty plays a large part in the shrinkage, and that machinery operates identically for honest researchers.

Common Sense, and Common Sense on the Other Side

News that a replication failed comes with two face-to-face pieces of common sense. One is the common sense that a failed replication means the original discovery was false. The other is the common sense that it simply happened because the conditions were different, so all we need to do is find the hidden moderating variable, meaning the condition that changes the size of the effect. Both are convenient, and both erase a third answer: the possibility that the originally reported effect size was never the true value to begin with—that is, the actual size of that effect in reality.

The Machinery — Statistical Power, the Winner's Curse, the Threshold

Four gears create the shrinkage.

The first gear is low statistical power. Statistical power is the probability of catching a true effect as a significant result when it actually exists. In a survey of 12,065 effect sizes across 200 psychology meta-analysis domains, the median power per domain was roughly 36%, and only 8% of studies exceeded the conventional standard of 80% sufficiency (PMID 30321017). One classic estimate in neuroscience was 21% (PMID 23571845; figures cited in PMID 28706080). However, a re-analysis of that estimate suggests that the variance across subfields is too large for a single summary value to suffice, and that low power is not a universal problem (PMID 28706080). Medicine is similar. The median achieved power, reconstructed from 23,551 pairs of effect estimates and standard errors gathered from the Cochrane Database, was 13% (PMID 34425632).

The second gear: low power produces not just failure, but exaggeration. When power is low, missing a true effect happens frequently. This much is obvious. What is less obvious is that in the process, the estimate of a study that gets lucky and turns out significant tends to be larger than the true value. Because small samples have large swings of chance, crossing the threshold requires chance to provide a major push in the same direction. In a world where you only stand out if you cross the threshold, the values that stand out first tend to be large. This phenomenon where results crossing the threshold look inflated compared to the true value is called the Winner's Curse. Genetics literature explains that first-reported effects are often exaggerated compared to follow-up studies, and that this exaggeration can lead follow-up studies to be designed with low power and thus fail (PMID 19140131, PMID 18703928).

Among 332 quantitative trait genome-wide association studies, in 39 cases and 707 loci that accurately reported the sample size of replication attempts per locus, the predicted number of replications adjusted for the Winner's Curse (458) matched actual replications (457). Other studies replicated less than expected even after adjustment, and studies with different ancestral populations replicated even less (PMID 28715421). The paper analyzing this Cochrane data calls this same exaggeration the exaggeration ratio. An estimate that barely achieves significance at the 5% level is expected to overestimate the true value by a factor of 1.7 (PMID 34425632).

The third gear: the threshold is not just about statistical significance. If you start a regression model with four predictor variables and their two-way interactions (terms where two variables act together) and delete nonsignificant terms, even if all null hypotheses are true—meaning there is truly no effect at all—the probability that at least one is significant is 40% (PMID 21297852). It has also been pointed out that researchers pick not only brain voxels (measurement points) in neuroimaging, but also stimuli, task settings, and variables measuring outcomes in the direction that shows the effect best (PMID 26162135). Journals are thresholds too. In the health economics literature, journals with higher impact factors published larger estimates (PMID 23182564).

The fourth gear: publication bias is layered on top. Publication bias, the skew where significant results are published more easily, was covered in Same Trials, Different Conclusions — Publication Bias and What Went Into the Pool. Effect sizes are often written as $d$. Because $d$ is the difference between two group means divided by the spread of the data, 0 means no difference and larger values mean bigger differences. In a survey of about 68,000 meta-analyses—meaning analyses combining multiple studies into one (A Meta-Analysis Does Not Take an Average — How the Diamond in a Forest Plot Is Built)—mostly in medicine (and 605 in psychology), adjusting for selection bias dropped the median effect size in psychology from $d$ 0.37 to 0.26, in economics from 0.20 to 0.07, and in medicine from 0.24 to 0.13 (PMID 38327122). Simulations suggest that if accumulating significance is the rules of the game, rolling several small samples is a more efficient strategy than having one large sample (PMID 26168111). In simulations mimicking clinical trials ended early—trials stopped midway because a benefit appeared—the true value of the proportion by which treatment reduced risk (relative risk reduction) for a typical early-stopped trial was two-thirds of the observed value, whereas for large trials (SPRINT-style), the overestimation was much smaller (PMID 36064448). These are results of the same family.

None of these gears require dishonesty.

Report Card ① Ego Depletion — Near Zero in Large-Scale Replications

A 2010 meta-analysis pooling 83 papers reported that the ego-depletion effect—the phenomenon where using self-control causes self-control to drop in an immediately following task—was significant, and the authors themselves wrote that this result gave "preliminary support" to the strength model (the model that self-control is a resource that shrinks when used) (PMID 20565167). The problem came next.

In replications using preregistration—where hypotheses and analysis plans are made public before the study—and repeating the same procedure across multiple laboratories, the effect size dropped close to 0. In a replication across 23 laboratories (N=2,141), d = 0.04, and the 95% confidence interval [−0.07, 0.15] included 0 (PMID 27474142). The fact that the confidence interval ("A Confidence Interval Is Not a Margin of Error — How to Read the Two Numbers in the Parentheses") includes 0 means that a true effect of zero is also consistent with these data. A replication across 12 laboratories using a different combination of tasks (N=1,775) showed a small but significant d = 0.10, and it was 0.16 in an analysis that removed participants who might have responded randomly (PMID 34113424). In a 36-laboratory multisite verification (N=3,531), the pre-specified confirmatory analysis had d = 0.06 and was not significant. Bayesian analysis—an analysis comparing how many times more plausible the data are under one hypothesis versus the other—judged that the data were 4 times more plausible under the "no effect" hypothesis than the "effect exists" hypothesis. However, an exploratory analysis of the full sample without exclusion criteria found a significant d = 0.08, at which point the Bayesian analysis saw the data as roughly equally plausible under both hypotheses (PMID 34520296). The effect sizes of the three large-scale verifications fell between 0 and a small positive number. The d = 0.10 in the 12-lab replication (0.16 after exclusions) and the significant d = 0.08 in the 36-lab exploratory analysis are small positive numbers, not 0. "Near zero" in the title is a shortened way of describing this range.

Sub-hypotheses were even weaker. The blood-glucose model—the idea that willpower uses glucose as fuel—yields three corollaries: using self-control reduces blood glucose, the remaining blood glucose moves in tandem with subsequent performance, and consuming glucose restores self-control. None of the three were supported in the meta-analysis. The effect of rinsing the mouth with glucose without swallowing it was significant, but signs of publication bias were clear (PMID 27492453). A p-curve analysis (a method examining the quality of evidence through the distribution of p-values from significant results) judged that the evidential value was weak (PMID 27485134), and two experiments of 70 and 115 people also found that sugary drinks did not offset depletion (PMID 24389240). The hypothesis that willpower beliefs moderate depletion (original sample of 60 people) also failed to survive a replication with 187 people (PMID 37384732).

It is only fair to present counterarguments as well. There is a re-meta-analysis that recalculated the effect, pointing out that the bias-correction technique (PET-PEESE) used in the earlier meta-analysis had been applied inappropriately. Its abstract reports differences in which depletion tasks are effective and does not present an overall effect size (PMID 28391367). However, a study gathering three protocols and 840 people strongly supported the null hypothesis in a Bayesian analysis (PMID 29314092). In an anonymous survey of 277 respondents out of 1,721 invited, 37.7% answered that they had engaged in questionable research practices themselves (PMID 29940020).

In adjacent literature, directions diverged. A meta-analysis of 73 papers and 2,581 people reporting that cognitive exhaustion reduces physical performance found g = −0.38 (a standardized difference of the same type as d), and the authors consider that this effect cannot be explained by chance and that previous meta-analyses may have underestimated the effect (PMID 31873926). An analysis looking separately at mental fatigue and athletic performance found that after correcting for publication bias, the effect became negligible under most methods, though −0.36 remained in the most optimistic scenario, and a Bayesian analysis could not determine whether an effect existed or not (PMID 37682411).

Report Card ② Growth Mindset — Small and Conditional

Growth mindset interventions are programs that aim to raise students' grades by teaching them the belief that ability grows through effort. A randomized experiment using a nationally representative sample in the United States reported that an online intervention taking less than an hour raised low-achieving students' grades and increased their enrollment in advanced math courses (PMID 31391586). Low-achieving students' grades were reported to have risen by 0.10 grade points on the GPA scale (follow-up commentary, PMID 31494041), and the grade change occurred in schools where peer norms aligned with the intervention message (PMID 31391586).

A 2023 meta-analysis pooled 53 independent samples and narrowed the focus to subgroups with the highest expected effects under conditions of high implementation fidelity, yielding academic achievement of d = 0.14 (95% CI [.06, .22]) and mental health of d = 0.32. The prediction interval for academic achievement was −0.08 to 0.35 (PMID 36227318). A prediction interval is the range of effects that future intervention studies are likely to produce, and starting below 0 means that, depending on the study, the effect might fall short of 0.

An intervention delivered directly by teachers was also reported to raise the grades of students at risk of academic failure by 0.27 standard deviations (1,996 students in grades 6-7, 50 teachers, randomized trial). This design was structured to co-create a supportive classroom environment, and the effect was largest for students whose teachers previously endorsed a fixed mindset (PMID 35699476). Because the scales differ, this cannot be directly compared to the 0.10-point figure mentioned earlier. Re-analyzing national data from the online intervention—a different design—using teacher variables supported the hypothesis that for the student mindset effect to be sustained, the teacher's own growth mindset must back it up (PMID 34936529). Because online interventions do not change the classroom environment, their effects persist where teachers are already supportive, whereas teacher-delivered formats are designed to create the supportive environment itself; thus, the two findings do not contradict each other. In a follow-up study from the same lineage, using observed associations rather than intervention effects, mindset beliefs at the start of the first year of high school predicted college enrollment several years later, with some evidence that teacher support moderated this relationship, though predictive power weakened after the pandemic (PMID 39073263). A meta-analysis of health professions students (including non-randomized trials) included only two studies, and the pooled effect size of the intervention for raising mindset scores was a standardized mean difference (SMD) of 0.25, with a 95% CI of [−0.18, 0.68]. The authors view this as preliminary evidence of potential benefit, but because there are only two studies, an effect of zero cannot be ruled out (PMID 38600797).

A retrospective by Dweck and Yeager, the proponents of mindset theory, describes how mindset research has transitioned into an era of field experiments and replication science, concluding that there is still much to learn (PMID 30707853). Practices in general education research are also an issue. In meta-analyses published over the past five years in two major education journals, publication bias corrections were largely omitted (though this survey was not limited to mindset meta-analyses, PMID 34081730), and without correction, it is impossible to know how much effect size estimates were inflated. A correlational meta-analysis of 59 Chinese samples (54,302 participants) reported r = 0.36 / −0.25 between mindset and mental health indicators (PMID 39504841), but correlation answers a different question than intervention effectiveness.

Report Card ③ The Implicit Bias Test — Not 0, but "What Are We Measuring?"

The Implicit Bias Test (IAT) measures associations you hold without realizing it by how quickly you can pair categories (e.g., racial groups) with evaluative words. The issue is whether these scores predict actual behavior. This predictive power is written as the correlation coefficient $r$, which is the degree to which two values move together, and the behavior or judgment you are trying to predict is called the criterion.

A 2009 meta-analysis of 122 reports and 14,900 people put the average predictive correlation of the IAT at $r = .274$, and reported that in 32 samples using interracial interaction behavior as the criterion, the IAT predicted better than self-reporting (PMID 19586237). In the same year, another team re-analyzed data from two frequently cited studies and wrote that even when putting IAT scores into a regression model, prediction errors shrank only very slightly and individual-level behavior was not predicted (PMID 19449998).

In a 2013 meta-analysis, the IAT was not a good predictor for any discriminatory criterion category once brain activity was removed, nor was it any better than simple self-reporting (PMID 23773046). A 2016 re-analysis cast doubt on the criterion side. Many of the outcome variables called "discrimination" in that literature were actually brain activity or voting preferences, and when only real discrimination was left, the overall effect was close to 0 and varied widely across studies (PMID 27109866). While strongly opposing practical applications that rely on the assumption that the IAT predicts discrimination, those authors wrote that it is useful as a tool for researchers, educators, and others interested in attitudes, prejudices, and stereotypes.

What comes next is interesting. The largest meta-analysis in 2019 pooled 217 reports and 36,071 people (previous meta-analyses had 3,471 and 5,433 people). In structural equation modeling, implicit measurement showed a unique contribution of $\beta = .14$, meaning the share left over even after accounting for self-reporting. Heterogeneity was very large. Heterogeneity means that the effect varies greatly from study to study ("I² Is Not the Size of the Scatter — How to Read Heterogeneity Statistics"), and the average correlation was $r = .37$ in 13 studies where all five methodological features—the research objective lies in correlation, using a standard IAT, high attribute polarity, measuring behavior relatively, and a good correspondence between the test and behavior—matched up, whereas it was $r = .02$ in 6 studies where all were low (this is out of 271 parts and is not a quality judgment). The authors also point out that most studies were severely underpowered (PMID 30550298). Because correlations split this way depending on methodological conditions, the question shifts from "Does the effect exist?" to "Under what conditions and what is being measured?" A review looking back over 20 years concludes that predictive power is weak and the explanatory power added on top of self-reporting (incremental validity) is negligible, while also offering reasons and directions for improvement and adding that the IAT has great potential (PMID 31787912).

There are also rebuttals written by Greenwald and others who developed the IAT. The difference between the 2009 meta-analysis ($r = .236$) and the 2013 meta-analysis ($r = .148$) looking at race-attitude and stereotype IATs was mostly due to differences in inclusion policies that determine which effect sizes to include, and whichever figures you use, they are large enough to account for socially meaningful discriminatory impacts (PMID 25402677). An IAT tailored to the phenotype of hair texture predicted additional support for legal judgments above and beyond race-attitude IATs and self-reporting (PMID 34487286). At the opposite extreme is the claim that structural equation models of multi-method data found no evidence that the IAT measures an implicit construct (PMID 33709849).

Measurement reliability—the degree to which the same person gets similar scores when taking the same test again—is also an ongoing problem. The test-retest reliability in a study of 519 children aged 6 to 11 was .34 to .48 (PMID 29251966). A meta-analysis of 29 studies on medical staff and medical students found pooled IAT scores (group average scores, not predictive power) of 0.28 for doctors and 0.35 for medical students (95% CI 0.03–0.67), and the authors conclude that doctors and medical students held anti-Black implicit attitudes, and that the magnitude was small (PMID 34495481).

Report Card 4 — What Vanished, What Survived Under Conditions

• The professor priming (Dijksterhuis and van Knippenberg, 1998): Vanished. Priming is a technique where making a specific concept pop into mind beforehand changes the behavior that follows. The result that participants primed with the category of "professor" did 13% better on a trivia quiz became a 0.14% difference between conditions in a registered replication (N=4,493) across 23 of 40 laboratories that met inclusion criteria (PMID 29463182).
• Verbal overshadowing: Survived as conditional. The result that describing a perpetrator's face in words drops lineup identification accuracy by 25% was 4% when the description happened right after the event and 16% when delayed by 20 minutes in a preregistered replication (PMID 26186758). The authors conclude it is a robust effect heavily swayed by the timing of the description.
• Facial feedback: Partial survival. This is the hypothesis that facial expressions alter emotions. In a multi-lab of 3,878 people across 19 countries, the tasks of mimicking facial expressions and making voluntary expressions amplified or produced feelings of happiness, but the evidence was less decisive in the classic task where people hold a pen in their teeth (PMID 36266452).
• Childhood poverty × mortality salience (clues that bring death to mind): Partially replicated (risk-taking only), with the size greatly shrunken. In a sample 14.2 times that of the original study (1,010 people versus 71), the risk-taking effect was replicated, but the effect size index $\eta^2$ dropped from 0.17 to 0.004, and the time preference effect was not replicated (PMID 39666532). The authors view the heavily shrunken effect size as making its practical meaning dubious.
• And others. Vicarious dissonance (the phenomenon of feeling uncomfortable upon seeing inconsistent behavior by an in-group member) went from g = 0.41 to 0.22 after correction for publication bias (PMID 39126282), and a multi-lab re-measurement of six classic phenomena from event-coding theory (the theory that action and perception share the same code) reported that the core of the original results consistently came out again, but the effect sizes were considerably smaller than originally reported (PMID 35978172).

Shrinkage Is Not Absence

The fact that it got smaller is neither proof that there is no effect nor proof that someone cheated. A shrinking effect is not entirely due to bias. In the earlier psychology meta-analysis survey (PMID 30321017), the median between-study heterogeneity was $I^2$ 74%, and residual reporting bias was small. $I^2$ is the share of the observed effect size variation that comes from genuinely different effects across studies. The magnitude of bias is assessed differently across surveys. The earlier publication bias survey (PMID 38327122) reported that after correction, effects shrank substantially in 605 psychology cases out of about 68,000 mostly medical cases, and this survey sees residual bias as small across 200 psychology meta-analyses. The subjects and methods differ. The authors of this survey view low statistical power and high heterogeneity as fully explaining recent replication difficulties. This interpretation suggests that a substantial portion of replication failures may stem from variation itself rather than dishonesty.

Heterogeneity also has types. They are population (who is measured), design (how it is measured), and analysis (how it is calculated). An analysis pooling 70 many-lab replications and 5 many-analyst studies (studies where multiple teams each analyze the same data) concludes that population heterogeneity is relatively small while design heterogeneity and analysis heterogeneity are large. According to the authors' framework, when accounting for heterogeneity, the probability that a hypothesis is true in the average population, design, and analysis can be much lower than significant individual studies suggest. The authors also add the caveat that the number of studies is small and the uncertainty of estimation is large, so the findings should be read with caution (PMID 39078672). There are also cases where the same data were analyzed differently by multiple teams. When pupil data from 30 nine-month-old infants were distributed to 7 teams, 2 teams found no significant results while the remaining 5 teams reported different patterns (PMID 37944367). Merely simulating disruptions to protocol fidelity—that is, the degree to which procedures were followed as originally intended—produced statistical patterns resembling those observed in many-lab replications (PMID 35049324). Standard statistical power formulas leave cross-study variation out of the calculation entirely, making power estimates overly optimistic, and replication studies whose sample sizes are determined using these formulas fail more often than expected (PMID 26186112). When interactions or measurement errors get involved, the calculations miss the mark by an even wider margin (PMID 32208875).

And shrinkage is not a universal law. Across 466 ecology meta-analyses, true directional changes over time were only about 5%, and most apparent declines were explained by regression to the mean (PMID 35302660). Regression to the mean is the tendency for an unusually large value on a first measurement to drift back toward the average on the next measurement. The correlation that "smaller samples have larger effects," which has been read as a fingerprint of publication bias, was not prominent across 150 meta-analyses and 57 many-lab replications, and was much larger only for the core effects of papers. The authors write that it is unclear whether this should be blamed on publication bias (PMID 38359021).

Honest Unresolved Points

We do not know the true value. Each correction method gives a different answer. Shrinkage estimators, which are methods that pull estimates toward the average to reduce exaggeration, showed better performance than unadjusted conventional estimators (PMID 34425632, PMID 38111969). In hierarchical Bayesian recalibration, which adjusts individual values by borrowing information from similar studies, the adjustment magnitude reached up to 0.97 across 21 neuroimaging case-control studies and was larger when the samples were smaller (PMID 38179200). Furthermore, the statistical power of the tests used to judge whether a study replicates is often low (PMID 30070547).

There is no rule separating shrinkage from absence. One counterargument is that the scientific method cannot prove a negative and that statistics alone cannot declare the non-existence of a psychological phenomenon (PMID 28626435). While this is a valid argument, it also risks sliding into an attitude that accepts no refutation whatsoever. We note both sides.

It is also unclear whether things are improving. Across 57,909 papers in 12 journals, the median sample size increased from 105 to 190 people. Although the reporting of effect sizes and confidence intervals increased, the proportion of significant p-values per paper remained unchanged at 69%, and the use of multiple testing corrections actually decreased (PMID 37163505).

Certain designs produce large effects to begin with. In an analysis of 63 studies and 52,979 people, the between-person effect for most brain regions—that is, the effect comparing differences among people—was |d| < 0.2. Even the strongest local effects required $n > 500$ to achieve 80% power. On the other hand, multivariate analyses and within-person task designs, which are designs looking at differences between conditions within the same person, captured effects even at $n < 50$ (PMID 42281980, PMID 35296861). Because effect sizes vary greatly depending on the design, it is difficult to expect studies with different designs to have magnitudes on the same scale.

Significance — How to Read "It Got Smaller"

To boil it down to a single line: Shrinkage is a forecast. It is not something to be read as a scandal.

That is why what we are left with is not a verdict, but five questions.

• How many people were in the first-reported sample?
• Was that discovery a pre-specified hypothesis, or one that crossed the threshold among multiple candidates?
• Was the replication pre-registered and multi-lab?
• Did the shrinkage go to zero, or to a condition? Professor priming is in the front, and language masking is in the back.
• Did the authors report corrected estimates alongside them?

The fates of paradigms tested by replication studies differed even across the three cases held in a single paper. One points to a true discovery across the whole study, one saw its effect shrink drastically, and one remains in dispute (PMID 33682522). This publication takes the stance that this distinction is necessary. There is also commentary arguing that since large-scale multi-lab replication projects have achieved their purpose, it is now time to move on to other kinds of replication (PMID 31064594).

The final cog is people. One paper by a single author reports that when Kahneman and Tversky’s classic experiment was run again with scientists instead of undergraduates, it yielded essentially the same results. This means that the tendency to try to craft a consistent story regardless of data quality and to look for patterns that are not there exists in scientists too, and the author adds that this can be reduced through training (PMID 30197928, single study). The news that "it got smaller" is not a betrayal. In psychology, it is close to the result one commonly encounters when re-measuring a first report that crossed the threshold. Not every field and every discovery is like that. We are only just now learning how to factor that point into our calculations.

Evidence Papers

• "Estimating the reproducibility of psychological science" (2015) — PMID 26315443 — 100 studies replicated. Replicated effect sizes were half of the original effect sizes, and only 36% were significant.
• "What meta-analyses reveal about the replicability of psychological research" (2018) — PMID 30321017 — 200 meta-analyses and 12,065 effect sizes. Median statistical power 36%, median heterogeneity $I^2$ 74%, and residual reporting bias was small.
• "Power failure: why small sample size undermines the reliability of neuroscience" (2013) — PMID 23571845 — A classic summary showing how low statistical power leads to overestimation of effect sizes and low replicability.
• "Power-up: A Reanalysis of 'Power Failure' in Neuroscience Using Mixture Modeling" (2017) — PMID 28706080 — A median statistical power of 21% is an overgeneralization as a single summary value. Subfield variations are large.
• "The statistical properties of RCTs and a proposal for shrinkage" (2021) — PMID 34425632 — Cochrane 23,551 pairs. Median achieved statistical power 13%, exaggeration ratio of barely significant estimates 1.7 times.
• "Quantifying and correcting for the winner's curse in genetic association studies" (2009) — PMID 19140131 — Formalization of the winner's curse. The exaggeration decreases as statistical power increases.
• "Curses--winner's and otherwise--in genetic epidemiology" (2008) — PMID 18703928 — The phenomenon where initial reported effects are exaggerated compared to subsequent studies, and overfitting.
• "Statistical correction of the Winner's Curse explains replication variability" (2017) — PMID 28715421 — In 39 cases and 707 loci accurately reporting locus-specific samples, predicted 458 vs. actual 457.
• "Cryptic multiple hypotheses testing in linear models" (2011) — PMID 21297852 — Model simplification starting from 4 predictors + interactions yields significant results with a 40% probability even if the null hypotheses are all true.
• "Voodoo Correlations Are Everywhere-Not Only in Neuroscience" (2011) — PMID 26162135 — Not just voxels, but stimuli, tasks, boundary conditions, and variable selection overall inflate effects.
• "Publication selection in health policy research: the winner's curse hypothesis" (2013) — PMID 23182564 — High-impact journals publish larger elasticity estimates.
• "Footprint of publication selection bias on meta-analyses" (2024) — PMID 38327122 — About 68,000 meta-analyses. Median $d$ after correction: psychology 0.37→0.26, economics 0.20→0.07, medicine 0.24→0.13.
• "The Rules of the Game Called Psychological Science" (2012) — PMID 26168111 — A simulation showing that if the goal is significance hunting, multiple small samples are more efficient.
• "Overestimation of benefit when clinical trials stop early" (2022) — PMID 36064448 — The true relative risk reduction of early-stopped trials is about two-thirds of the observed value.
• "Ego depletion and the strength model of self-control: a meta-analysis" (2010) — PMID 20565167 — 83 studies meta-analysis. Significant effects and "preliminary support".
• "A Multilab Preregistered Replication of the Ego-Depletion Effect" (2016) — PMID 27474142 — 23 labs $N=2,141$, $d=0.04$ [−0.07, 0.15].
• "A Multilab Replication of the Ego Depletion Effect" (2021) — PMID 34113424 — 12 labs $N=1,775$, $d=0.10$, 0.16 after exclusion.
• "A Multisite Preregistered Paradigmatic Test of the Ego-Depletion Effect" (2021) — PMID 34520296 — 36 labs $N=3,531$. Confirmatory $d=0.06$ nonsignificant, Bayes 4 times null dominance.
• "Testing the role of glucose in self-control: A meta-analysis" (2016) — PMID 27492453 — All three corollary propositions of the blood glucose model unsupported.
• "The Bitter Truth About Sugar and Willpower" (2016) — PMID 27485134 — p-curve analysis. Evidentiary value weak.
• "Sweet delusion. Glucose drinks fail to counteract ego depletion" (2014) — PMID 24389240 — Both direct replications failed.
• "Does willpower mindset really moderate the ego-depletion effect?" (2023) — PMID 37384732 — Original sample 60 people vs. replication 187 people. Both main effect and moderation effect replication failed.
• "An updated meta-analysis of the ego depletion effect" (2018) — PMID 28391367 — A re-meta-analysis pointing out the inappropriateness of applying bias correction techniques.
• "Bayesian analysis of multimethod ego-depletion studies favours the null hypothesis" (2018) — PMID 29314092 — Three protocols, 840 people. Strong support for the null hypothesis.
• "Self-reports from behind the scenes" (2018) — PMID 29940020 — 37.7% of 277 respondents admitted using questionable research practices.
• "Effects of Prior Cognitive Exertion on Physical Performance" (2020) — PMID 31873926 — 73 studies, 2,581 people, $g=−0.38$.
• "Assessing the Evidential Value of Mental Fatigue and Exercise Research" (2023) — PMID 37682411 — Mostly negligible levels after correction, even in optimistic scenarios −0.36.
• "A systematic review and meta-analysis of growth mindset interventions" (2023) — PMID 36227318 — 53 samples. Conditional academic $d=0.14$, mental health $d=0.32$, academic prediction interval −0.08 to 0.35.
• "A national experiment reveals where a growth mindset improves achievement" (2019) — PMID 31391586 — Nationally representative sample. Low-achieving student grade improvement, peer norm dependent.
• "When Do Growth Mindset Interventions Work?" (2019) — PMID 31494041 — The improvement magnitude is 0.10 GPA points.
• "Growth-Mindset Intervention Delivered by Teachers Boosts Achievement" (2022) — PMID 35699476 — Teacher-delivered type, 0.27 standard deviations.
• "Teacher Mindsets Help Explain Where a Growth-Mindset Intervention Does and Doesn't Work" (2022) — PMID 34936529 — Teacher mindsets moderate student effects.
• "Mindsets, contexts, and college enrollment" (2024) — PMID 39073263 — 9th-grade mindsets predict college enrollment 4 years later, weakened post-pandemic.
• "Mindsets: A View From Two Eras" (2019) — PMID 30707853 — Description from the original discoverers. Acknowledges transition to field experiments and replication science eras.
• "Grit, resilience and growth-mindset interventions in health professional students" (2024) — PMID 38600797 — Growth mindset intervention SMD 0.25 [−0.18, 0.68].
• "Neglect of publication bias compromises meta-analyses of educational research" (2021) — PMID 34081730 — Omission of publication bias correction in educational meta-analyses.
• "A Metaanalysis of the relationship between growth mindset and mental health in Chinese samples" (2024) — PMID 39504841 — 59 studies, 54,302 people. $r=0.36$ / −0.25.
• "Understanding and using the Implicit Association Test: III" (2009) — PMID 19586237 — 122 reports, 14,900 people, average $r=.274$.
• "Strong claims and weak evidence" (2009) — PMID 19449998 — Reanalysis of two frequently cited studies. Failure to predict individual-level behavior.
• "Predicting ethnic and racial discrimination: a meta-analysis of IAT criterion studies" (2013) — PMID 23773046 — Poor across all criterion categories except brain activity, no better than self-report.
• "A closer look at the discrimination outcomes in the IAT literature" (2016) — PMID 27109866 — After refining criteria, effects are close to zero and highly inconsistent.
• "Relationship between the Implicit Association Test and intergroup behavior" (2019) — PMID 30550298 — 217 reports, 36,071 people. $\beta=.14$, methodological conditions top $r=.37$ vs. bottom $r=.02$.
• "Predicting Behavior With Implicit Measures" (2019) — PMID 31787912 — 20-year overview. Weak predictive power, negligible incremental validity.
• "Statistically small effects of the IAT can have societally large effects" (2015) — PMID 25402677 — The difference between the two metas is inclusion policy. Either way, sufficient for explaining societal impact.
• "Invalid Claims About the Validity of Implicit Association Tests" (2021) — PMID 33709849 — No evidence of measuring implicit constructs in multimethod structural equation models.
• "Specificity and incremental predictive validity of implicit attitudes" (2021) — PMID 34487286 — Incremental prediction of phenotype-tailored IATs.
• "Test-retest reliability and predictive validity of the IAT in children" (2018) — PMID 29251966 — Test-retest .34 to .48, predictive validity .10 to .46.
• "Implicit racial bias among medical graduates and students" (2022) — PMID 34495481 — Combined 0.28 and 0.35, sizes are small.
• "Registered Replication Report: Dijksterhuis and van Knippenberg (1998)" (2018) — PMID 29463182 — Original 13% → 0.14% in 23 labs $N=4,493$.
• "Registered Replication Report: Schooler and Engstler-Schooler (1990)" (2014) — PMID 26186758 — Original 25% → 4% immediately, 16% at 20-minute delay.
• "A multi-lab test of the facial feedback hypothesis" (2022) — PMID 36266452 — 3,878 people in 19 countries. Partially survived depending on task type.
• "Childhood poverty and its impact on financial decision making under threat" (2025) — PMID 39666532 — 14.2 times sample. $\eta^2$ 0.17→0.004, time preference replication failed.
• "Vicarious Dissonance: Pre-Registered Meta-Analysis" (2026) — PMID 39126282 — $g=0.41$ → 0.22 after selection model correction.
• "Perception and action as viewed from the Theory of Event Coding" (2023) — PMID 35978172 — 6 classic phenomena multi-lab. Replicated but effect sizes considerably smaller.
• "Heterogeneity in effect size estimates" (2024) — PMID 39078672 — Population heterogeneity is small, while design and analysis heterogeneity is large.
• "The pupil collaboration" (2023) — PMID 37944367 — Same data, 7 teams. Two teams had no significant results.
• "Is replication possible without fidelity?" (2023) — PMID 35049324 — Fidelity manipulation simulation reproduces multi-lab patterns.
• "You Cannot Step Into the Same River Twice" (2014) — PMID 26186112 — Power formulas ignoring cross-study variation are optimistic.
• "On Attenuated Interactions, Measurement Error, and Statistical Power" (2020) — PMID 32208875 — Six $2\times2$ design papers with $N>500$ can also have low statistical power.
• "Decline effects are rare in ecology" (2022) — PMID 35302660 — 466 meta-analyses. True declines are about 5%, mostly regression to the mean.
• "Publication bias in psychology: A closer look" (2024) — PMID 38359021 — Not prominent in meta-analyses, large only in the paper's core effect.
• "Evaluating a shrinkage estimator for the treatment effect in clinical trials" (2024) — PMID 38111969 — Shrinkage estimators outperform unbiased estimators.
• "Recalibrating single-study effect sizes using hierarchical Bayesian models" (2023) — PMID 38179200 — Adjustment magnitude 0 to 0.97, larger for smaller samples.
• "Statistical analyses for studying replication" (2019) — PMID 30070547 — Low statistical power of the replication decision test itself.
• "Reproducibility in Psychological Science: When Do Psychological Phenomena Exist?" (2017) — PMID 28626435 — Counterargument that science fails to prove the null.
• "Changes in methodological study characteristics in psychology between 2010-2021" (2023) — PMID 37163505 — Median sample 105→190, significant $p$ proportion 69% unchanged, multiple testing correction decreased.
• "Effect sizes in human functional neuroimaging" (2026) — PMID 42281980 — Inter-individual effects mostly $|d|<0.2$, multivariate and within-subject designs $n<50$.
• "Reproducible brain-wide association studies require thousands of individuals" (2022) — PMID 35296861 — Stable brain-behavior associations require thousands of individuals.
• "What the replication reformation wrought" (2018) — PMID 31064594 — Multi-lab projects achieved their purpose, time to move on to other replications.
• "Saving Science Through Replication Studies" (2022) — PMID 33682522 — Three branches: true discoveries, greatly reduced effects, and debated ones.
• "Why Is It so Hard to Do Good Science?" (2018) — PMID 30197928 — Kahneman and Tversky experiments returned to scientists.