Researcher Degrees of Freedom and p-Hacking — How You Get the Result You Wanted Without Faking Anything

The Question This Concept Answers

When people talk about research misconduct, they usually picture scenes of fabricating data that does not exist, altering numbers, or copying someone else’s writing. Yet when surveys asking researchers whether they have ever engaged in certain practices are collected, categorized, and ranked, the order of the list is completely reversed. The most commonly reported items were choosing analyses, choosing citations, and choosing covariates (other variables included in an analysis because they can affect the outcome), while manipulation, falsification, and plagiarism sat at the rarest end. This study also pointed out that acts of omission—taking information away—are reported more frequently than acts of adding or changing it (PMID 41963575).

Acts of omission are the subject of this article. None of the top items on the list above (choosing analyses, citations, or covariates) are rule violations in themselves. Removing outliers (measurement values that fall far outside the rest of the data), controlling for covariates, or collecting a slightly larger sample can all be legitimate judgments, and some are recommended by textbooks. The problem arises when these judgments are made while glancing sideways at the results.

A 2011 paper gave this problem a name. This paper showed through both simulations and two experiments that even though psychologists nominally support keeping the false-positive rate (the rate at which a study with no real effect turns out significant) at 5% or below, flexibility in data collection, analysis, and reporting drives the actual false-positive rate up dramatically. In the paper's phrasing, researchers in many cases are more likely to falsely declare that an effect exists than to correctly declare that it does not (PMID 22006061). The source of this flexibility—the name given to the choices a researcher can legitimately make differently while collecting, analyzing, and reporting data—is researcher degrees of freedom. The strategy of tweaking these choices back and forth to turn a non-significant result into a significant one is collectively called p-hacking (PMID 36778954). Here, "significant" means a p-value smaller than .05, and the p-value is the probability that a difference this large or larger would occur by chance assuming there is actually no difference (What Is p = 0.03 the Probability Of? — The One Question a p-Value Answers). The background of how this issue gained attention is covered in Psychology's Replication Crisis — Why the Findings We Believed In Wobbled.

Exercising degrees of freedom is not rare. When an anonymous survey was conducted among some 2,000 psychologists with rewards for honest answers, the proportion who had engaged in questionable research practices (gray-zone practices that are not outright fabrication but fall short of responsible research) was strikingly high, and the authors concluded that some practices may be standard norms in the research community (PMID 22508865). In a survey of 6,813 researchers across all fields in the Netherlands, 51.3% answered that they frequently engaged in at least one problematic practice, while 4.3% reported falsification and 4.2% reported data alteration (PMID 35171921). Because fabrication and alteration were asked about using a separate method to protect respondents, it is not a strict comparison, but the gap is large. The center of gravity of the problem lies in discretion, not fraudsters.

Where Are the Degrees of Freedom?

Degrees of freedom are present at every stage of research, from forming the hypothesis to design, execution, analysis, and reporting. One review paper organized the degrees of freedom held by researchers in psychology research into 34 items (PMID 27933012). Let us open just a few of them.

Outlier handling is the first. In studies measuring reaction time (how quickly one responds upon seeing a stimulus), several methods for handling outliers are commonly accepted side by side as standards. Simulations showed that depending on what is chosen, the p-value is inflated and confidence intervals and effect sizes are distorted. Using Bayesian estimation and heavy-tailed distributions eliminates the need to handle outliers separately, but the paper adds that the price of doing so opens yet another flexibility (PMID 37691812).

Exclusion criteria are the rules that separate participants to be dropped from the analysis. In fear conditioning research (an experiment that pairs a certain signal with an unpleasant stimulus to learn a fear response), it is standard practice to drop "those who did not learn" and "those who did not respond," but there is no consensus on how to define these people. After confirming through a literature search that exclusion criteria vary wildly, the authors re-analyzed existing data by simply changing the definitions to show how the conclusions differed (PMID 31841112).

Covariates are other variables included together in the analysis as mentioned earlier, and what should be included is covered in "What "Adjusted for Age, Sex and BMI" Actually Did — How to Read Confounding and Adjustment". Across two years of structural neuroimaging (imaging that measures the sizes of brain regions) papers, the covariates entered into a single model ranged from 0 to 14, and 37 different combinations were used across 68 models. When verified with the same open dataset, some combinations barely changed the results, while other combinations drastically altered the pattern of the results (PMID 31568872).

Preprocessing is the step of refining measured raw data to make it suitable for analysis. When applying 14 processing sequences (pipelines) chosen to show the range of methods found in the literature to a single set of ERP (event-related potential) N400 experiment data, the high-pass filter cutoff frequency, artifact removal method, baseline length, reference electrode, measurement time points and locations, and amplitude measurement method all affected at least part of the result indicators. Only the low-pass filter had no clear effect on any indicator (PMID 38961523).

Degrees of freedom exist even in places researchers do not perceive as choices. In a study where the same statistical test was run in SPSS, SAS, Stata, and R and matched against hand calculations, the p-values for a chi-square test of a 2×2 table differed by anywhere from .005 to .162 depending on the software, and the differences varied mainly with sample size. The authors discuss that the conclusion can change depending on which software was used (PMID 35953660).

The final place is the reporting stage. The practice of writing a hypothesis created after seeing the results as if one had it from the beginning was already named HARKing (Hypothesizing After the Results are Known) back in 1998 (PMID 15647155).

Why Rational Choices Combine to Become a False Positive

The reason that even when individual choices are rational, combining them increases false positives lies in multiple testing, which means running tests on the same question multiple times. A significance level of .05 is a threshold set so that roughly once in every twenty times, a significant result will appear even if there is actually no effect. When there are multiple degrees of freedom and each can take several values, the number of possible analyses multiplies. If you report only the single best-looking result while keeping the significance level at .05, the actual false-positive rate is not .05. If the analyses are similar to one another, the rate does not increase strictly as calculated, but the direction is the same.

The more decisive point is that you do not even need to run all those analyses in reality (DOI: 10.1002/essoar.10500564.1). Simply looking at the data and deciding, "It is right to do it this way in this case," can be enough. Because you would have made a different decision if different data had arrived, the analysis you actually ran is the result of picking one among several analyses that were already possible.

This inflation can also be seen through simulation. A preprint (not peer-reviewed) addressing degrees of freedom in phonetics directly showed that false positives swell severely when exploring degrees of freedom through Type I error (which means the same thing as a false positive) simulations (DOI: 10.31234/osf.io/fp4jr). Though not a simulation, there is also an argument. In cyclostratigraphy, which searches for cycles in geological strata, a preprint (not peer-reviewed) argued that the fact itself that the success rate of statistical cycle detection is close to 100% is a sign of confirmation bias (the tendency to lean toward confirming the result you want). The logic is that each choice made after looking at the data means there was a road not taken (DOI: 10.1002/essoar.10500564.1). Because p-hacking strategies are not singular, a study that extracted and simulated 12 strategies from the literature found that the extent to which false-positive rates are increased differs by strategy, so looking at each strategy separately broadens the basis for countermeasures (PMID 36778954).

How to Measure From the Outside ① Giving the Same Data to Multiple Teams

The most direct way to see how degrees of freedom shake results is to give the same data and the same hypothesis to multiple teams and let them each analyze it. In a study that grouped 161 researchers into 73 teams to test whether "an increase in immigration reduces support for social policies" using the same data, the teams' numbers and conclusions diverged widely. The tricky part comes next. Neither expertise, prior beliefs, nor expectations could predict this difference, and even after coding every identifiable decision made by each team, over 95% of the variance in numerical results remained unexplained. The authors called this a "universe of hidden uncertainty" that is invisible when looking at just one study (PMID 36306328). This means the result of a single study may be the outcome of just one path chosen by that team.

Measuring From the Outside ②: The Multiverse and Specification Curves

Whether a paper's results stand on only a specific combination of choices can be known by running through all the combinations that can be considered justifiable. This collection of all possible analytical paths is called the multiverse, and the combination of choices that makes up a single analysis is called a specification. Specification curve analysis is a method that lines up the results of all specifications on a single graph, and it consists of three steps. It finds all specifications that are theoretically justified, statistically valid, and non-redundant, unfolds the results on a graph so readers can see which decisions drove the results, and performs joint inference (a statistical judgment made on the whole set at once) across all specifications (PMID 32719546). What this method shows is how well the results hold up across the entire set of specifications deemed justifiable. Validity outside the set of specifications (measurement and design) or whether the results are true are separate questions.

In a multiverse, results can even split in completely opposite directions depending on the choices. A study addressing the relationship between red meat and all-cause mortality analyzed unprocessed red meat using U.S. NHANES observational data (10,661 people). Seventy analytical methods were extracted from existing literature to run 1,208 specifications. If the hazard ratio (the rate at which events happen over the same period divided between two groups, "What Is a Hazard Ratio of 0.7 the 0.7 Of? — How to Read a Survival Curve") is 1 or greater, it points to red meat increasing mortality; if it is less than 1, it points to decreasing it. 435 (36.0%) were 1 or greater, 773 (64.0%) were less than 1, and 48 (3.97%) were statistically significant—among which 40 pointed to red meat lowering mortality and 8 to raising it (PMID 38354868). The median hazard ratio was 0.94 (interquartile range 0.83 to 1.05) (PMID 38354868). This means that justified analyses on the same data yielded results in different directions. However, the purpose of this study was not to judge the effect of red meat, but to show that this method can be used in nutritional epidemiology.

When 1,240 analyses were applied to literature measuring the relationship between anxiety and fear learning using skin conductance activity (electrical signals measured by sweat gland responses), a significant effect consistent with theory appeared in only 1.45% of them. The authors concluded that the difference in fear learning between high-anxiety and low-anxiety groups measured by skin conductance activity is vulnerable to researcher degrees of freedom (PMID 39142133). Meta-analysis (an analysis that gathers multiple studies and recalculates them, "A Meta-Analysis Does Not Take an Average — How the Diamond in a Forest Plot Is Built") is also subject to choices. When 256 meta-analyses were run with 530 effect sizes extracted from 185 studies on oxytocin administration by varying inclusion criteria, synthesis models, and publication bias correction methods, the summary effect sizes (the difference between two group averages divided by the spread of the data, $d$) ranged from $d = -0.16$ to 1.45. However, that same paper also reported that over 90% of them exceeded a bootstrap range (a range created by randomly resampling data) built assuming no effect, and the authors considered that the evidence generally supported the existence of an effect regardless of the choices (PMID 40848964).

In the same way, there is a study that unfolded data from registered replication reports (projects where multiple laboratories repeated the same experiment using pre-registered identical procedures). This study simulated what kind of bias would have occurred if there had been no pre-registration and results could have been selectively reported based on statistical significance. Although the multiverses of the 294 studies typically contained thousands of results, only about 30% of the studies yielded significant effects in the hypothesized direction. Selective reporting could alter the conclusions of meta-analyses and create bias, and because the effects of a single degree of freedom were inconsistent across different samples using the same protocol, the multiverse analysis results themselves frequently failed to replicate across samples (PMID 37166859).

There are also limitations. Putting together a multiverse is heavy lifting, and records of which choices were made and why they were accepted as justifiable alternatives are easily omitted. This means that what goes into a multiverse is also the researcher's judgment, and registration tools to standardize this are emerging (PMID 41164340). Interpretation also requires nuance. A working paper (not peer-reviewed) that ran 16,128 specifications on smartphone usage logs reported that absolute values such as usage time varied by up to nearly three times, but the rankings among participants were stable (DOI: 10.2139/ssrn.7008242). This allows for the interpretation that comparing absolute values across studies is difficult, while comparisons between individuals are relatively solid.

How to Measure from the Outside ③ p-curve

There are also tools that test entire bodies of literature rather than individual papers. p-curve is a method that collects the p-values of studies that came out significant (p < .05) on a single topic and looks at the shape of their distribution. When a true effect exists, the shape shows more p-values in the .01 range than the .04 range, meaning a distribution "tilted to the right." When there is no effect, it is flat or tilts the other way. Therefore, only a right-skewed p-curve diagnoses evidentiary value, and otherwise, selective reporting cannot be ruled out as the sole explanation (PMID 23855496). Selective reporting happens through publication bias (Same Trials, Different Conclusions — Publication Bias and What Went Into the Pool), where only significant studies get published, or through p-hacking.

Whether the tool is reliable is also tested by applying it to subjects where it is reasonable to assume there is no effect. When applied to placebo-controlled trials of ultra-diluted homeopathic preparations with a high likelihood of not containing a single molecule of the active substance, the p-curve in this single set accurately rejected evidentiary value (PMID 30697492). In 43 studies exploring action-related language comprehension by stimulating the motor cortex, the result was that it could not be asserted that a real effect was being investigated, and the estimated statistical power (the probability of catching a real effect when it exists) was also under 30% (PMID 36037977).

However, the tool itself is under debate. Critics argue that it is difficult to apply to heterogeneous sets of p-values gathered via text mining (a method that automatically scrapes p-values from papers), and that it misses p-hacking in the form of measuring multiple outcome variables and reporting only the significant ones. In simulations where outcome variables were uncorrelated with each other, a "bump" just below .05 did not form, so the absence of a bump does not mean the absence of p-hacking (PMID 26925335). A reanalysis applied to the debates over power posing (the claim that confident postures change hormones) and HPA (stress hormone system) reactivity advised halting its use, stating that conclusions are not replicated depending on which p-values are selected and plugged in (PMID 38990830). On the other side, there is a counterargument that the p-curve distinguishes between replicable and non-replicable findings even in observational studies, which was shown with two examples (PMID 30856227).

Measurements of how widespread p-hacking is across the literature also vary. A study investigating general science through text mining concluded that while p-hacking is widespread, its effect is weak compared to the actual effect size, so it does not seem to greatly alter the consensus of meta-analyses (PMID 25768323). About 960,000 abstracts were searched across roughly 100 radiology journals, and analyzing 4,105 abstracts where one p-value was randomly drawn per abstract, the phenomenon of p-values clustering just below .05 was not found. However, the authors caveat that this analysis does not catch all forms of p-hacking (PMID 36412994). Among 21,332 abstracts from 4 epidemiology journals, the proportion just below .05 has decreased since 2000, but the authors viewed that it fits the data better to explain this as increased statistical power rather than a decrease in p-hacking (PMID 40838582).

To sum up, the existence of degrees of freedom and how much they have contaminated the actual literature are different questions, and the second question still has divided answers.

Setting Rules in Advance — Preregistration and Registered Reports

If checking methods from the outside examines results that already exist, setting rules in advance locks in choices before the results are seen. Preregistration is the act of nailing down a research question and an analysis plan in a public registry before knowing the results. Making up a hypothesis with existing observations and testing a hypothesis with new observations are conceptually distinct, but in practice this is not well kept, and because of hindsight bias (the tendency to feel that you knew it all along once you know the result), researchers themselves do not feel the difference well (PMID 29531091). Preregistration has two functions: it makes you make decisions before knowing the results to reduce bias, and it increases transparency so others can evaluate that risk (PMID 36707644). A registered report is a publishing format where the introduction, methods, and analysis plan undergo peer review before results come out and receive in-principle acceptance for publication (PMID 39052343).

So what actually changed? The results are mixed.

Comparing 193 psychology studies that received preregistration badges or awards with 193 paired non-preregistered studies, the preregistered side did not have a lower rate of positive results, smaller effect sizes, or fewer statistical errors. This was contrary to the authors' prior expectations. However, power analyses were done more often and samples were larger, and the authors summarized that they found no clear evidence that preregistration prevents p-hacking and HARKing (PMID 37950113). In a study matching 300 preregistrations with their corresponding papers, part of the reason why preregistrations fail to tie down all degrees of freedom is visible. Methodological details were frequently missing from preregistrations, departures from the plan were often not disclosed, and inconsistencies were concentrated mainly in data collection procedures, statistical models, and exclusion criteria (PMID 39388105).

In a registered report (Stage 2, peer review completed) comparing 103 preregistrations written with a structured quantitative research template to 52 general OSF format ones, the binding score of the former was significantly higher, and preregistrations that went through peer review were higher still. Still, 73.68% of the 19 articles matched had undeclared departures (DOI: 10.1177/25152459261432216). In the clinical field, randomized controlled trials identified as registered reports were rare and concentrated in a single journal family, where 79 out of 93 (84.9%) made their protocols public after registering the first patient, and 44% changed their primary outcome variables, with only 33% of those mentioning the change. The authors concluded that these trials failed to keep the basic features of the registered report format (PMID 37245701). A preprint (before peer review) applying z-curve (a method that estimates and adjusts for selection bias from the distribution of significant results) to the entire literature of a psychology journal estimated the risk of false positives at 15% overall (95% confidence interval 6–32%, "A Confidence Interval Is Not a Margin of Error — How to Read the Two Numbers in the Parentheses"), and 20% for core tests (95% confidence interval 7–52%). This is a different metric (estimated false positive risk) from the 5% false positive rate earlier. This preprint reported that after the so-called reliability revolution, selection bias decreased but was still strong, that the reason for the decrease was mainly increased sample size and power, and that the effect of preregistration was small (DOI: 10.31234/osf.io/6ybeu_v2).

There are also criticisms. Campbell's law is the principle that when a useful tool hardens into an indicator of good science, getting that indicator itself becomes the goal and distorts the tool. A commentary applying this law to preregistration badges pointed out that papers chasing badges are already breaking the rules and intent, and warned of a new practice called "preregistering after knowing the results" (PMID 38783515). On the other side, a review in the context of toxicology viewed that some criticisms stemmed from misunderstandings and that the criticisms overall lacked empirical evidence, while acknowledging the risk that whether something is preregistered is used as a proxy indicator of quality as a valid concern (preprint before peer review) (DOI: 10.31234/osf.io/dqap7).

Honest Unsolved Problems

Degrees of freedom are hard to eliminate completely. Exploratory analysis itself is not bad. The problem lies in presenting exploration as confirmation, and what is appropriate depends on intention and transparency (PMID 34780242). Even if you switch to an estimation method that is robust against assumption violations, deciding which method to use and when becomes a degree of freedom all over again, which leads to proposals to preregister even sensitivity analyses (analyses that check whether results hold up even when analytical choices are changed) (PMID 41048396).

The replication side has the same degrees of freedom. A study reanalyzing the data of a single debate points out that the flexibility of replication experimenters can manufacture a "replication failure" even while appearing to follow strict criteria, and that this problem is not sufficiently resolved by preregistration (PMID 31767750).

It is linked to multiple factors. In a survey of 3,050 researchers, publication pressure ($\beta = 0.12$) and motives of fame and self-esteem showed a positive association with engaging in questionable practices, while commitment to scientific norms and the motive to falsify theories showed a negative association. $\beta$ is a coefficient indicating the direction and strength of the association, and the respondents were a portion of those invited who chose to respond, primarily European and US researchers (PMID 42469920). There is also a multilevel analysis showing that contract type and career stage are predictor variables while variance between institutions and countries is negligible (PMID 39397013).

Five Things to Check in a Paper

When reading a paper, red flags generally reveal themselves in five places.

• Whether the analysis plan was registered before looking at the data, and whether the registry entry and the paper actually match, is the first. Registration alone is not enough. Because unreported deviations are frequent, whether the registry entry and the paper match is also important (PMID 39388105, DOI: 10.1177/25152459261432216).
• Whether exclusion criteria, outlier handling, and covariate selection are specified is the second. These three spots are where cases have been reported in which the results changed depending on the choices made (PMID 31841112, PMID 37691812, PMID 31568872).
• How sensitive the results are to specification choices is the third. If a sensitivity analysis or multiverse is attached and the amounts that wobble and the amounts that do not wobble are distinguished, that is a good sign (PMID 32719546, DOI: 10.2139/ssrn.7008242).
• Whether the reported results are everything that was measured or just picked ones is the fourth. In studies compiling questionnaires, the practice of dropping information has been reported more commonly than the practice of adding or changing it (PMID 41963575).
• Whether the hypothesis preceded looking at the data is the fifth. Presenting post-hoc hypotheses as a priori ones lowers the reliability of the results (PMID 29531091, PMID 15647155).

Each of the above items may be a justified judgment on its own. However, surveys suggest that publication pressure or prestige motives can be layered onto choices made after seeing the results (PMID 42469920). In a study of 73 teams, differences between teams were hardly explained by expertise or prior beliefs, and the authors concluded that when stating the results of a single study, they should report them more humbly and clearly. Methods of freezing choices before seeing the results and methods of running and laying out all possible choices are current countermeasures, and the honest state of affairs right now is that neither is yet complete.

Source Papers

• "Questionable Research Practices: A Principled Classification and Ranking Based on Survey Data" (2026) — PMID 41963575 — Systematically gathers, classifies, and ranks survey definitions. The most common items are analysis, citation, and covariate selection, while the rarest axes are fabrication, falsification, and plagiarism. The 'subtraction' of information is more common than 'addition or alteration'.
• "False-positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant" (2011) — PMID 22006061 — The starting point for the concept of researcher degrees of freedom. Features simulations and 2 actual experiments, proposing 6 requirements for authors and 4 guidelines for reviewers.
• "Measuring the prevalence of questionable research practices with incentives for truth telling" (2012) — PMID 22508865 — Over 2,000 psychologists combined with incentives for truth-telling. Suggests that some practices may be effectively standard norms.
• "Prevalence of questionable research practices, research misconduct and their potential explanatory factors" (2022) — PMID 35171921 — 6,813 Dutch researchers. 51.3% frequently engage in at least one practice, with fabrication at 4.3%, falsification at 4.2%, and publication pressure having an OR of 1.22.
• "Degrees of Freedom in Planning, Running, Analyzing, and Reporting Psychological Studies: A Checklist to Avoid p-Hacking" (2016) — PMID 27933012 — A 34-item checklist covering degrees of freedom across the entire process. Used for training and evaluating preregistration quality.
• "The hazards of dealing with response time outliers" (2023) — PMID 37691812 — Choosing how to handle outliers inflates p-values and distorts confidence intervals and effect sizes. Alternatives (Bayesian, heavy-tailed) also open up yet another layer of flexibility.
• "Navigating the garden of forking paths for data exclusions in fear conditioning research" (2019) — PMID 31841112 — Heterogeneity in exclusion criteria for non-learners and non-responders, along with re-analysis examples. Proposes consensus criteria.
• "The quandary of covarying" (2020) — PMID 31568872 — 0 to 14 covariates across 2 years of structural neuroimaging data, forming 37 combinations across 68 models. Public data validation shows that the pattern of results flips depending on the combination.
• "Researcher degrees of freedom in statistical software contribute to unreliable results" (2023) — PMID 35953660 — Comparing SPSS, SAS, Stata, and R. Differences in p-values across packages range from .005 to .162 in a 2×2 chi-square test.
• "Garden of forking paths in ERP research" (2024) — PMID 38961523 — Applying 14 literature-based pipelines to the same N400 data. Only the low-pass filter does not affect the results.
• "HARKing: hypothesizing after the results are known" (1998) — PMID 15647155 — Definitions, types, and deterrents for the practice of presenting post-hoc hypotheses as a priori ones.
• "Researcher degrees of freedom in phonetic research" (2018) — DOI: 10.31234/osf.io/fp4jr — Demonstrates inflated false positives via Type I error simulations (preprint prior to peer review).
• "The Garden of Forking Paths: … in Cyclostratigraphic Analysis" (2019) — DOI: 10.1002/essoar.10500564.1 — Identifies a 100% periodicity detection success rate as a sign of confirmation bias. Uncorrected multiple testing and data-dependent procedure modifications (preprint prior to peer review).
• "Big little lies: a compendium and simulation of p-hacking strategies" (2023) — PMID 36778954 — A list of 12 p-hacking strategies, false-positive rate simulations, and strategy-by-strategy evaluations of countermeasures.
• "Observing many researchers using the same data and hypothesis reveals a hidden universe of uncertainty" (2022) — PMID 36306328 — 161 researchers across 73 teams. Even when all decisions are coded, over 95% of the variance remains unexplained.
• "Meta-analyzing the multiverse: A peek under the hood of selective reporting" (2025) — PMID 37166859 — Counterfactual reconstruction of 294 registered replication studies. Statistically significant effects in the hypothesized direction appear in only about 30% of studies, with degrees-of-freedom effects inconsistent across samples.
• "Specification curve analysis" (2020) — PMID 32719546 — A 3-step process of specification identification, visualization, and combined inference. Applied to three findings to distinguish between robust, weak, and non-robust.
• "Grilling the data: application of specification curve analysis to red meat and all-cause mortality" (2024) — PMID 38354868 — 1,208 specifications derived from 70 methods extracted from the literature. Of the 48 significant ones (3.97%), 40 point in the direction of reduced mortality.
• "Using electrodermal activity to estimate fear learning differences in anxiety: A multiverse analysis" (2024) — PMID 39142133 — Only 1.45% of 1,240 analyses are significant in a way that matches theory.
• "A multiverse meta-analysis of oxytocin administration studies" (2025) — PMID 40848964 — 256 meta-analyses using 185 papers and 530 effect sizes. d = -0.16 to 1.45, though over 90% exceed the zero-effect bootstrap range.
• "The Systematic Multiverse Analysis Registration Tool for defining multiverse analyses" (2025) — PMID 41164340 — A registration tool that step-by-step documents the decisions and rationales behind constructing a multiverse analysis itself.
• "From Logs to Metrics: The Impact of Researcher Degrees of Freedom on Smartphone Usage Metrics" (2026) — DOI: 10.2139/ssrn.7008242 — 16,128 specifications. Absolute values differ by up to 3-fold, but individual rankings remain stable (average Spearman .87 to .93) (working paper prior to peer review).
• "P-curve: a key to the file-drawer" (2014) — PMID 23855496 — A method for judging evidential value based on the right-skewness of significant p-value distributions.
• "P-curve accurately rejects evidence for homeopathic ultramolecular dilutions" (2019) — PMID 30697492 — Correctly rejects effects when applied to phenomena considered non-existent.
• "Does the involvement of motor cortex in embodied language comprehension stand on solid ground?" (2022) — PMID 36037977 — P-curve and excess significance tests on 43 papers. Estimated statistical power is below 30%.
• "Problems in using p-curve analysis and text-mining to detect rate of p-hacking and evidential value" (2016) — PMID 26925335 — Phantom-variable p-hacking leaves no 'bump' and thus gets missed. The absence of a bump is not evidence for the absence of p-hacking.
• "The inconsistency of p-curve: Testing its reliability using the power pose and HPA debates" (2024) — PMID 38990830 — Conclusions fail to replicate depending on the permutation of p-value choices. Recommends discontinuing use.
• "P-curve won't do your laundry, but it will distinguish replicable from non-replicable findings in observational research" (2019) — PMID 30856227 — A counter-argument defending the applicability to observational studies.
• "The extent and consequences of p-hacking in science" (2015) — PMID 25768323 — Confirmed broadly through text mining. However, the effects are relatively weak compared to true effect sizes and do not appear to drastically alter meta-analytic consensus.
• "Is There Evidence of P-Hacking in Imaging Research?" (2023) — PMID 36412994 — 967,981 abstracts and 293,687 p-values. No concentration just below .05. Other forms of bias cannot be ruled out.
• "Trends in the distribution of P values in epidemiology journals" (2025) — PMID 40838582 — 21,332 abstracts. A drop just below .05 is better explained by an increase in statistical power than a decrease in p-hacking.
• "The preregistration revolution" (2018) — PMID 29531091 — The confusion between hypothesis generation and testing, hindsight bias, and the logic and practical strategies of preregistration.
• "Reducing bias, increasing transparency and calibrating confidence with preregistration" (2023) — PMID 36707644 — The two functions of preregistration (reducing bias and increasing transparency) and a summary of metascience.
• "Want to try a registered report? Here are our lessons learned" (2024) — PMID 39052343 — An overview of the registered report format and 8 practical lessons.
• "Preregistration in practice: A comparison of preregistered and non-preregistered studies in psychology" (2024) — PMID 37950113 — A 193-to-193 comparison. No differences in positive results, effect sizes, or statistical errors, but advantages in power analysis, sample size, and impact.
• "The potential of preregistration in psychology: Assessing preregistration producibility and preregistration-study consistency" (2024) — PMID 39388105 — Comparing 300 preregistrations. Detailed omissions and unreported departures are frequent, with inconsistencies concentrated in data collection, statistical models, and exclusion criteria.
• "Stage 2 Registered Report: Restriction of Researcher Degrees of Freedom Through the Psychological Research Preregistration-Quantitative Template" (2026) — DOI: 10.1177/25152459261432216 — 103 structured templates versus 52 OSF forms. Binding force is significantly higher, even more so when peer-reviewed. Still, 73.68% of 19 corresponding papers contain unreported departures.
• "Published registered reports are rare, limited to one journal group, and inadequate for randomized controlled trials in the clinical field" (2023) — PMID 37245701 — 93 RCTs. 84.9% published protocols after the first patient enrollment, 44% changed primary outcome variables, and only 33% mentioned it.
• "False Positive Psychology? An Empirical Investigation of the False Positive Risk … Before and After the Credibility Revolution" (2025) — DOI: 10.31234/osf.io/6ybeu_v2 — z-curve estimated false positive risk is 15% overall (6–32%) and 20% for core tests (7–52%). The effect of preregistration is small (preprint prior to peer review).
• "Campbell's Law Explains the Replication Crisis: Pre-Registration Badges Are History Repeating" (2025) — PMID 38783515 — The argument that when a measure becomes a target, the tool becomes distorted. Warns against 'preregistering after knowing the results'.
• "The Benefits of Preregistration and Registered Reports" (2024) — DOI: 10.31234/osf.io/dqap7 — A counter-argument that most criticisms stem from conceptual confusion or a lack of empirical evidence (preprint prior to peer review).
• "Understanding the exploratory/confirmatory data analysis continuum" (2022) — PMID 34780242 — The distinction between exploratory, confirmatory, and 'rough confirmatory' data analysis, and the ethical boundaries with p-hacking and HARKing.
• "Robust statistical methods and the credibility movement of psychological science" (2025) — PMID 41048396 — A proposal to preregister sensitivity analyses as well, since handling assumption violations can itself become a degree of freedom.
• "Replicator degrees of freedom allow publication of misleading failures to replicate" (2019) — PMID 31767750 — Flexibility on the replication side can produce misleading failures to replicate, which is not fully solved by preregistration.
• "Why researchers cut corners: motivations, pressures, and personality in questionable research practices" (2026) — PMID 42469920 — 3,050 researchers. Publication pressure has β = 0.12; status and ego motives show a positive association, while commitment to scientific norms and falsification motives show a negative association.
• "Investigating the links between questionable research practices, scientific norms and organisational culture" (2024) — PMID 39397013 — Contract type, career stage, and norm adherence are predictive variables. Variance between institutions and countries is negligible.