What Is p = 0.03 the Probability Of? — The One Question a p-Value Answers

The story begins with an international trial that gave aspirin to patients with acute ischemic stroke. 467 hospitals across 36 countries gathered 19,435 patients, and a draw determined who would receive aspirin. Six months later, the proportion of people who had died or could not live without others' help was 62.2% in the group that received aspirin and 63.5% in the group that did not. That is a difference of 13 per 1,000 people.

To gauge whether this difference is by chance, we calculate a probability. It is the probability of getting a difference this large, or even wider, assuming aspirin has no effect at all. This probability is the p-value. In this trial, it was calculated as 0.07.

There was one more calculation. When entering the trial, patients each had a different prognosis, meaning an outlook predicting their future course. Adjusting for these differences at the starting line in the calculation is called adjustment. Once the prognosis was adjusted for, the difference became 14 per 1,000 people, and the p-value became 0.03 (PMID 9174558). The abstract lists this value as 2p, which is an indication that it accounts for the difference spreading in either direction.

The difference remains almost the same, moving from 13 to 14 per 1,000 people. Yet the p-value crossed from this side of the 0.05 line to the other. That is the baseline papers commonly use to divide what is "statistically significant." Behind the small difference of 14 per 1,000 people crossing the line lies the large enrollment of 19,435 people. This single trial contains two things together that move the p-value. One is the size of the sample, which asks how many people were observed, and the other is the analyst's choices, such as whether to make adjustments or not. This article adds a third factor, running multiple comparisons, to look at how these three move the p-value in actual papers.

⚠️ The names of the drugs and trials appearing below are all examples intended to show how to read numbers. They are not judgments on whether or not to use a particular treatment, and each number holds meaning only within that study's design and subjects.

Four Terms to Recap First

We write down here again four terms established with a single coin in "What Does "Statistically Significant" Actually Say? — The p-Value, Explained with a Coin Toss" (hereafter the primer).

The null hypothesis is the assumption that "there is no effect at all." The p-value is the probability of obtaining the observed value, or a value even more extreme, assuming the null hypothesis is true. The probability calculated in the stroke trial is precisely this. We decide in advance where to draw the line below which we call it "hard to dismiss as a coincidence," and that line is the significance level. The most common line is 0.05, and when the p-value drops below this line, we call it "statistically significant."

The effect size is a value that measures how large the difference is. In the stroke trial, the difference in proportions between the two sides is the effect size. If the p-value tells us "how rare it is," the effect size tells us "how large it is." Lastly, "not significant" only goes as far as saying "these data failed to push aside the assumption that there is no effect."

The primer showed through calculation that if you flip a coin 10,000 times, even if heads comes up only 51.5%, it crosses the line. Even though it is skewed only very slightly from half, when the number of flips is large, that small skew becomes "significant." This article looks at how the exact same thing happens in an actual paper.

When a Study Has Many People, Even a Small Difference Crosses the Line

The authors of the stroke trial interpreted aspirin as producing a small but worthwhile improvement at the 6-month mark. Placed alongside a similarly sized Chinese trial, they wrote that aspirin during the first few weeks modestly but genuinely reduced deaths or recurrent strokes by about 10 per 1,000. They also wrote that both trials suggest starting aspirin as early as possible after an ischemic stroke begins. They added that earlier trials had shown that continuing low-dose aspirin provides protection over a longer duration. This interpretation is from 1997. This trial enrolled only patients with suspected ischemic stroke, and this article does not translate this interpretation into a prescribing decision. The harms are written in the same units. In the group that received aspirin, there were 5 more major extracranial bleeds per 1,000 that required transfusion or proved fatal. In people who did not receive heparin alongside it, that difference was 2 and was not significant (PMID 9174558).

14 per 1,000 is a small difference, just as the authors said. Behind the fact that this small difference was able to cross the line lies the circumstance that there were many people. The more people there are, the smaller the jaggedness due to chance becomes, and to that extent, even a small difference receives the verdict of being "too rare to be a coincidence." Along with that verdict, this trial recorded the magnitude of a few people per 1,000. Therefore, readers can directly evaluate the interpretation that it is "small but worthwhile."

What happens when the magnitude is not reported is clearly seen in an extreme example. In roughly 950,000 artificial joint surgery records, 14 out of 20 pairs of variables matched for no medical reason crossed the line. One of those pairs was the surgery code number and a blood glucose marker, and the value measuring the degree to which the two moved together was −0.0435. Because this is a scale where 0 means no movement together at all and 1 or −1 means moving completely together, −0.0435 is very close to 0. This means there is no meaningful relationship between the two variables. Yet the p-value was $2.14 \times 10^{-40}$, a value where numbers only appear down at the fortieth decimal place. The authors conclude that these results were created purely by sample size without clinical meaning. They also point out the danger of overinterpreting significance in large database studies, and write that a method of establishing hypotheses beforehand and the interpretation of effect sizes are necessary (PMID 40885217).

The same thing can be seen when gathering randomized trials. Researchers determine the expected effect size before starting a trial and set the number of people so that they can catch that magnitude. Among 59 randomized trials adding adjuvants to peripheral nerve blocks (a method of placing an anesthetic around a nerve to block pain in the area served by that nerve), 45 trials (76%) reported significant results in their main outcomes. However, 22% of those fell short of the effect size expected at the beginning. Trials properly reporting sample size calculations were 78%. The authors conclude that we should use these outcome metrics and expected effect sizes in future trials to make methods more uniform (PMID 38978187). What this article reads from the same numbers is singular. A difference that falls short of the magnitude the researcher expected can also receive the verdict of being significant. Whether that difference was clinically small and why it fell short of expectations is not in the abstract.

In studies with many people, a small p-value does not mean a large difference. Magnitude is told separately by the effect size.

When the Sample Size Is Small, "Not Significant" Does Not Mean "No Effect"

Sample size moves p-values in the opposite direction, too. When there are few people, the choppiness is large, so even if there is a real difference, it is hard for the p-value to drop below the line. There is the example of 35 adolescent male soccer players. Seventeen neither ate nor drank during daylight hours for three days, while 18 lived as usual. In none of the items—memory, attention, or arousal response—did a significant difference emerge in the changes between the two sides.

The authors frame this as preliminary, hypothesis-generating evidence that no cognitive impairment was significantly detected under this condition. This study was not designed to show that the difference between the two sides was smaller than a meaningful magnitude. Therefore, they write, effects smaller than the targeted medium size cannot be ruled out. The conclusion is that from just the absence of a significant difference, one cannot infer that fasting and athletic training can safely go together (PMID 42588098). To go from "we did not find a difference" to "we showed there is no difference," one must design the study to show that from the start.

The same problem tripped up drug approvals, too. The initial approval of the COVID-19 treatment remdesivir relied mainly on two trials. A third trial was judged inconclusive because it was not large enough to tease out an effect. With ordinary calculations, it is hard to tell whether this trial's "not significant" stems from too few people or no effect. Bayesian hypothesis testing is a different calculation method that can also count evidence for the lack of an effect. A research team reanalyzed the three trials using this calculation method in 2021. For the primary outcome of clinical improvement, the evidence was ambiguous either way. For the secondary outcome of reduced mortality, moderate evidence emerged toward no effect. Additional analyses of the three studies published after approval pointed the same way. The authors advise regulators to judge based on the totality of available evidence. They also write that this calculation method is particularly useful when results are not significant, and that this task is more urgent when clinical efficacy data are scarce (PMID 34297766).

There is also a practice of using a small sample size as an excuse after the results are out. In a study that ends without finding an effect, one works backward to calculate "how much power this study had to catch an effect," and explains that "we could not find a benefit because there were too many patients [sic — too few patients]." An editorial introduction attached to a statistical series in a bone marrow transplantation journal singles this out as the most severe misuse. Because the power to reject the null hypothesis is already built into the p-value calculation, one must not interpret a negative result using power calculated after the results are out. The evidence for this point is a single editorial commentary (PMID 36869191).

A "not significant" result from a study with few people cannot separate "there is no effect" from "there were not enough people." There are two ways to get closer to separating the two. One is to design the study from the beginning to show that the difference is smaller than a meaningful magnitude, and the other is a calculation method that also counts evidence for the lack of an effect.

However, a paper summarizing these methods notes that no statistical approach can prove the null hypothesis is true. Even so, it concludes that now that these tools have become easy to use, the time has come to complement or move beyond traditional testing and say more about the side where there is no effect (PMID 30873486).

If You Compare Many Times, Rare Events Become Common

The football player study mentioned earlier asked four items together using the same questionnaire, and one of them was self-perceived sleep quality. The p-value measuring the difference in change between the two sides for sleep quality was 0.029, which crossed the 0.05 line. However, because this study compared four items together, it set the line strictly at 0.0125, which is one-fourth of 0.05. Because it compared four times, it divided the line into four parts. The value 0.029 failed to cross this strict line. The authors interpret this result only as a way to generate a new hypothesis and write that it must be confirmed by replication studies. This difference was the result of a combination of worsening sleep in the group that went about their usual routine and a non-significantly slight improvement in the fasting group (PMID 42588098).

Performing multiple tests in a single study is called multiple comparisons. If you measure five outcome indicators, divide participants into three groups, and set four time points, the number of tests quickly runs into the dozens. Even if there is no effect at all, a few of them can cross the line. If you flip 20 fair coins 10 times each, the probability that at least one crosses the line is about 35% (calculated as in the primer).

There is also a device that converts how rare 0.05 is into a sensory scale. The value obtained by converting the p-value to −log₂(p) is called the S-value. This value is a count of how many times the whole must be folded in half to become as small as p. It was proposed to align intuition against simple experiments like flipping a fair coin (PMID 32998683, PMID 37637018). If the S-value is s, it is read as meaning it is as surprising as flipping a fair coin s times in a row and getting heads every time. The S-value for p=0.05 is about 4.3, which is the degree of surprise of flipping a fair coin just over four times and getting heads every time.

However, this surprise is only when you look just once. The probability of getting four consecutive heads somewhere while flipping a coin 20 times is close to half. The value 4.3 and 「close to half」 are values translated into arithmetic in this article. The record of 20 flips has seventeen overlapping spots where four consecutive flips can occur, and because it is a matter of examining those spots all at once, it is a different calculation from the 20 coins earlier. Transferred to comparisons, if 20 comparisons that do not affect each other are each set at the 0.05 line, the probability that at least one crosses the line is about 64% (this is also the arithmetic of this article). What is rare when looked at just once becomes common when looked at multiple times.

If you divide a map into small cells and test the trend over time for each cell, the number of tests becomes in the thousands. If you do not account for all of them in the calculation, false discoveries increase in unpredictable ways. The reality of a judgment that a result is significant also depends on how many results were tested together but were not significant and what those values were. In the context of multiple comparisons, you cannot correctly determine what is significant until you know the whole story. This diagnosis came from a discussion paper that the authors themselves call a selective and non-exhaustive overview (PMID 39637466). A guidance paper for health professions education researchers also notes that if the number of comparisons is not properly accounted for, the 「p<0.05」 criterion can mislead results (PMID 38680196).

To read a single p-value written in a paper, you must also know how many comparisons lay behind it.

An Analyst's Choices Can Also Move the p-Value

Let us return to the stroke trial. The unadjusted calculation and the calculation adjusted for prognosis came from the same data and are written side by side in the abstract. Adjusting for prognosis at the time of entry into the trial is a commonly used analysis. Yet, the single choice of whether or not to adjust caused the p-value to cross the line. Because this trial recorded both values, readers could see that movement.

Not all choices are revealed like this. Even with the same data, there are multiple paths of analysis. One must decide whether to remove or keep outlier values, which variables to put into the calculation, and when to stop data collection. The range of choices available to a researcher along the path of analyzing the same data is called researcher degrees of freedom. If you choose paths while looking at the results, it is equivalent to performing a test for each path, but only the final chosen one is written in the paper. The task of deciding which variables to include—namely, the process of selecting a model—generates a little-known form of multiple testing. An educational paper for nursing researchers notes this point, recording that the abuse of researcher degrees of freedom has received almost no attention in that field. The same paper concludes that, despite these limitations, the framework of hypothesis testing helps rule out the possibility that the observed effect is explained by chance and will remain the main tool going forward (PMID 31349121).

Deciding which results to report is also a choice. The choice to selectively report only significant results is revealed only when gathering multiple papers. An analysis gathering social psychology presentations from scientific meetings of two major Japanese psychology societies is one such example. Meeting presentations effectively do not undergo peer review. Therefore, it was possible to test whether researchers selectively report only significant results even without pressure from editors or reviewers. In 2013 presentations, the proportion of significant results was 76%. However, the proportion expected based on the power of those studies—that is, the likelihood of catching an effect—was 7%. The authors read this gap as a signal that selective reporting had occurred.

However, the presentations were not completely devoid of evidence. The proportion expected to be significant again if the same study were replicated identically, the expected replication rate, was 31%. This is a value significantly higher than the 5% expected when there is no effect at all. In 2018, the expected replication rate rose to 44%. The maximum estimated proportion of falsely discovered results among those reported as significant that could actually be due to no effect fell from 68% to 35%. Even so, the authors viewed this value as similarly suggesting that a substantial portion of the reported results were false positives. The authors' conclusion is that selective reporting took place, but the field as a whole was not covered by false discoveries. They add that while there are signs of slight improvement, even in 2018 the value of evidence is weak so one cannot be optimistic (PMID 38250729).

Choices also happen at the stage of putting results into words. When 580 papers published between 2019 and 2020 in a neurophysiology journal were audited, 40% did not define the significance level for their own tests. 64% of papers with p-values between 0.05 and 0.1 misinterpreting them as a "statistical trend." Values that failed to cross the line were read under the name of a "trend." This journal had already been publicly cited for poor reporting in a 2016 audit and had revised its author guidelines. The authors summarize that public citation had little effect on improving reporting, and that requiring or strongly encouraging reporting practices was also only partially effective. Although the audit was limited to a single journal, they judge that it is likely not an isolated case and conclude that stronger measures are needed (PMID 38434651).

An analyst's choices move the p-value, and readers can see that movement only when those choices are fully written in the text.

Three Questions to Ask When You Meet a p-Value in a Paper

Putting these three together determines what to ask in front of the abstract's "p=0.03".

The first is how many people were looked at. If there are very many people, look at whether the size, such as how many per 1,000 people, is written next to that p-value. If people are few and the result is not significant, look at whether the study was designed to show that the difference is smaller than a meaningful size.

The second is how many times comparisons were made. Look at how many indicators, groups, and time points were looked at, and whether the line was adjusted for that number. Papers that write down before adjustment and after adjustment together, like the soccer player study, answer this question themselves.

The third is whether the analysis was decided in advance. Look at whether what the pre-decided analysis was is written down, whether multiple calculations were written side by side like the stroke trial, and whether the conclusion holds up in other analyses. The earlier primer paper also recommends reporting the effect size, the confidence interval—which is the range that writes down how much the estimate can wobble—and the null hypothesis together, and checking whether the result holds up even if the analysis is done differently (PMID 38680196).

The p-value answers a single question of how rare such a result would be if there were no effect at all, and that answer moves according to the sample size, the number of comparisons, and the choices of the person analyzing.

Where the line of 0.05 came from, and how to count how many patient events a significant result actually rests on, is looked at next in "Where Did 0.05 Come From? — Counting How Many Patients a Finding Rests On". How to read the range attached next to the size was covered in "How Much Does a Number Counted Once Wobble? — The Two Numbers in the Parentheses, the Confidence Interval", and how that range is commonly misunderstood is looked at next in "A Confidence Interval Is Not a Margin of Error — How to Read the Two Numbers in the Parentheses".

Source Papers

• "The International Stroke Trial (IST): a randomised trial of aspirin, subcutaneous heparin, both, or neither among 19435 patients with acute ischaemic stroke." (1997) — PMID 9174558 — 19,435 patients. Aspirin 6-month death and dependency 62.2% versus 63.5% (13 per 1,000, 2p=0.07), 14 per 1,000 after baseline prognosis adjustment (2p=0.03). Authors' interpretation is a small but valuable improvement. Aspirin side of transfusion and fatal extracranial bleeding exceeds 5 per 1,000 (2 without heparin, non-significant).
• "Fully Randomized Predictor-Outcome Pairings Using a National Database Yield Frequent Statistical Significance Without Clinical Meaning." (2025) — PMID 40885217 — 955,092 joint replacements, 14 out of 20 pairs matched by random numbers have p<0.05. Surgery code versus glycated hemoglobin r=−0.0435, P=2.14×10⁻⁴⁰. Authors' conclusion is significance created by sample size, requiring hypothesis-driven methods and effect size interpretation.
• "Primary outcomes and anticipated effect sizes in randomised clinical trials assessing adjuncts to peripheral nerve blocks: A scoping review." (2024) — PMID 38978187 — Primary results significant in 45 out of 59 papers (76%), among which 22% fall short of expected effect size. Adequate sample size calculation reported in 78%. Authors' conclusion is improving methodological homogeneity.
• "Effects of Three Consecutive Days of Diurnal Intermittent Fasting on Cognitive Function, Psychomotor Vigilance, and Sleep Quality in Adolescent Male Soccer Players." (2026) — PMID 42588098 — 35 participants (17 versus 18). No significant difference in cognitive items, not an equivalence or non-inferiority design so effects smaller than a medium size cannot be ruled out. Sleep quality p=0.029 fails to cross the 4-item adjustment threshold of 0.0125, making it hypothesis-generating.
• "Rethinking remdesivir for COVID-19: A Bayesian reanalysis of trial findings." (2021) — PMID 34297766 — The non-significance of the third trial, which yielded no conclusion, makes it hard to distinguish between lack of statistical power and absence of effect under the standard framework. In the reanalysis, clinical improvement is ambiguous, and mortality reduction shows moderate evidence toward an absence of effect. Regulators advise looking at the totality of evidence.
• "Making 'null effects' informative: statistical techniques and inferential frameworks." (2018) — PMID 30873486 — How to evaluate null results using equivalence testing, Bayesian estimation, and Bayes factors. No statistical approach can prove that the null hypothesis is true. Conclusion that tools have become easy enough that it is time to complement or move beyond traditional testing.
• "Inappropriate use of statistical power." (2023) — PMID 36869191 — Editor's introduction to a journal statistics series (not an original study). Highlights calculating observed power after a negative result as the most severe misuse, noting that power is already factored into the p-value calculation.
• "Semantic and cognitive tools to aid statistical science: replace confidence and significance by compatibility and surprise." (2020) — PMID 32998683 — S-value = −log₂(p). A device to match intuition by comparing it to simple experiments like coin tossing.
• "P-value, compatibility, and S-value." (2022) — PMID 37637018 — Proposal to read p-values as indicators of compatibility between data and models, and to gauge that compatibility using S-values compared to coin tossing.
• "The ghost of selective inference in spatiotemporal trend analysis." (2025) — PMID 39637466 — Discussion paper (selective and non-exhaustive overview). Trend testing for every grid cell becomes thousands of multiple tests, and in the context of multiplicity, one cannot determine what is significant without knowing the full story.
• "How to Use and Report on p-values." (2024) — PMID 38680196 — The p<0.05 threshold can mislead unless practical importance, number of comparisons, and test appropriateness are considered. Recommends reporting effect sizes, confidence intervals, null hypotheses, and sensitivity analyses.
• "Pervasive errors in hypothesis testing: Toward better statistical practice in nursing research." (2019) — PMID 31349121 — Educational paper. Multiple testing arising from model selection, abuse of researcher degrees of freedom. Concludes that despite limitations, the hypothesis testing framework helps rule out chance and will remain a primary tool.
• "A systematic review of conference papers presented at two large Japanese psychology conferences in 2013 and 2018." (2024) — PMID 38250729 — Unreviewed presentation abstracts. Social psychology presentation abstracts. 2013 observed discovery rate 76% versus expected 7%, expected replication rate 31% (significantly higher than 5%). 2018 at 44%, maximum false discovery rate dropping from 68% to 35%. Conclusion is that the field as a whole is not producing false discoveries, but the evidential value remains weak in 2018, precluding optimism.
• "Poor statistical reporting, inadequate data presentation and spin persist despite Journal awareness and updated Information for Authors." (2023) — PMID 38434651 — 580 papers from a single journal. 40% leave significance levels undefined, 64% of papers with p between 0.05 and 0.1 misinterpret them as a "trend". Public pointing-out had little effect on improvement, and arguing these are not isolated cases demands stronger countermeasures.