What Does "Statistically Significant" Actually Say? — The p-Value, Explained with a Coin Toss

The Headline "Statistically Significant Effect"

"New Treatment Confirms Statistically Significant Effect." You have probably seen a sentence like this once or twice in news headlines or article introductions. The word "significant" sounds like the effect is large or certain.

The opposite headline is also common. "○○, No Statistically Significant Difference." This sentence sounds like "there is no effect."

Both headlines come from a single number. It is the number written in the paper's abstract as "p=0.03" or "p<0.05". This article covers what question that number answers, and what question it does not answer.

The explanation uses a single coin. A friend hands you a coin and says, "This seems to land on one side a lot." After tossing it 10 times, heads came up 9 times. Is this coin a biased coin, or is it a normal coin that happened to turn out that way by chance?

This coin is a fabricated example for the explanation, and the probabilities of the coin in the article are values calculated directly by this article. We carry this coin all the way to the end of the article. Later, real studies dealing with exercise or counseling also appear. Those studies are merely examples showing how numbers are read, not judgments telling you to do or not do something.

If It Is a Fair Coin

There is no direct way to know right away whether a coin is bent. Instead, we can ask from the opposite side: "If this coin is a fair coin where heads and tails each come up half the time, how often does it skew to one side 9 times or more out of 10?"

The math goes like this. When you flip a coin 10 times, there are 1,024 possible sequences of heads and tails. That is because the branches split in two with every single flip. If the coin is fair, every single one of these sequences has the same probability of appearing.

Among them, the sequences with 9 or more heads are 11. That is the sum of the one sequence where all 10 flips are heads, plus the 10 sequences where tails sneaks in exactly once. Because the question was about skewing to "one side," we also count the tails side. The sequences with 9 or more tails are also 11. Put together, that is 22, which is about 2% out of 1,024. This means if you repeat setting out to flip a coin 10 times 100 times using a fair coin, about 2 times it will skew by this much.

The starting assumption, "this coin is fair," has a name. The null hypothesis is the assumption that "there is no effect at all." In terms of a coin, it is "this coin is fair," and in terms of medicine, it is "there is no difference at all between the group that took the medicine and the group that did not." It is an assumption set up on the opposite side of what the researchers want to show, and a baseline against which results are tested.

The 2% we just calculated also has a name. The p-value is the probability of obtaining the observed value, or a more extreme value, assuming the null hypothesis is true. The "observed value" is the result actually obtained, and the "more extreme value" is a result farther away from the assumption of no effect. We do not count just the single result we actually saw, but also results skewed even further than that (PMID 32539165). Translated to a coin, it is "the probability that a fair coin yields 9 or more of a skew, or an even greater skew." This is why we counted all 9 or more earlier. The "p = 0.03" written in the abstract is precisely this probability.

A small p-value means the result does not fit well with the assumption that "there is no effect at all." However, when calculating a p-value, there are other underlying assumptions we rely on besides the null hypothesis. In the case of a coin, it is the assumption that it was flipped honestly. Because of this, there is a suggestion to read the p-value as a value indicating how well the data fits the package combining both the null hypothesis and all the other underlying assumptions beneath it (PMID 37637018, PMID 32998683). This means a small p-value can be a signal that the coin is bent, or it can be a signal that somewhere in the flipping process went awry.

The p-value answers only one question: If there is no effect at all, how often would a difference this large or larger pop up by chance?

0.05 the Line

What if your friend's coin landed on heads 8 times? By the same math, the sequence of landing on one side 8 times or more has 112 variations, about 11%. For 9 times it is about 2%, and for 8 times it is about 11%.

Then the question remains of what percentage onward to call 「rare」. Papers set this threshold in advance before analysis. The significance level is a line decided in advance to call a p-value smaller than it 「hard to dismiss as a coincidence」. The most common line is 0.05, which is 1 in 20, and written as a percentage is 5%. If the p-value is smaller than the significance level, it is called 「statistically significant」.

Comparing this to your friend's coin, the p-value of about 2% for 9 times is smaller than this line, so it is significant. The about 11% for 8 times is larger than the line, so it is not significant. In this article, we will write that a p-value falling below the line 「crosses the line」. Your friend's coin crosses the line at 9 times, and fails to cross it at 8 times.

0.05 is a human-made convention. For a century, almost all research papers have used this line, but the line itself is arbitrary. A short review reflecting on its history explains why this line spread so widely as follows: the people using it were drawn to a simple method, and statisticians did not feel a need to forcefully suppress that enthusiasm (PMID 41748420).

There is a problem that arises when you draw a line. The p-value is a value that continues seamlessly from 0 to 1, but drawing a line splits 0.049 and 0.051 to opposite sides (PMID 28698825, PMID 32710113). This means that out of two almost identical values, one receives the judgment of 「significant」 and the other 「not significant」.

Therefore, there is an argument that we should not divide p-values into above and below a line, but read them as a continuous scale representing the strength of evidence (PMID 28698825). The approach is to read smaller p-values as stronger evidence against the assumption that 「there is no effect」. There is also a recommendation to teach people to interpret exact p-values rather than keeping their eyes on whether it is smaller than 0.05 (PMID 32710113). The recommendation to write the actual value rather than just 「p<0.05」 comes from the same reason (PMID 32157489).

Results near the line are in practice read with a slant to one side. Out of 97 sentences interpreting results that were not significant because the p-value was slightly larger than 0.05, 40 wrote the results as if they were significant. Conversely, out of 303 sentences where the p-value was slightly smaller than 0.05, 4 wrote them as if they were not significant (PMID 34355494). This is the result of extracting and counting sentences interpreting p-values near 0.05 from the most prominent general medical journals. The authors conclude that such misleading sentences were more common above 0.05, and urge researchers, clinicians, and editors to use p-values in line with their context and purpose.

What to do with this line is under debate. One side says we should scrap the line. In 2019, the executive director of the American Statistical Association urged that the phrase 「statistically significant」 no longer be used (PMID 32539165). Several statisticians and researchers also propose keeping p-values as they are, but discarding the line of 0.05 and the phrase 「statistical significance」 (PMID 41748420).

There are voices on the opposite side as well. An article in the field of radiology writes that we should not carelessly change the current 0.05 line. The reason is that clinical research must make decisions, so it needs yes-or-no answers. Regarding the proposal to lower the line to 0.005, it views that research costs could increase. This article also recommends writing the actual value rather than just 「p<0.05」 (PMID 32157489).

The line has one more property. 0.05 is a value set to cap the rate of getting a result that crosses the line when there is no effect at all at, at most, 1 in 20. In a coin flipped 10 times, only 9 or more on one side crosses the line, so the rate at which a fair coin crosses the line is a bit lower than that, at about 2%.

Even if it is 2% at a time, the story changes if you make multiple comparisons. If you flip 20 fair coins 10 times each, the probability of at least one crossing the line is about 35%. This means that even if there is no effect at all, if you make many comparisons, a 「significant」 result is likely to pop up somewhere among them. Making multiple comparisons without adjusting for them accordingly—in other words, failing to make the line stricter by the number of comparisons made—is cited as a common statistical error in medical research (PMID 37234723).

0.05 is a human-made convention, and p-values just above and just below the line are nearly the same degree of evidence.

How Big Is the Difference?

This time, let us say we flipped a different coin 10,000 times. It landed on heads 5,150 times. That is 51.5%, which is very close to the 50% of a fair coin. But if you calculate the p-value—that is, the probability of getting a skew this large or larger if the coin were fair—it is about 0.003. That is about 3 times in 1,000, and the p-value is much smaller than your friend's coin.

Let us put the two coins side by side. Your friend's coin came up heads 90% of the time and the p-value was about 0.02. The second coin came up heads 51.5% of the time and the p-value was about 0.03. The coin with the much smaller skew has the smaller p-value. What separated the two is the number of tosses. The more times you toss, the smaller the range in which the proportion of heads fluctuates purely by chance. That is why even a tiny skew gets judged as 「too rare to be a coincidence」.

That is why, separately from the p-value, we need a number that measures how large the difference is. Effect size is the value that measures how large a difference is. For a coin, it is 「what percentage is heads」, and for a medicine, it is a number like 「50% reduction in risk」 or 「from 2 out of 100 people down to 1」. How to read these two numbers was covered in 「If the Risk Was Cut in Half, How Many Fewer People Got Sick? — Relative and Absolute Risk」. More than 70 types of effect sizes are described in the literature, and differences in means, differences in risk, and ratios of risk all fall under this (PMID 29337724).

Hypothesis testing, the procedure for calculating p-values, tells us neither how large the effect is nor whether it is large enough to matter to a patient (clinical significance) (PMID 29337724). Things like the second coin happen in real records too.

In a national database containing about 950,000 knee and hip replacement surgery records, 20 pairs of record categories with no medical reason whatsoever to be paired were randomly selected, paired, and compared. In 14 of those pairs, the p-value was smaller than 0.05 (PMID 40885217). If a comparison had no effect at all, it is normal for about 1 out of 20 pairs to cross the line, so 14 pairs cannot be explained away as the result of multiple comparisons. The authors conclude that this significance was produced entirely by the sample size—that is, the fact that there were very many records examined—and has no clinical meaning. They also recommend setting hypotheses first before analyzing and interpreting the effect size.

Conversely, there are cases where a finding is judged to be statistically significant, but the question remains whether the magnitude of that difference is meaningful. In adults with rheumatoid arthritis, aerobic exercise statistically significantly reduced self-perceived fatigue (p=0.006). This is the result of recalculating data from 5 randomized trials in which people were divided by a draw (PMID 29609204).

The yardstick is the minimum difference that patients feel is meaningful. The average reduction in fatigue was about one-third of that yardstick, and even when a plausible range of values was attached, the upper end barely exceeded half of the yardstick. The authors conclude, in light of established criteria, that the number of people whose fatigue decreased enough to be clinically meaningful may not have been large. They also note that more research is needed in people with severe fatigue (PMID 29609204).

The word itself invites confusion. In everyday English, "significant" means "important", so when people call a result with a p smaller than 0.05 by that name, some immediately take it as proof that it is clinically meaningful (PMID 32710113). The Korean word 「유의 (有意)」 literally means 「has meaning」, so it can invite a similar misunderstanding.

The magnitude of the p-value and the magnitude of the difference are different numbers, and effect size tells us how large the difference is.

Where Did That Coin Come From?

Let us return to our friend's coin. Out of 10 flips, 9 came up heads, and the p-value was about 2%. It is tempting to read this as "the probability that this coin is fair is 2%." But the two sentences run in opposite directions. The p-value is "the probability of getting a result like this if it is fair," while what we want to know is "the probability that it is fair when a result like this happened."

Even with the same 9 heads, how we read it changes depending on where the coin came from. If it is a coin freshly swapped at a bank, the chance that it was bent from the start is very low. So 9 heads can be read as a rare coincidence. If it is a coin handed over by a magician, the chance that it was rigged from the start is high, so the story is different. Yet in both cases the p-value is the same, about 2%. That is because the source of the coin does not go into calculating the p-value.

What corresponds to the source of the coin in research is "the plausibility that a real difference exists before doing the study." How high this possibility was is decisive for interpreting the results, and the p-value is not the same as "the probability that the result happened by chance" (PMID 32169596).

One commentary that organizes twelve misconceptions about p-values picks out one belief as the most serious consequence of those mistakes. It is the belief that you can calculate "the probability that this conclusion is wrong" using the data of just a single experiment alone. To figure out that probability, evidence from outside the experiment and whether the rationale (mechanism) for that effect to happen is plausible must be taken together (PMID 18582619).

A review article that outlines the history and misconceptions of p-values also counts believing that the p-value is the probability that the null hypothesis is true as a misconception fostered by the practice of relying on 0.05. At the same time, it concludes that p-values can provide useful information if interpreted properly, just that they should not be used as the sole criterion for drawing conclusions (PMID 41250654).

The p-value is "how rare a result like this is if there is no effect," and its direction is opposite to "the probability that there is no effect given this result."

"Not Significant" and "No Effect"

For this example, let's say there is a coin that was intentionally bent so that it lands on heads 60% of the time. This coin is definitely bent. To flip this 10 times and cross the 0.05 line, either heads or tails needs to come up 9 times or more. Calculating the probability of getting a result like that with this coin is about 5%. That is just a little more than twice the roughly 2% for a fair coin. If you repeat 10 flips 100 times, about 95 of those times will end with "no statistically significant difference."

Even though the coin is bent, the result is usually "not significant." This is because 10 flips is too small a number to catch a 60% bias. "Not significant" only says "this data did not push away the assumption that there is no effect," and "no effect" goes one step further.

The absence of evidence is not evidence of absence. Standard p-value analysis can only argue in the direction of opposing the null hypothesis, and can never argue in the direction of supporting the null hypothesis (PMID 41151754). This means that a large p-value does not build up evidence for the side that says "there is no effect."

However, paper titles and abstracts frequently miss this distinction. The proportion of writing off a nonsignificant result as "no effect" was 84% in titles and 64% in abstracts, and there was not a single title that wrote "not significant" (PMID 36919170). This is the result of manually classifying and counting expressions like this one by one in papers from fields close to animal cognition. In the results section of the main text, expressions like "not significant" were about half.

The authors write that research in this field often lacks the statistical power to catch an effect of a meaningful size. Power here is the probability of catching a real effect when it is there. It is the same situation as flipping a coin bent to 60% only 10 times. The authors advise looking carefully at the basis for claims that there is no effect, and accurately writing nonsignificant results according to the format.

Medicine has examples too. Short counseling sessions to help reduce drinking have had their effectiveness confirmed by pooled analyses combining multiple trials. But because the effect is small, individual randomized trials frequently yield nonsignificant results. Because of those results, people sometimes doubt whether counseling is beneficial in routine clinical practice (PMID 25076917). A review addressing this issue calls treating the absence of a significant difference as evidence of no difference the fallacy of "proving the null hypothesis" (PMID 25076917).

Even when a real effect exists, whether it crosses the line fluctuates. Let's say there are two studies designed so that a real effect exists, and they catch that effect 8 out of 10 times. In this case, conflicting results where one study is significant and the other is not come out about 1 in 3 times (PMID 28698825). Therefore, just because the result of redoing the same study is nonsignificant, you cannot view it as having failed to reproduce the earlier result.

When investigating the side where there is no effect, there are methods that change the question itself. These are methods that ask "Is the difference smaller than a meaningful size?" instead of "Is there a difference?", and these methods can draw conclusions about the null hypothesis directly (PMID 41151754). However, the papers introducing those methods also note their limitations together. No statistical method can prove that the null hypothesis is true (PMID 30873486).

If You Read One Article Headline

"New Treatment, Statistically Significant Improvement" and "Supplement B, No Effect on Memory." Let us imagine encountering two such headlines. Both are hypothetical headlines made up by this article.

The "significant" in the first headline tells you only one thing. If the treatment had no effect at all, a difference this large would rarely appear. How large the difference is, or what the probability is that the treatment really works, is not contained in this phrase. Therefore, the first question to ask is, "How large was the difference?" The answer comes as an effect size, such as "how many people out of 100" or "how many points in score."

The question to ask alongside it is, "How many people were looked at?" A coin flipped 10,000 times crossed the line even at 51.5%, and in 950,000-some surgical records, 14 out of 20 pairs of items matched without medical reason crossed the line (PMID 40885217). In a study that looked at a very large number of people, "significant" can point to a very small difference.

To the second headline, "No Effect," a different question attaches. Did this study fail to find an effect, or was it designed to show that the effect is smaller than a meaningful size? Even a coin weighted to 60% usually returned "not significant" when flipped only 10 times. If the article does not state how many people were looked at or what the study was designed to show, what that headline says is simply, "This study did not find an effect" (PMID 41151754).

In both headlines, "How large was the difference?" does the same job. That answer is not enough with just a single number. Only when you attach the entire range of plausible values together with the effect size can you weigh how large or small the real effect could be, and therefore how important it might be (PMID 29337724). Information on the magnitude of the effect that matches the data must be obtained from a single value and its range (PMID 28698825).

The range makes you reread "No Effect." Even for a study reporting "no effect," if that range stretches all the way to a large effect, that study has failed to rule out either the side that there is no effect or the side that there is. Even in data from three large-scale replication projects retesting already published studies, many of the original and replication studies classified as "null results" because they were not significant were actually cases that could not reach this conclusion (PMID 38739437).

Behind headlines like "Statistically significant" or "No significant difference," the question "How large was the difference?" remains. Effect size and its range answer that question, which p-values do not.

Where to Look Next

This article covered only the meaning of the p-value using a single coin. How sample size, multiple comparisons, and the analyst's choices actually move the p-value in a real paper is continued in "What Is p = 0.03 the Probability Of? — The One Question a p-Value Answers".

Where the line of 0.05 came from, and how to count how many patient events a significant result actually rests on, is looked at in "Where Did 0.05 Come From? — Counting How Many Patients a Finding Rests On". "A 50% risk reduction" and "how many people out of 100," which are used as effect sizes, were covered in "If the Risk Was Cut in Half, How Many Fewer People Got Sick? — Relative and Absolute Risk".

How to read the range attached to an effect size—that is, the rest of the answer to "How big was the difference?"—is continued in "How Much Does a Number Counted Once Wobble? — The Two Numbers in the Parentheses, the Confidence Interval" from Numbers from Scratch. Confounding, the fact that moving together does not mean something is the cause, is looked at in "If People Who Take It Live Longer, How Were They Different to Begin With? — Confounding".

Source Papers

• "Moving Beyond p < 0.05 in Ecotoxicology: A Guide for Practitioners." (2020) — PMID 32539165 — A p-value is the probability of seeing a value equal to or more extreme than the observed one when the null hypothesis is true. In 2019, the executive director of the American Statistical Association recommended retiring the phrase "statistical significance."
• "P-value, compatibility, and S-value." (2022) — PMID 37637018 — The p-value is interpreted as an index of compatibility between the data and a model that incorporates test hypotheses and background assumptions.
• "Semantic and cognitive tools to aid statistical science: replace confidence and significance by compatibility and surprise." (2020) — PMID 32998683 — It recommends re-reading p-values and interval estimates as the compatibility of data with the entire set of analysis assumptions, and teaching "compatibility" instead of "significance."
• "Goodbye P<0.05. P-value is simply one item among many to gauge scientific evidence." (2026) — PMID 41748420 — The threshold of 0.05 used for a century is an arbitrary (yet sanctified) line. Several statisticians and researchers recommend discarding not the p-value itself, but the 0.05 line and the term "statistical significance" (short review).
• "The earth is flat (p > 0.05): significance thresholds and the crisis of unreplicable research." (2017) — PMID 28698825 — At 80% statistical power, even when a true effect exists, two studies will disagree one-third of the time. A non-significant result alone cannot be deemed a replication failure. The p-value should be read as a grade indicating the strength of evidence, and information on effect size must be obtained from point and interval estimates.
• "Reflection on modern methods: Statistics education beyond 'significance'…" (2021) — PMID 32710113 — Because "significant" means "important" in everyday language, it is mistaken as proof of clinical relevance. It recommends teaching how to interpret the exact p-value rather than whether p < 0.05.
• "Statistical significance: p value, 0.05 threshold, and applications to radiomics-reasons for a conservative approach." (2020) — PMID 32157489 — It argues that clinical research requires a binary answer for decision-making, so a conservative approach—namely maintaining the current 0.05 threshold and binary judgments—should be kept. Introducing the proposal to lower it to 0.005, it expresses concern over increased research costs and recommends always reporting the actual p-value.
• "Misleading medical literature: An observational study." (2022) — PMID 34355494 — Across 20 years of JAMA, Lancet, and NEJM, 400 sentences interpreted p-values near 0.05: among 97 sentences above 0.05, 40 were written as if significant; among 303 below 0.05, 4 were written as if non-significant. Misleading descriptions were more common above 0.05.
• "Statistics in medical research: Common mistakes." (2023) — PMID 37234723 — Common statistical errors include omitting corrections for multiple comparisons, and misinterpreting the p-value as a measure of effect size or clinical relevance.
• "Statistical Significance Versus Clinical Importance of Observed Effect Sizes: What Do P Values and Confidence Intervals Really Represent?" (2018) — PMID 29337724 — More than 70 effect size metrics are described. Because test results tell us neither magnitude nor clinical importance, a plausible range of values must be attached to the point estimate as a whole (tutorial).
• "Fully Randomized Predictor-Outcome Pairings Using a National Database Yield Frequent Statistical Significance Without Clinical Meaning." (2025) — PMID 40885217 — Out of 20 pairs randomly matched using 955,092 joint replacement surgery records, 14 pairs (70%) had p < 0.05. The authors conclude this is due to sample size and recommend hypothesis-driven methods and effect size interpretation.
• "Aerobic Exercise and Fatigue in Rheumatoid Arthritis Participants: A Meta-Analysis Using the Minimal Important Difference Approach." (2018) — PMID 29609204 — Data from 5 randomized trials: aerobic exercise significantly reduced self-reported fatigue (p = 0.006), but compared to existing criteria, it is unlikely that a substantial number achieved a clinically meaningful reduction. Further research is needed in individuals with severe fatigue.
• "How to understand and teach P values: a diagnostic test framework." (2020) — PMID 32169596 — The probability that a difference is true before the study is critical for interpretation. The p-value is not equal to the probability of a chance finding (diagnostic test analogy tutorial, also noting the limitations of the analogy).
• "The P Value: What It Is and What It Is Not." (2025) — PMID 41250654 — The 0.05 convention has fostered misunderstandings, such as believing the p-value is the probability that the null hypothesis is true or that significance means clinical importance. Properly interpreted, the p-value can provide useful information, but it concludes that it should not be used as the sole criterion for inference (review).
• "A dirty dozen: twelve p-value misconceptions." (2008) — PMID 18582619 — The most severe consequence among p-value misconceptions is believing that the probability of a conclusion being wrong can be calculated from single-experiment data alone, without external evidence or mechanistic plausibility (commentary).
• "Saying 'no' with confidence: statistical approaches to test for the absence of an effect." (2025) — PMID 41151754 — Standard p-value analyses can only argue against the null hypothesis. Addressing the absence of a meaningful effect through equivalence tests or intervals, or the absence of an effect itself using likelihood ratios and Bayes factors, allows direct conclusions to be drawn about the null hypothesis (guide for biologists).
• "Reporting and interpreting non-significant results in animal cognition research." (2023) — PMID 36919170 — In animal cognition and related fields, the proportion of papers describing non-significant results as "no effect" was 84% in titles, 64% in abstracts, and 41% in results sections, while "not significant" appeared in 0% of titles, 26% of abstracts, and 52% of results sections. This aligns with a distribution rich in low-powered studies.
• "Interpreting null findings from trials of alcohol brief interventions." (2014) — PMID 25076917 — Brief alcohol counseling has an established effect via meta-analysis, but because the effect is small, non-significant results in individual trials are common, which breeds skepticism. It highlights the fallacy of "proving the null hypothesis," which treats the absence of a difference as evidence of no difference (review).
• "Replication of null results: Absence of evidence or evidence of absence?" (2024) — PMID 38739437 — When both an original study and a replication study are non-significant, it does not constitute evidence of no effect. Data from large-scale replication projects show that many original and replication studies with "null results" were actually inconclusive.
• "Making 'null effects' informative: statistical techniques and inferential frameworks." (2018) — PMID 30873486 — How to evaluate null results using equivalence testing, Bayesian estimation, and Bayes factors. No statistical approach can prove that the null hypothesis is true.