Same Trials, Different Conclusions — Publication Bias and What Went Into the Pool

Two meta-analyses gave different answers on whether vitamin D supplements reduce falls in older adults. A meta-analysis is a method that gathers the results of multiple studies tackling the same question and combines them into a single summary value. One meta-analysis stated that giving vitamin D along with calcium reduces falls, and that vitamin D alone has no effect. The other combined overlapping trials yet found no effect whether calcium was given or not.

The point where the two analyses split was the numbers picked from each trial. The raw data of 25 randomized trials (experiments where groups are assigned by a draw, like pulling lots) included in the first analysis were pulled out again, and the treatment effect was calculated and matched for each trial. As a result, in 14 out of the 25 trials (56%), the actual result values of the individual trials differed between the two analyses.

Four trials used only a subset of falls or participants, three used only people who finished to the end, and one used an estimated substitution from fracture data. Beyond these, there were cases where the adjusted results of the original paper were used, the method of combining data from trials with multiple arms differed, and group sizes did not match. The authors conclude that methodological differences in using data from the same trials lead directly to different conclusions (PMID 25093621). This means that even when gathering the same trials, the answer of the meta-analysis changes depending on which numbers are picked and used from each trial.

The previous two pieces covered how to combine and read studies that have already been gathered. "A Meta-Analysis Does Not Take an Average — How the Diamond in a Forest Plot Is Built" showed that the weight assigned to each study by the forest plot (a graph that plots the results of each study as a line and stamps the combined value at the very bottom as a diamond) determines the position of the diamond. "I² Is Not the Size of the Scatter — How to Read Heterogeneity Statistics" dealt with the numbers that measure how scattered the results of studies are from one another. This article tackles a question prior to that: what got into that plot in the first place, and what failed to get in?

The mismatch comes from two directions. On the outside, studies that were never published and thus nobody can include are missing. On the inside, authors put data in, take data out, or use it differently. And there is one more ingredient spanning both of these directions: who funded the research.

There are seven new terms to learn in this article, and each term is explained where it first appears. The names of medicines and treatments appearing below are examples to show how to read numbers, not a judgment on which treatment is good.

---

Distortion from the Outside — Research That Was Never Published In the First Place

A Plot That Tries to See What Is Missing

Publication bias is a phenomenon where studies with lackluster results are less likely to be published, causing the studies that end up in our hands to skew toward positive results. Sometimes authors choose not to write them up, sometimes they write them up but journals refuse to publish them, and sometimes they are published late or in less visible places. When this happens, no matter how diligently someone searches later on, the only studies that end up in their hands are the survivors.

Translated to a fishing tournament, it works like this. If participants only post photos when they catch big fish, counting every single one of those photos and taking the average will still not give you the true fish size in that lake. Each and every photo is real. The problem is the photos that were never uploaded.

Since you cannot see the studies that do not exist, you look at the shape that the existing studies form. A funnel plot is a graph where each study is plotted as a single dot, showing how the studies are scattered. The horizontal axis represents the effect size, which is a value indicating how much the treatment changed the outcome. The vertical axis represents precision, meaning how little the result fluctuates, and sample size is sometimes used instead. Precise studies cluster near the true value so they gather in the narrow upper space, while small studies fluctuate widely so they spread out wide at the bottom. Therefore, if there is no bias, the dots form the shape of an inverted funnel. If one of the lower corners of the funnel is empty, you ask where the small and lackluster studies that should have been in that spot went.

The paper that reviewed the uses of this plot attaches a caveat. It serves as a useful test to gauge whether bias is likely to be present based on simple analysis alone, but when a meta-analysis is based on a small number of small trials, its detection power is limited, so its results should be interpreted with considerable caution (PMID 10724898). This means that when studies are few and small, it is difficult to reach a conclusion from a single plot.

Smaller Studies, Bigger Effects

When a funnel tilts to one side, a common sight emerges: the smaller the study, the more the effect, with smaller studies tending to report larger benefits. Small studies that turn up lackluster results are easily buried in silence, while large, expensive studies tend to get published regardless of their findings. The small studies left in the literature then become a sample consisting only of the ones that "turned out well." But that is not the only cause. Small studies tend to have looser methods, and that looseness can inflate the effect.

This trend was observed in trials of osteoarthritis pain treatments. On average, the treatment effect was larger in smaller trials (difference in effect size −0.21, 95% confidence interval −0.34 to −0.08). A confidence interval is a range showing how much an estimate can fluctuate, and here the entire interval lies on the side favoring smaller trials. The analysis included 153 trials across 13 meta-analyses, involving 41,605 patients.

In 6 of the 13 papers, the combined pooled estimate suggested a clinically meaningful and significant benefit. Yet recalculating using only large trials yielded a smaller, non-significant value. The authors conclude that the small-study effect can often distort the results of meta-analyses (PMID 20639294). This means that even if a combined value says there is a "benefit," if that benefit came from small trials, it may vanish when looking only at large trials.

Besides the tilt of the funnel, there is another signal that arouses suspicion of publication bias. It is the excessive frequency of significant results. This is the other of the two items that "A Meta-Analysis Does Not Take an Average" left off by merely noting as "two signs of bias." "Significant" here means that the p-value came out smaller than a pre-determined threshold. Suppose ten studies each have only about half the statistical power to catch an effect. Even if an effect truly exists, the number of studies coming out as "significant" should be roughly five. Yet if nine studies come out significant, the math does not add up. It is a signal that studies turning out non-significant have vanished somewhere.

But Some Fields Don't Show It

The small-study effect does not appear everywhere. In neurosurgery meta-analyses, on average, smaller studies did not show greater benefit. The difference was measured by dividing the odds ratios (the ratio of the probability of an event happening to the probability of it not happening, compared between two groups) of the two sides, and that value was 1.32 (95% confidence interval 0.89 to 1.75). The analysis included 16 meta-analyses from five neurosurgery journals and 90,629 patients.

The authors did not read this result as "therefore, there is no publication bias." Their interpretation is this. The absence of a small-study effect may reflect a widespread drop in quality in the neurosurgery literature across both large and small studies, rather than the absence of publication bias (PMID 34656237). If large studies are also shoddy, the difference between large and small studies is erased. Not catching a signal is different from having no problem.

The Causes of Asymmetry Are Not Just Publication Bias

Unevenness between the left and right sides of the funnel is called asymmetry. Seeing asymmetry does not immediately allow us to read it as "papers were buried." There are several other causes.

First, the shape of the funnel depends on how the graph is drawn. Simply redefining the precision on the vertical axis or changing the metric measuring the effect alters the shape. When 198 published meta-analyses were redrawn this way, conclusions about the shape flipped in 37 of the 43 papers (86%) that had looked asymmetrical and suggested selection bias (bias stemming from certain studies being selectively left behind). The authors conclude that the shape of a funnel plot heavily depends on arbitrary choices in how the plot is constructed (PMID 10812319).

Second, asymmetry occurs even without bias in certain effect metrics. Outcomes measured in numbers, such as pain scores, use different scales across studies, so they are converted into standardized effect sizes with aligned units and combined. Plotting this standardized effect size against the standard error (a value indicating how much an estimate fluctuates) easily distorts the graph, which consequently overestimates the presence and degree of publication bias. The distortion was more severe when samples were small and the intervention effect was actually real. The authors conclude that this combination is unsuitable for evaluation and can lead to false-positive results where a non-existent bias is seen as present (PMID 28884685).

Third, the list of causes is not closed. Most statistical tests cannot rule out other explanations that create asymmetry. These are explanations such as heterogeneity (the degree to which results actually differ across studies), the choice of effect metrics, and chance (PMID 37684502). This evidence is an educational review of sports and exercise medicine literature.

Plots attempting to separate the causes have also been proposed. This is a method of overlaying contour lines on the funnel plot that divide the regions where results come out statistically significant from those that do not. If the apparently missing studies lie in the non-significant region, weight leans toward publication bias. If they lie in the region of high significance, other factors such as variation in study quality are more likely (PMID 18538991). This means that the region where the empty space is located serves as a clue to distinguishing the cause.

Neither Humans Nor Tests Can Fill This Void

The accuracy of visually reading a funnel plot was little better than a coin flip on average. Forty-one medical researchers were given funnel plots of 10 studies each and asked whether they were affected by publication bias. Ten studies is the median number of studies included in a medical meta-analysis. The average accuracy rate was 52.5% (95% confidence interval 50.6–54.4). Weighted accuracy rates also hovered between 48.3% and 62.8% depending on the conditions (PMID 16085192). This means that judging by the shape of the plot alone can lead to being fooled by that shape.

Statistical tests also weaken when the number of studies is small. Across 5,014 meta-analyses from Cochrane (an international collaborative network that systematically collects and organizes medical evidence), when the median number of studies dropped from 87 to 14, the rate at which four widely used asymmetry tests detected asymmetry fell by 28.3–42.6% depending on the test. In simulations as well, when publication bias was present, p-value-based tests underestimated its presence, particularly when the number of studies was small (PMID 31743750). This means that just because a test fails to find asymmetry is no guarantee that no bias exists.

Discrepancies in verdicts between different tests are also a problem. Across 28,655 Cochrane meta-analyses, the degree of agreement between test verdicts was measured. For binary outcomes (outcomes measured as whether something happened or not), four regression tests agreed quite strongly with each other (mostly around 0.6, with two exceeding 0.9). On the other hand, the agreement between three tests of different types—namely, Begg's rank test, the trim-and-fill method (covered below), and Egger's regression test—was below 0.5. Therefore, the authors advise against relying on a single test (PMID 29663281).

Consequently, the earlier educational review selects prevention over detection as the best solution. There are four preventive measures: pre-registration, which publicly discloses what and how things will be measured in advance; registered reports, which undergo peer review based solely on the plan before results are obtained; the disclosure of deviations from the plan (protocol deviations); and reporting all results regardless of their direction or magnitude. Because true publication bias is very difficult to confirm, the authors recommend using the term "risk of publication bias" rather than "publication bias" (PMID 37684502).

And Even Shakes Heterogeneity Statistics

The representative numbers used to measure heterogeneity are two. The Q test is a test that sees whether the results of the studies are more scattered than can be explained by chance, and $I^2$ is a value representing the proportion of the scatter in results that comes from true differences rather than chance as a percentage. The effect of publication bias on these two is large, complex, and non-linear. Publication bias can either dramatically reduce or increase the heterogeneity of the true effect size, especially when the number of studies is large and the population effect size is small. The authors conclude that when publication bias is present, neither the Q test nor heterogeneity measures including $I^2$ are generally valid (PMID 30489099). The reading method established in "I² Is Not the Size of the Scatter — How to Read Heterogeneity Statistics" remains entirely valid. However, even that number is a value calculated on top of a vacant seat.

Adjusting Changes the Answer — and in Both Directions

Calculations That Fill in What Is Missing

There are also methods that try to fill in the empty spaces through calculation. One approach is to estimate and fill in studies that appear to be missing and then recalculate the summary value, while another builds a model where the probability of being published differs depending on the result and reweights accordingly. This article refers to these methods of using calculations to fill in the share missing due to publication bias and recalculate the summary value as adjustment tools. The widely used trim-and-fill method estimates the number and impact of missing studies and fills in the empty side of the funnel. The paper proposing this method notes that after adjustment, the point estimate is approximately correct and the proportion of times the confidence interval captures the true value (coverage probability) improves considerably (PMID 10877304).

Adjusting changes the answer. The problem is that which way it will change is not fixed. The two cases below show opposite directions.

The Direction Where the Effect Disappears

There is a meta-analysis addressing the association that lower blood vitamin D is linked to a higher risk of hypertension. It combined 13 cohort studies that followed groups over time. The combined relative risk (the risk of two groups divided by each other) was significant at 1.22 (95% confidence interval 1.05 to 1.41). However, the p-value of Egger's test, which measures funnel asymmetry, was 0.015, and the funnel plot was also asymmetric. When adjusted using the trim-and-fill method, the filled value became 1.03 (0.89 to 1.18), and statistical significance disappeared. The authors' conclusion includes a qualifier. It states that significant results from previous systematic reviews can be interpreted as being due to publication bias (PMID 35568961). A systematic review is a study that uses a set method to find and gather all related studies without omission.

The Direction Where the Effect Grows Larger

The opposite direction has also been measured in practice. In a meta-analysis of 25 randomized trials addressing whether exercise treats depression, the standardized effect size adjusted for publication bias was large and significant at 1.11 (95% confidence interval 0.79 to 1.43). A calculation also showed that it would take 1,057 studies with no effect to make this effect non-significant. And most adjustment analyses suggested that publication bias made the effect size underestimated. The authors conclude that previous meta-analyses may have underestimated the benefits of exercise due to publication bias (PMID 26978184).

What Adjustment Tools Can and Cannot Do

Selection models are adjustment tools that assume the probability of being published varies depending on the result, and reweight based on that probability. Evaluation of this family through simulation studies yielded mixed results. Two recently proposed approaches worked fairly well in the limited scenarios where all three methods were designed to work, though not as well as the original selection model. They performed poorly in more realistic situations, whereas variants of the original approach performed well. Therefore, the authors recommend using selection models for sensitivity analysis rather than producing a single post-adjusted estimate. Sensitivity analysis is the process of examining the range of estimates that emerge when different shapes and severities of bias are assumed (PMID 27694467). Rather than accepting a single adjusted value as the right answer, the recommendation is to look at how much the value moves depending on the assumptions.

The fact that there are multiple methods creates yet another problem. In simulation studies, no single method consistently outperformed the others across a broad range of conditions. Consequently, when methods yield conflicting conclusions, one could cherry-pick the method that produces the desired result. The paper raising this issue offered a solution: instead of choosing one, average multiple approaches with weights supported by the data. And it demonstrated this benefit through real data application, simulations, and comparison with pre-registered replication studies (PMID 35869696).

When Unpublished Data Was Actually Added In

If neither detection nor correction is reliable, the remaining path is to directly obtain and insert unpublished data. A systematic review covering the antidepressant reboxetine did just that. It gathered 13 trials with 4,098 patients comparing reboxetine against placebo and selective serotonin reuptake inhibitors (SSRIs). The data was obtained directly from trial registries, regulator websites, and the manufacturer. Data for 74% (3,033) of these 4,098 patients was unpublished.

There was no significant difference in remission rates (the proportion of patients reaching a state where symptoms have almost disappeared) compared with placebo. Reboxetine was inferior to SSRIs in both remission and response rates (the proportion of patients whose symptoms decreased beyond a certain level), and inferior to placebo in both harm outcomes. And if only published data was used, the benefit compared to placebo was overestimated by up to 115% and compared to SSRIs by up to 23%, while harms were underestimated. The authors conclude that reboxetine is generally ineffective and potentially harmful as an antidepressant, that the published evidence is affected by publication bias, and that mandatory disclosure of trial data is urgently needed (PMID 20940209). This means that if only published papers had been collected, this drug's benefits would have looked larger than reality, and its harms smaller.

Distortions from Within — What Was Put In and Left Out

Cherry-Picking Within a Single Study

Up to now, we have looked at cases where entire studies never see the light of day. Even when a study is published, the results within it can be cherry-picked. Selective reporting means publishing a study but choosing to include only some of the measured outcomes. For instance, ten things might be measured, but only three favorable ones appear in the paper, or instead of a pre-specified primary outcome, a result that turned out well later becomes the star. Because the trial sits safely on a publication list, it cannot be caught by a search.

Along with publication bias, well-known causes of small-study effects include result reporting bias arising from this selection, and clinical heterogeneity (differences in target patients or treatment methods across studies). Different selection processes—some studies failing to be published as a whole and others reporting only some of their results—can operate together (PMID 32720712). However, the literature used as evidence for this article merely names the phenomenon and categorizes its mechanisms, without counting how often it occurs. Preventive measures like the pre-registration we saw earlier are precisely what target this space.

When You Cut It Off — Cumulative Meta-Analysis

A meta-analysis is a snapshot gathering studies published up to a certain point in time. Cumulative meta-analysis does not fix that point in time, but instead adds trials one by one in the order they were published, recalculating the summary value each time. Doing this shows when the answer settled and until when it fluctuated.

The myocardial infarction risk of the diabetes drug rosiglitazone was re-examined using this method. Cumulative meta-analyses were run separately on data using only published studies and data using all studies that were conducted. The final summary estimates were not noticeably different between the two sets of data (odds ratio 1.4, 0.95–2.05 / 1.42, 1.03–1.97). However, the all-data side reached the decision-making threshold — in other words, the benchmark sufficient to make a clinical judgment — 36 months earlier. Three correction methods were applied to the published-data set, but none of them brought that time point forward (PMID 35761433). This means that the price of publication bias may be time rather than the final answer, and correction tools cannot buy back that time.

Same Topic, Different Ingredients

Meta-analyses on vitamin D and fractures also used different ingredients. This is the result of a detailed comparison of seven papers published in top-tier general medical journals out of 24 meta-analyses on this topic. Viewed through quality assessment tools, the seven papers were generally of high quality. But looking closer revealed important differences in trial selection, data extraction, and analysis methods. Three papers "missed" 3 to 8 trials that met their criteria, and two included trials that appeared ineligible.

These differences skewed in one direction. The comparison benchmark is the value analyzed without adjustment for all randomly assigned participants (unadjusted intention-to-treat analysis). The majority of the differences (11 out of 16) led to estimates more favorable to vitamin D than this benchmark. The conclusions were also scattered, ranging from "vitamin D prevents fractures" to "vitamin D without calcium does not prevent fractures" (PMID 25551377).

Not only what was included and excluded, but also how bias was handled, divides conclusions. Eleven overlapping meta-analyses examining whether the effect of the antiplatelet drug clopidogrel is related to loss-of-function variants in the gene CYP2C19 serve as that example. The reason the result values differed was that more recent meta-analyses included more primary studies, and some omitted conference abstracts. On the other hand, the reason the conclusions differed was that they handled heterogeneity and publication bias differently. Across clinical endpoints, all meta-analyses observed significant heterogeneity, and several reported evidence of publication bias. Yet among the eight papers reporting a significant association, only one concluded that the association was therefore unproven (PMID 24946154). This means that even after reporting the same warning signals, how to draw the conclusion differed from one meta-analysis to the next.

A Case Where It Was Actually Solved

If you reveal the source of the disagreement and align the methods to recalculate, the disagreement can be resolved. That is the case with the vitamin D and fall example that opened this article. When recalculated by unifying the methodological choices to the unadjusted analysis and all randomly assigned participants, vitamin D showed a barely catching effect on falls regardless of calcium (relative risk 0.95, 0.90–1.00). This means the upper end of the confidence interval touches the boundary of 'no effect.'

When broken down into subgroups, conditions are attached. Vitamin D alone had no effect (1.00, 0.92–1.08). Comparing vitamin D and calcium with a placebo or control group also showed no effect (0.95, 0.88–1.02). The modest effect remained only when vitamin D and calcium were compared with calcium alone (0.84, 0.76–0.92) (PMID 25093621). Unifying the methods reduces the disagreement, but such conditional clauses attach to the remaining effect.

Funding Source — Who Paid for the Research

The final ingredient is who paid for the money. This ingredient spans both the outside and the inside. Funding bias is the tendency for results and conclusions more favorable to the funder to appear more often.

When gathering 48 papers covering drug and medical device studies, industry-sponsored research produced results favorable to the sponsor more frequently. Here, the risk ratio is the value obtained by dividing the rate at which favorable results appeared in industry-sponsored research versus other research. For efficacy results, it was 1.24 (95% confidence interval 1.14 to 1.35), for harm results 1.87 (1.54 to 2.27), and for conclusions 1.31 (1.20 to 1.44).

However, there was no difference in risk of bias assessments. Risk of bias assessment is the process of checking item by item whether a study design has weaknesses that could skew the results. In the blinding item (making it so people do not know which treatment they are receiving), industry sponsorship actually had a lower risk of bias more often (1.32, 1.05 to 1.65). The authors' conclusions stand on top of that. Manufacturer sponsorship leads to more favorable results and conclusions, suggesting that an industry bias exists which is not explained by standard risk of bias assessments. However, the same review also shows less balance. Out of ten papers reporting effect sizes, five showed larger effects for industry-sponsored research, and five found no difference (PMID 23235689).

The size of this trend varies by field. A systematic review that asked the same question in nutrition yielded a different answer. It reviewed 775 papers and included 12. Two papers looking at the association with statistical significance both found no association. In eight papers (340 studies) looking at the association with conclusions, industry-sponsored research was more likely to reach favorable conclusions, but the difference was not significant (risk ratio 1.31, 0.99 to 1.72). The authors conclude that this result suggests an association between food industry research sponsorship and favorable conclusions, but does not confirm it (PMID 27802480).

The direction is there. The magnitude depends on the context.

The Two Misunderstandings This Article Addresses

① "The funnel is symmetrical, so there is no publication bias." Symmetry is not evidence that there is no publication bias. When there are around ten or so studies, the test severely lacks statistical power—that is, the ability to catch a bias that actually exists (PMID 37684502). The test underestimates the presence of bias (PMID 31743750), and the accuracy rate of visual judgment hovers around half (PMID 16085192). In neurosurgery, when there was no signal, the judgment made by the authors was also not "there is no publication bias" (PMID 34656237). Conversely, asymmetry does not mean publication bias either (PMID 10812319, PMID 28884685).

② "This result has been repeatedly confirmed in multiple meta-analyses." When the same trials are repeatedly used across multiple meta-analyses, the evidence can look like multiple independent pieces. If those multiple papers reach different conclusions, that difference may come from how the materials were chosen and handled rather than from new data (PMID 25551377, PMID 24946154).

Questions to Ask Before the Summary Value

When you encounter a diamond or a single line stating "confirmed in a meta-analysis," there are four questions to ask.

1. On top of which trials was this value produced? Is it written how each result was extracted from each trial and in what way (PMID 25093621).
2. What did the authors write that they confirmed with a funnel plot or asymmetry test? Almost nothing is confirmed by that alone (PMID 31743750, PMID 37684502).
3. Does it state that unpublished data were searched for (PMID 20940209).
4. If the conclusion differs from other meta-analyses, did they clarify whether the difference is due to the data or the methods (PMID 24946154).

If you leave behind just one thing, what determines the conclusion of a meta-analysis is the ingredients, not the calculation. If you do not ask what got in and what could not get in, the diamond cannot be read.

The Gap This Article Left Open — and Closing the Series

These flaws are not a reason to abandon evidence synthesis. Systematically gathering and summarizing available evidence remains the foundation of evidence-based practice. What remains is a single unresolved issue: true publication bias is very difficult to confirm (PMID 37684502). Therefore, what this article can give you is not "it is correct if you adjust for it," but rather "you can just ask what it was calculated upon and read it with that in mind."

And with this article, this series comes to a close. We have repeatedly asked one question. What is this number a number of? In "What Is p = 0.03 the Probability Of?", it was not the probability that the hypothesis is true, and in "A Confidence Interval Is Not a Margin of Error", the two numbers inside the parentheses were not the probability that the true value lies there. In "An Odds Ratio Is Not a Risk Ratio", "how many times" was not how many times the risk, and in "What "Adjusted for Age, Sex and BMI" Actually Did", adjustment did not manufacture a cause. In "A Meta-Analysis Does Not Take an Average", the diamond was not an average, and in "I² Is Not the Size of the Scatter", I² was not the size of the scatter. And in this article, the combined value was not the value of all the evidence.

Not being afraid of numbers does not mean believing in numbers. It means being able to ask what of, that number is a number.

Source Papers

Same Trial, Different Conclusions

• "Differences in o…" (2014) — PMID 25093621 — When the raw data were recalculated, 14 out of 25 papers (56%) showed different result values. Reanalyzing with no adjustment and all participants shows an effect that barely reaches falls (RR 0.95, 0.90–1.00). In subgroups, no single effect (1.00, 0.92–1.08), and calcium combined with placebo/control also showed no effect (0.95, 0.88–1.02), while only calcium combined with calcium alone showed a modest effect (0.84, 0.76–0.92).
• "A case study of…" (2014) — PMID 25551377 — A sample of 7 top general medical journals. Three missed 3 to 8 trials matching their own criteria, and 2 included ineligible trials. The majority of relative risk differences (11 out of 16) favored vitamin D.
• "A systematic rev…" (2015) — PMID 24946154 — CYP2C19 and clopidogrel. Differences in results came from the number of included studies and whether conference abstracts were used, while differences in conclusions came from handling heterogeneity and publication bias differently. Out of 8 that were significant, 1 was called unproven.
• "Adjustment of pu…" (2022) — PMID 35761433 — Rosiglitazone. Final estimates were not noticeably different (1.4 / 1.42), but the total data reached the threshold 36 months earlier, and none of the 3 adjustment methods brought that timing forward.

Funnel Plots and Small Study Effects

• "Bias in meta-ana…" (1999) — PMID 10724898 — Plotted sample size against effect size, it can be used to investigate bias (it is not a proposed paper). Although it is a useful test with simple analysis alone, when based on a small number of small trials, detection power is limited, requiring considerable caution.
• "Small study effe…" (2010) — PMID 20639294 — 13 meta-analyses, 153 trials, 41,605 patients. Effects were larger on the side of small trials (difference −0.21, −0.34 to −0.08). In 6 out of 13 reviews, using only large trials lost significance. Often can distort.
• "Absence of small…" (2021) — PMID 34656237 — 16 meta-analyses, 90,629 patients. With a ratio of odds ratios of 1.32 (0.89 to 1.75), smaller studies did not show larger benefits. The authors' conclusion is that this absence may reflect not the absence of publication bias, but a widespread low quality of literature spanning both large and small studies.

Causes of Asymmetry and Limits of Detection

• "Misleading funne…" (2000) — PMID 10812319 — 198 meta-analyses. When precision definitions or effect metrics were changed, conclusions flipped in 37 out of 43 papers (86%) that had been asymmetric.
• "Standardized mea…" (2017) — PMID 28884685 — Plotting standardized mean differences against standard errors distorts the plot and overestimates publication bias. This was worse when samples were small and intervention effects were present.
• "Contour-enhanced…" (2008) — PMID 18538991 — Adds significance contours. If missing studies fall in non-significant areas, weight shifts toward publication bias; if they fall in highly significant areas, it shifts toward other factors like variation in study quality.
• "Testing small st…" (2020) — PMID 32720712 — Lists publication bias, outcome reporting bias, and clinical heterogeneity as well-known causes of small-study effects, noting that some studies are unpublished entirely while others selectively report subset results. It does not count frequencies.
• "The Perils of Mi…" (2024) — PMID 37684502 — Educational review. Visual reading is unreliable, most tests fail to rule out alternative explanations, and statistical power is severely lacking even at the arbitrary threshold of ten or more studies. The prescription is prevention and writing "risk of publication bias".
• "In an empirical…" (2005) — PMID 16085192 — 41 researchers were given funnel plots with 10 studies. Mean accuracy was 52.5% (50.6–54.4%), and weighted averages ranged from 48.3% to 62.8% depending on conditions.
• "Empirical Compar…" (2018) — PMID 29663281 — Seven tests across 28,655 meta-analyses. For binary data, agreement among the Tang, Macaskill, Deeks, and Peters regression tests was fairly strong (κ usually 0.6, over 0.9 for Tang and Deeks), while agreement between Begg, trim-and-fill, and Egger was weak to moderate (κ < 0.5). Do not rely on a single test.
• "P value-driven m…" (2020) — PMID 31743750 — 5,014 meta-analyses. When the median number of studies dropped from 87 to 14, detection rates decreased by 42.6% for Egger, 41.1% for Harbord, 29.3% for Peters, and 28.3% for Begg. Particularly when the number of studies is small, bias is underestimated.
• "The effect of pu…" (2019) — PMID 30489099 — Derives expected values for H² and I² and examines the Q test via Monte Carlo. Publication bias can dramatically decrease or increase true heterogeneity. The Q test and H² and I² are generally not valid.

Adjustment, and Actually Plugging In Unpublished Data

• "Trim and fill: A…" (2000) — PMID 10877304 — A rank-based data augmentation technique that estimates the number of missing studies and their impact. After adjustment, the point estimate is approximately correct and the confidence interval coverage probability improves.
• "Circulating 25-h…" (2022) — PMID 35568961 — 13 cohort studies. 1.22 (1.05–1.41) was significant, but Egger p = 0.015 and asymmetry. After adjustment, 1.03 (0.89–1.18), with the significance lost. Can be interpreted as due to publication bias.
• "Exercise as a tr…" (2016) — PMID 26978184 — 25 randomized trials. Adjusted standardized effect size 1.11 (0.79–1.43), which is large and significant, with 1,057 null studies needed to wipe out the significance. Suggests that most adjustment analyses made publication bias underestimate the effect size.
• "Adjusting for Pu…" (2016) — PMID 27694467 — Evaluation of selection model simulations. The two recent approaches performed fairly well in limited situations and poorly in realistic situations (a variation of the original approach performed well). The recommendation is to use them for sensitivity analyses.
• "Robust Bayesian…" (2023) — PMID 35869696 — A solution-proposing paper. Building on the problem that no single method is consistently superior across all conditions, it bundles the two families together via Bayesian model averaging and demonstrates gains through application, simulation, and preregistered replication comparisons.
• "Reboxetine for a…" (2010) — PMID 20940209 — Out of 4,098 patients, 74% (3,033 patients) were unpublished. No difference in remission rate compared to placebo, and inferior to SSRIs (0.80 each). Using only published data overestimates the difference versus placebo by up to 115% and versus SSRIs by 23%, while underestimating harms.

Funding Source

• "Industry sponsor…" (2012) — PMID 23235689 — A Cochrane methodological review of 48 papers. Industry sponsorship is associated with favorable efficacy (1.24), harm (1.87), and conclusions (1.31). The risk of bias showed no differences in sequence generation, allocation concealment, or follow-up, and for blinding, the industry-sponsored side was actually lower (1.32, 1.05–1.65). The authors' conclusion is that industry bias exists that is not explained by standard risk of bias assessments. Among 10 papers that looked at effect size, 5 showed larger effects for industry sponsorship and 5 showed no difference.
• "Association of I…" (2016) — PMID 27802480 — Reviewed 775 papers, included 12. Two papers that looked at association with significance both found no association. In 8 papers (340 studies) that looked at association with conclusions, industry sponsorship had a higher likelihood of favorable conclusions, but the difference was not significant (1.31, 0.99–1.72). The authors' conclusion is that it suggests but does not confirm.