A Meta-Analysis Does Not Take an Average — How the Diamond in a Forest Plot Is Built

Does exercise help cognitive function in older adults with cognitive impairment? When you gather 40 meta-analyses addressing this single question in one place, the evidence is thinner than it looks. A meta-analysis is an analysis that pools the results of multiple studies addressing the same question into a single combined result. If you count all the randomized trials cited as evidence across those 40 papers, there are 728 instances. A randomized trial is an experiment where participants are split into two groups by a random draw. However, when counting each unique trial only once, there are just 276 trials (37.9%). 153 trials appeared in only a single meta-analysis, while 123 trials appeared repeatedly across 2 to 19 different meta-analyses. Ultimately, 452 out of the 728 instances (62.1%) were overlaps—meaning trials that were included across more than one meta-analysis.

The authors write in their conclusion: when the same trials are reused across multiple meta-analyses, it creates an illusion of robustness—an illusion that the evidence is stronger than it actually is—and the strength and diversity of the evidence may actually be more limited (PMID 40253011). This means that simply having multiple meta-analyses does not mean there is an equivalent amount of independent evidence.

A similar pattern appears in other questions. When you gather 21 systematic reviews addressing whether or not to surgically treat proximal humeral fractures—breaks at the upper end of the arm bone near the shoulder—there are actually only 8 randomized trials published on this topic. A systematic review is an article that gathers and reviews studies comprehensively using a predefined procedure, which is explained in more detail below. Of the 21 reviews, 18 received the lowest grade in methodological quality assessment, and just 1 received the highest grade. The conclusions conflicted across all three outcome domains: function, quality of life, and harms. This was true even among reviews that wrote down similar inclusion criteria. However, what the authors noted in their conclusion was that the quality was so uniformly low that it was impossible to evaluate the association between quality and conclusions itself (PMID 34718123). Therefore, the conflicting conclusions could not be explained by differences in quality either.

Meta-analyses themselves have also grown extremely rapidly. From 1991 to 2014, the annual publication volume increased by 2,728% for systematic reviews and 2,635% for meta-analyses. Over the same period, total papers published in PubMed increased by 153%. Thus, this investigation notes that by now, systematic reviews pooling randomized trials probably outnumber the newly published randomized trials each year. Meta-analyses on the same topic sometimes exceed 20 papers. The authors' final verdict includes a hedge: the majority of systematic reviews and meta-analyses being produced are likely unnecessary, misleading, or entangled with conflicts of interest (PMID 27620683).

Therefore, we need a way of reading that does not swallow a single line stating "confirmed in a meta-analysis" whole. This article covers how to read a single picture produced by a meta-analysis: the forest plot. It does not cover how to perform a meta-analysis yourself.

There are seven new terms to learn, and each term is explained where it first appears. Confidence intervals were already covered in "How Much Does a Number Counted Once Wobble? — The Two Numbers in the Parentheses, the Confidence Interval" and "A Confidence Interval Is Not a Margin of Error — How to Read the Two Numbers in the Parentheses", so they will only be revisited in a single sentence where they are used. The treatment and surgery names appearing below are all examples to demonstrate how to read the plots and are not judgments on which treatments are good.

---

The Collection Procedure and the Combination Calculation

A meta-analysis is a method that, when there are multiple studies asking the same question, statistically combines their results to produce a single summary value. The name sounds difficult, but that is all it does. The goal is to obtain a value that can better represent the true value of the population than any single individual study could. The true value of the population refers to the real value across all the people the study wants to know about. Various types of statistics, such as means, proportions, risk ratios, and odds ratios, can be combined in this way (PMID 33027562).

However, there is something to decide before combining them: out of the studies published in the world, which ones to collect. A systematic literature review is the process of gathering studies to be combined comprehensively through a defined procedure. Search terms and inclusion and exclusion criteria are written down in advance, and multiple databases are searched according to a set procedure. Then, the filtered studies are evaluated for the risk that they are skewed in one direction due to their design.

If you reduce the relationship between the two terms to a single line, it is this: a systematic literature review is the collection procedure, and a meta-analysis is the calculation that combines what was collected. You can also do a literature review alone without combining them. The steps of a meta-analysis also follow this order. A systematic literature search is the first step, and then risk of bias evaluation, data extraction, and data pooling follow (PMID 39239131).

What Meta-Analysis Actually Does — It Weights Each Study

Let's take three studies as an example. The participants are 40, 400, and 4,000, and the values reported by the three studies are −0.6, −0.2, and −0.1. (These six numbers are made up by us for the sake of explanation. In this example, let's assume that the more participants a study has, the less its value wobbles.) The simple average of the three values is −0.3. However, a meta-analysis does not output this −0.3 as the summary value.

This value produced by each study is called the effect size. The effect size represents how large the effect found by that study is in a single number.

Meta-analysis weights each effect size. The weight is a number that determines how much of that study's value to reflect when creating the summary value. The standard way of assigning weights is inverse-variance weighting. Variance is a number that measures how much the value produced by that study wobbles, in other words, the magnitude of the wobble. Inverse-variance weighting uses the inverted value of this magnitude of the wobble, in other words, the reciprocal, as the weight. Therefore, a study with a small wobble becomes heavy, and a study with a large wobble becomes light. Although the two models that will appear later have different assumptions, both use this inverse-variance method to create weights (PMID 34550583).

Therefore, the study with 4,000 participants is much heavier than the one with 40 participants, and the summary value is pulled heavily toward −0.1 rather than −0.3. This weight distribution determines what the conclusion of the meta-analysis will be. Deciding which studies to include and exclude determines the ingredients, and the weight distribution determines the result that will come out of those ingredients.

The Forest Plot — A Picture That Lets You See Weight

A forest plot is a single chart that shows each study's effect size, weight, and the combined summary value. The results of a meta-analysis are presented using this plot (PMID 33027562).

Here is what the plot looks like. Studies are stacked in rows vertically. Each row features a square box, with horizontal lines stretching like whiskers to the left and right of the box. The horizontal position of the box is the effect size of that study. By convention, the size of the box is the weight assigned to that study. The length of the whiskers is the confidence interval. The confidence interval is the range written as two numbers in parentheses in the body of a paper, representing the range of values compatible with this data. The shorter the whiskers, the more precise that study's value. A single vertical line marks the position of "no difference." At the very bottom, the summary value combining all rows is marked as a diamond, and the left-to-right width of the diamond is the confidence interval of the summary value.

When you encounter this plot for the first time, the most useful habit is to look at the box sizes rather than the number of rows. Twelve rows do not mean twelve pieces of evidence. If one box is larger than the other eleven combined, a single study has taken more than half of the weight. Then the diamond is pulled heavily toward that study's value.

Even the Result of a Single Study Is Drawn as a Forest Plot

Forest plots were initially used to show meta-analysis results, but today they are also used to show the results of individual studies. As long as a confidence interval (usually 95%) can be calculated, they can be drawn for categorical variables as well as continuous variables. Not only analyses with a single variable but also the results of analyses incorporating multiple variables together are drawn using this plot. Forest plots are one of the most widely used graphs in recent literature, and their popularity is related to this broad utility (PMID 41774848). There are also proposals to use this plot instead of tables to draw the results of statistical models that jointly estimate the contributions of multiple factors. When tables grow long, they become tedious to read and difficult to interpret. Therefore, three visualizations have been proposed to lay out multiple models in a single plot (PMID 38306346).

For the reader, this is a trap. If you look at a plot lined with rows and whiskers and guess that it is "a combination of multiple studies," you might be wrong. There are two ways to tell. Look to see if there is a diamond at the very bottom. And look to see whether the label for each row is the name of a study or the name of a variable.

When Studies Multiply, This Plot Breaks Down

In ecology and evolutionary biology, it is common for a single meta-analysis to have over 100 effect sizes. When there are that many rows, the classical forest plot loses much of its usefulness. In fact, among 102 meta-analyses in this field, only 11% used a classical forest plot. Most used a "forest-plot-like plot" that plots point estimates by subgroup or category. The plot proposed by the authors as an alternative plots individual effect sizes together with sizes proportional to precision. The idea is the same as in a forest plot where the box size represents weight. This plot attaches one more interval, and what that interval answers is covered in "I² Is Not the Size of the Scatter — How to Read Heterogeneity Statistics" (PMID 32445243).

Who Can Read the Same Plot?

Because statistical plots are used widely, who reads them also becomes an issue. The same plot does not tell everyone the same thing. In 2013, the UK published risk-adjusted performance scores for general surgeons for the first time. Risk-adjusted performance scores are surgical outcomes adjusted to account for differing patient risk levels. An experiment was conducted where a plot of the format used at that time was shown to 40 people (10 each of the general public, medical students, non-surgeons, and surgeons) and they were given 5 minutes. There was no difference between non-surgeons and surgeons. The general public answered that the plot was significantly more difficult than surgeons and non-surgeons did. The authors' conclusion is that different ways of presentation may be needed for the general public to understand (PMID 25600357). This means doctors, whether surgeons or not, read it similarly, while the same plot was more difficult for the general public. The name of that plot is the funnel plot. What a funnel plot is a tool to answer in a meta-analysis is covered in "Same Trials, Different Conclusions — Publication Bias and What Went Into the Pool", and is not taught in this article.

Fixed Effect and Random Effects — Two Rules for Splitting the Weight

Even with the same set of studies and the same data, two people can produce different summary values. That is because they chose different rules for splitting the weight. The rules you encounter overwhelmingly often in papers are two, and the two diverge on what they assume before calculation begins.

The fixed-effect model is an approach assuming that the studies being combined are all measuring a single true value. If this assumption is correct, the only reason the observed effect sizes differ from one another is the randomness of the sampling process (PMID 34550583). This assumption is plausible when the studies you want to combine are broadly similar in design, methods, and samples. In that case, you can view each study's value as a value wobbling around a single true value (PMID 41180834).

The random-effects model is an approach assuming that the true value can differ from study to study. This model estimates the average of the distribution formed by those multiple true values. Under this assumption, the difference in the observed values comes from two places. One is the randomness of the sampling process, and the other is the real difference between studies (PMID 34550583). This is the model to use when designs, methods, and samples differ meaningfully so that you must view the true values as multiple (PMID 41180834). Why real differences arise between studies and how to measure them is covered in "I² Is Not the Size of the Scatter — How to Read Heterogeneity Statistics". What this article needs is the single fact that those differences change the weight distribution.

In the random-effects model, weights are spread more evenly. This is because the size of the difference between studies is added in common to all studies. As a result, smaller studies receive relatively more weight, and larger studies receive less. The confidence interval of the summary value also becomes wider than it is under a fixed effect. Conversely, if there are actual differences between studies and you calculate using a fixed effect, an overly narrow interval may result (PMID 34550583).

The two models can yield mutually opposing results. Even if similar values come out, the meaning of the summary value must be read differently (PMID 24692250). This is because the summary value of a fixed effect is an estimate of a single true value, whereas the summary value of a random effects is an estimate of the average of a distribution made of multiple true values. The authors of the same paper also recommend using prediction intervals together under random effects. What prediction intervals answer is covered in "I² Is Not the Size of the Scatter — How to Read Heterogeneity Statistics". And the two models are not everything. There are other fixed-effect models tailored to data recorded as "presence/absence," random-effects models also have extensions, and both can be performed in a Bayesian way (PMID 34550583).

How They Are Actually Chosen

Which model to use should be decided on conceptual grounds before looking at the results. It is not decided after skimming the analysis results with your eyes (PMID 41180834).

Looking at the reality in the field of dental health, this principle is not always followed. The subjects are 313 systematic reviews containing meta-analyses published in 21 dental health journals from 2021 to 2023. Among these, 89% (278 papers) used a random-effects model. The model was decided in advance in 49% (152 papers), and among them, 41% (63 papers) stated the reason for that choice. And 42% (131 papers) out of the 313 chose the model after looking at the heterogeneity statistic that came out as an analysis result (PMID 39962342). The heterogeneity statistic is a number indicating how much the results of studies diverge from one another, and details are covered in "I² Is Not the Size of the Scatter — How to Read Heterogeneity Statistics".

Reporting of the calculation method itself is also sparse. It has been pointed out since 2014 in two journals that the calculation method most widely used when combining different studies can yield falsely high precision—in other words, a biased estimate that looks more precise than reality. Yet, more than half—55.7%—of studies published in top oncology journals from 2015 to 2022 did not describe in detail which method was used in the random-effects meta-analysis. Even among those that did describe the method, that calculation method remained dominant (PMID 38219287).

Two Common Misconceptions

① "It's a Meta-Analysis, So It's Always Better Than a Single Study"

How well meta-analyses matched the results of larger trials that came later varied from study to study. There are two studies that measured this accuracy, and the results of the two studies are different.

The 1997 study matched 12 large randomized trials involving over 1,000 patients with 19 earlier meta-analyses on the same topics. There were 40 comparisons made. The agreement statistic, which measures how much two judgments overlap on a scale from 0 to 1, was 0.35. The authors described this as being at a "fair" level (range compatible with this data 0.06 to 0.64). When a meta-analysis stated there was an "effect," the subsequent trial did as well in 68% of cases, and when it stated there was "no effect," the proportion was 67%. The hit rates in both directions were almost identical.

This study also records the opposite fact. The difference between the values produced by the two methods themselves was statistically significant in only 5 of the 40 cases (12%). Statistical significance means the p-value was smaller than a pre-determined threshold. Every single instance where the judgments disagreed took the form of one side calling it significant and the other calling it not significant. What the authors wrote in their conclusion is this: the results of the 12 large trials we looked at were not accurately predicted by the preceding meta-analyses in 35% of instances (PMID 9262498).

The 2016 study matched 18 large trials published since 2000 in the field of perioperative and anesthetic medicine with 44 earlier meta-analyses. Out of 57 results, 35 (61.4%) were correct. However, here the hit rates for the two directions are heavily skewed. The correct proportion when stating an "effect" was 22.7%, and the correct proportion when stating "no effect" was 85.7%. This means meta-analyses missed particularly often when they declared an "effect." The agreement statistic was 0.094, which in the authors' phrasing is "slight."

This asymmetry was not explained by the power to detect an effect. The paper refers to this power as statistical power. The estimated statistical power of both sides was similar, and only the median study size (the middle value when lined up in order) was larger on the side of the large trials (4,482 versus 1,389). What the authors wrote in their conclusion is this: there was a strong tendency on the side of meta-analyses to lean toward positive results—in other words, results showing an "effect"—which subsequent large trials failed to back up. Therefore, clinical judgments in this field should be made with caution when relying on meta-analysis results (PMID 28077529).

Meta-analysis conclusions sometimes change as more trials accumulate. When 100 meta-analyses in the field of mental health (1,024 trial items, 99,303 participants) are re-accumulated and calculated in the order the trials were published, eight trials reached p<0.05 at some point and then lost significance as more trials piled up. Mostly, this was a case where the large effect size of initial trials became scattered as evidence accumulated. For death, recurrence, and failure outcomes, cases where the effect size decreased were somewhat more frequent than those where it increased (157 versus 125). However, that difference itself had p=0.06, failing to cross the threshold. This trend was clearest when comparing drug treatments to placebos (79 versus 51, p=0.009). The authors' conclusion carries a qualifier: evidence relying on a small number of participants must be interpreted with caution, and the initial efficacy of drug treatments is sometimes overestimated (PMID 15612138).

In emergency medicine, the effect of the earliest study within each meta-analysis showed, on average, a larger benefit than the overall summary effect. This is the result of extracting and examining 3,129 individual study results (83% of which were randomized trials) from 431 meta-analyses. Out of the 431, only 57 (13%) favored the experimental treatment and showed zero signs of suspected bias. Among those, 12 had even a single study with over 80% power to detect an odds ratio effect of 0.70. An odds ratio is a number calculated by dividing the odds of the two groups—that is, the odds of an event happening divided by the odds of it not happening—by each other again. Among those 12, no intervention (treatment or procedure) with adequately powered trials significantly lowered mortality rates. The authors' conclusion is that among the interventions covered by systematic reviews and meta-analyses in this field, few appear to have strong, unbiased evidence showing they improve outcomes. The authors also note the flip side: the statistical power of the studies has increased somewhat but the majority still fall short, so more properly powered trials are needed (PMID 34133813). What the two signs of bias mentioned here (small-study effects and excess significance) are is covered in "Same Trials, Different Conclusions — Publication Bias and What Went Into the Pool".

Should we then not trust meta-analyses? The answer given to that field by a review surveying the interventional oncology literature is this: a meta-analysis is a methodological tool rather than a definitive benchmark, and its conclusions depend heavily on the quality, consistency, and relevance of the underlying studies (PMID 41942708).

② "A Random Effect Is More Conservative, So It's Safe"

The random-effects model yields a wide interval. Because of this, it passes as the "more humble choice." This idea is only half right.

First is statistical power, which is the ability to catch an effect. One reason meta-analyses are popular is the common belief that their power will be stronger than that of individual studies. This belief is inevitably true for the fixed-effect model. The random-effects model is different. Because the magnitude of the differences between studies enters the calculation and that value must be estimated, it can work to a disadvantage for statistical power. When retroactively calculating from 1,991 meta-analyses in the Cochrane Database, getting a level where the power of a random-effects meta-analysis is consistently higher than that of its constituent studies required five or more studies. In the authors' words, when there are very few studies, statistical inference in the random-effects model is not only difficult but also less worth doing (PMID 28378395). This means a random-effects meta-analysis with just a few studies is not guaranteed to have stronger power than the individual studies.

Next is the weight itself. In random effects, the weight of each study is inversely proportional to the sum of the variance of the value produced by that study (the magnitude of the wobble) and the estimated value of the difference between studies. Because this estimated value is calculated from the data, it varies from dataset to dataset and is also correlated with the effect size estimates of each study. Therefore, when the number of studies is not large, the approximation calculations commonly used for meta-analytic estimates are very inaccurate (PMID 26688589). This is because the weights hang on estimated values rather than values we actually observed. However, the same paper also reports results in the other direction. Under conditions where the number of studies is fixed or grows slower than study size, it shows that the usual meta-analytic estimator is always at least as efficient as an estimate calculated by pooling individual participant data (PMID 26688589). Here, efficient means estimating just as precisely with the same data.

The fixed-effect model also has a useful place. Even in situations where there are substantial differences between studies, the fixed-effect model is a valuable tool if precision is the priority (PMID 37858407). Conversely, when both within-study variation and between-study variation must be accounted for, the random-effects model is appropriate. Meta-analyses in the field of surgery, where differences between studies are large or intrinsic differences among patients and groups are common, are examples of this (PMID 37945406).

To sum up, the random effects model sits on different assumptions and is not a safer choice in its own right. The reason the interval widens is because one more uncertain term—namely, the estimated value of the difference between studies—goes into the calculation. The fact that the weights are spread more evenly means that the share of small studies grows larger (PMID 34550583).

Questions to Ask in Front of the Figure

When you encounter a forest plot, there are four things to ask.

1. Rather than how many lines there are, what do the boxes look like? If a single study took most of the weight, the diamond will be close to that study's value.
2. Is each line a study or a variable? Analyses that put multiple variables together from a single individual study are also drawn with this figure (PMID 41774848).
3. Is it a fixed-effect or random-effects model? This choice should be made based on conceptual rationale before looking at the results, and should not be decided after glancing over the analysis results (PMID 41180834).
4. Is this diamond using the same trials as the diamond from another meta-analysis? If the same trial is repeatedly included across multiple meta-analyses, it looks as though there are multiple independent pieces of evidence (PMID 40253011).

If you are to leave behind only one thing, it is this: A meta-analysis does not simply average the results of multiple studies, but divides and pools the weights. The position of the diamond is determined by where the weight is concentrated.

The Spaces Left Empty by This Article

Let us first place one balanced evaluation. One journal commentary writes this. Systematic literature reviews and meta-analyses, after being introduced in the 1970s, have replaced unsystematic and selective review articles. Review articles are papers that summarize multiple studies, and older reviews varied widely in access, structure, and content. The summary provided by a forest plot is something that no single study can reach, and these tools hold a legitimate place among respected research instruments. However, despite standard procedures, their quality varies widely, and so concerns about quantity outpacing quality are justified. This commentary views that the appropriate response is not to reject these tools, but to instill in researchers a sense of how complex this work is, creating better-informed authors, reviewers, and editors (PMID 36639571).

In the field of thoracoabdominal surgery, high-quality reviews were actually cited more often. When counting the citation counts of related reviews up to 2020, higher methodological quality scores were independently associated with citation counts, controlling for other factors. However, the same investigation also noted that overlapping reviews were common, and that their variation yielded different effect size estimates for the same topic (PMID 34791075).

This article has only come as far as reading a single plot. The two remaining spaces are taken up by "I² Is Not the Size of the Scatter — How to Read Heterogeneity Statistics" and "Same Trials, Different Conclusions — Publication Bias and What Went Into the Pool".

The first is how the "real difference between studies" assumed by the random-effects model is actually measured. There are statistics that represent that size, and "I² Is Not the Size of the Scatter — How to Read Heterogeneity Statistics" covers what those statistics do and do not say. Another interval, often drawn near the diamond, is also covered on that side.

The second is the studies not carried on the forest plot in the first place. If there were studies that were not published because their results were lackluster, those studies do not appear in this plot. The plots and methods used to detect such studies, and why meta-analyses on the same topic reach different conclusions, are covered by "Same Trials, Different Conclusions — Publication Bias and What Went Into the Pool".

There are also things this article did not cover. Frameworks that grade the quality of evidence, sequential analysis, forms that compare multiple treatments in a single net, forms that collect individual participant data, and forms that are continuously updated. A guide addressing the interpretation of meta-analyses has organized this list (PMID 39239131).

Source Papers

The Same Trials Are Counted Multiple Times

• "The redundant landscape of met…" (2025) — PMID 40253011 — Out of 40 meta-analyses and 728 randomized trial items, 276 were unique (37.9%). 153 were in only one review, and 123 were in 2 to 19 reviews. The authors' conclusion is that repeated use creates an "illusion of robustness" and that strength and diversity may be more limited.
• "The methodological quality was…" (2022) — PMID 34718123 — There are 21 systematic reviews but 8 published randomized trials. 18 are the lowest grade, and 1 is the highest grade (as indicated in the abstract). Discrepant conclusions across all three outcome areas. The authors' conclusion is that quality was uniformly low, so the association between quality and conclusions could not be evaluated.
• "The Mass Production of Redunda…" (2016) — PMID 27620683 — Annual growth rates from 1991 to 2014 were 2,728% (reviews) and 2,635% (meta-analyses) compared to 153% for PubMed overall. Meta-analyses on the same topic sometimes exceeded 20. The conclusion is conditional ("possibly, the large majority").
• "High variability in results an…" (2021) — PMID 34791075 — There were 57 thoracoabdominal surgery reviews (48 topics) in 2015, with 146 overlapping reviews from the preceding 5 years covering 29 topics (60.4%). The variation produced different effect size estimates within the same topic. Higher quality was independently associated with 5-year citation counts.

What Are Meta-Analysis and the Forest Plot

• "Understanding the Basics of Me…" (2020) — PMID 33027562 — A method of combining multiple study results to obtain a summary estimate that better represents the true value of the population. Applicable to means, proportions, relative risks, odds ratios, and more. Results are shown as a forest plot.
• "Interpretation of meta-analyse…" (2022) — PMID 39239131 — A guide to the steps (systematic literature search, risk of bias assessment, data extraction, data pooling) and interpretation.
• "[Anatomy of the Forest Plot gr…" (2026) — PMID 41774848 — One of the most widely used graphs in recent literature. Initially for meta-analysis, now also used for individual study results (both qualitative and quantitative variables, and multivariate results, as long as confidence intervals can be calculated).
• "Say farewell to bland regressi…" (2024) — PMID 38306346 — The problems with the practice of presenting coefficients in tables, and a set of visualizations that arrange multiple models in a single figure.
• "The orchard plot: Cultivating…" (2021) — PMID 32445243 — In ecology and evolution, it is common for a single meta-analysis to have over 100 effect sizes. In a survey of 102 papers, the use of the classical forest plot was 11%. Alternative plots plot individual effect sizes proportional to precision.
• "Surgeon-level reporting presen…" (2015) — PMID 25600357 — Five minutes were given to four groups of 10 people each (40 people) after turning virtual data into a graph. There is no difference between non-surgeons and surgeons. The prescription is that different presentation formats may be needed for the general public.

Fixed and Random Effects — Assumptions and Weights

• "Fixed- and Random-Effects Mode…" (2022) — PMID 34550583 — Fixed effects assume a single common effect, attributing all observed variation to sampling error, whereas random effects estimate the mean of a distribution of effects and divide the variation into sampling error and between-study variance. The most common form is a weighted average, and both use inverse-variance weighting. Random effects spread the weights more evenly, giving larger relative weight to small studies and less to large studies. Confidence intervals are wider, and using a fixed-effect model when heterogeneity is present can yield overly narrow intervals.
• "Fixed Effect Versus Random Eff…" (2025) — PMID 41180834 — If designs, methods, and samples are mostly similar, use fixed effects (a single true value); if they differ meaningfully, use random effects (multiple true values). The choice must be made in advance based on conceptual rationale, not decided after glancing at the results.
• "Demystifying fixed and random…" (2014) — PMID 24692250 — The two approaches can yield similar or conflicting results, and even when similar, the summary estimates must be interpreted differently. For random effects, reporting prediction intervals alongside is recommended.
• "Fixed-effect Versus Random-eff…" (2023) — PMID 37858407 — Even in situations with substantial between-study heterogeneity, the fixed-effect model remains a valuable tool if precision is the priority.
• "Fixed-effect Versus Random-eff…" (2023) — PMID 37945406 — Random effects can account for both within-study and between-study variation, making them suitable for surgery meta-analyses where high heterogeneity or inherent differences among patients and groups are common.
• "On random-effects meta-analysi…" (2015) — PMID 26688589 — Each study's weight in a random-effects model is inversely proportional to the sum of the variance of its effect-size estimate and the estimated variance component of the random-effects distribution. Because that estimate is a random variable, the usual normal approximation is highly inaccurate unless the number of studies is large. A different axis of results shows that when the number of studies is fixed or grows more slowly, the meta-analytic estimator is always at least as efficient as the pooled maximum likelihood estimator using individual data.
• "Appropriateness of conducting…" (2024) — PMID 38219287 — Despite the pointing out that the most widely used calculation method can produce biased estimates with falsely high precision, 55.7% of studies in top oncology journals from 2015 to 2022 did not describe their methods in detail.
• "Statistical Heterogeneity in O…" (2025) — PMID 39962342 — 313 systematic reviews containing meta-analyses from 21 dental journals (2021–2023). Random effects were used in 89% (278 reviews). Models were decided in advance in 49% (152 reviews), of which 41% (63 reviews) provided justification. 42% (131 reviews) selected their model after looking at heterogeneity statistics.

Is Meta-Analysis Better Than Individual Studies?

• "Discrepancies between meta-ana…" (1997) — PMID 9262498 — 12 large-scale randomized trials versus 19 preceding meta-analyses, yielding 40 outcomes. Agreement 0.35 ("fair", 0.06 to 0.64). Positive predictive value 68%, negative predictive value 67%. Point estimate differences were significant in only 5 (12%), with all discrepancies taking the form of only one side being significant. The authors' conclusion is that the results of the 12 trials were not accurately predicted in 35% of cases.
• "Poor agreement in significant…" (2016) — PMID 28077529 — 18 large-scale trials versus 44 preceding meta-analyses, hitting 35 out of 57 outcomes (61.4%). Positive predictive value 22.7%, negative predictive value 85.7%, agreement 0.094 ("slight"). The authors' conclusion is that meta-analyses had a strong tendency to lean toward positive results that subsequent large-scale trials failed to support, and that this is not explained by differences in statistical power. Median sample size 4,482 versus 1,389.
• "Effect sizes in cumulative met…" (2004) — PMID 15612138 — 100 meta-analyses (1,024 trials, 99,303 participants). 8 reached significance and then lost it. In mortality, recurrence, and failure outcomes, decreases were somewhat more common than increases (157 versus 125, P=.06), most markedly in drug versus placebo (79 versus 51, P=.009). The conclusion is that initial efficacy is sometimes overestimated.
• "An umbrella review of effect s…" (2021) — PMID 34133813 — 431 meta-analyses, 3,129 individual outcomes (83% randomized trials). Median odds ratio 0.70. Within each meta-analysis, the effect of the initial study on average showed a greater benefit than the summary effect. Favorable and without bias signals were 57 (13%), among which 12 had even one study with 80% or greater statistical power, and among those, no interventions significantly lowered mortality.
• "Power analysis for random-effe…" (2017) — PMID 28378395 — While it is necessarily true that meta-analysis increases statistical power under fixed-effect models, it can be disadvantaged under random-effects models because the variance between studies must be estimated. Retrospective calculation of 1,991 Cochrane meta-analyses: 5 or more studies are needed to fairly consistently achieve higher statistical power than the constituent studies.
• "Meta-Analysis: The Holy Grail?" (2026) — PMID 41942708 — An overview review of the interventional oncology literature. The conclusion is that meta-analysis is a methodological tool rather than a definitive benchmark, and that its conclusions depend heavily on the quality, consistency, and relevance of the underlying studies.
• "Missing the forest-plot for th…" (2023) — PMID 36639571 — Journal commentary (not an original study). Acknowledging the role systematic reviews have played in replacing narrative reviews and the summary value provided by forest plots, it simultaneously notes variations in quality and concerns that "quantity outstrips quality." The response is not rejection, but producing better-informed authors, reviewers, and editors.