The Effect Lives in the Average — An Introduction to Applying Trial Results

The 「25% Reduction」 in Front of an Eighty-Two-Year-Old Patient

Let us imagine an eighty-two-year-old woman sitting in an outpatient clinic. She has heart failure, diabetes, slightly reduced kidney function, and takes nine different medications. There is a report that a drug used for her condition reduced the risk of events by 25% in a large randomized trial. The average age of the trial participants was around sixty, seven out of ten were men, and people with severely impaired kidney function were not included. The setting is hypothetical, but it is not very different from the participant lists of the actual trials we will look at later. Is this 25% also 25% for this person?

This question has a name. Applicability asks whether trial results can be transferred and used for a specific individual patient or a specific clinical setting. Judging a person by trial results is like predicting their future using the results of people who "resemble" them. Evidence-based medicine has generally used the entire pool of people eligible for a trial as that reference group. However, trial participants differ from one another in various ways that affect outcomes and treatment benefits, and what personalized medicine tries to do is narrow that reference group to make a closer estimate for a single individual (PMID 30530757).

This article carries this single reference group all the way through. There are three questions to ask. First, does this person belong to that group? Second, does the effect vary among people within that group? Third, where does this person's risk lie within that group? There is one thing to write down in advance. A 2005 review noted that clinicians are concerned trial results do not generalize well to real patients, especially in some pharmaceutical company trials, and stated that this perception has led to the underuse of effective treatments (PMID 15639683). The opposite can also be wrong. If you apply the average as-is to a patient who could not get into the trial, you end up expecting a benefit for that person that has never been verified. We will see examples of that later. The three questions in this article are about how to read the trial average tailored to a single person, and both dropping the average entirely and transferring it as-is skip that reading.

Average Effects and Individual Effects

Let's say an 82-year-old patient takes the medication for three months and comes back saying they are less short of breath. Did the medication work, was it just that the weather warmed up, or was it thanks to an adjustment in another medication? The individual treatment effect is the difference between the outcome when a person receives a treatment and the outcome when they do not. This difference cannot be observed directly. A person can only live through one of the two states at the same time: either taking the drug or not taking it. One of the two possibilities—how things would have gone with the drug or how they would have gone without it—always remains an event that never happened.

Random assignment overcomes this wall at the group level. Because two groups divided by a draw are split so that they resemble each other on average, the difference in outcomes between the two groups can be read as the effect of the treatment. What random assignment supports is precisely this kind of comparison, namely the effect between treatments, such as the hazard ratio or the difference in response probability (PMID 37835368). The same paper notes that the price paid for this is also modest. Trial participants are a convenience sample gathered sequentially over time, so they do not represent any fixed patient population. Therefore, numbers for a single arm, such as the median survival time in a single treatment group, have limited meaning as values for a broader patient population, and even the comparison numbers require a separate assumption that they can be carried over to a target population (PMID 37835368).

A paper critiquing overconfidence in randomized trials makes the same point more sharply. It argues that the estimates from a randomized trial apply only to the sample drawn for the trial, and generalizing them to a different group or to any individual—even if that individual is a trial participant themselves—requires separate justification. The authors immediately add a warning on the flip side. They write that demanding generalization from a randomized trial to other populations expects too much of the trial and undercuts the share it can actually contribute (PMID 29331519). Their conclusion is that a trial produces knowledge and useful predictions when it forms part of cumulative research built alongside other methods, including the refinement of concepts and theories (PMID 29331519). This means a trial cannot tell us an individual's effect, and its average is useful when read together with other evidence.

The observation that a single person has improved also fails to show the individual treatment effect. A review summarizing strength training studies notes that the apparent differences in how people respond to varying degrees are mixed with multiple layers of noise. These are chance during sampling, measurement error, and biological day-to-day variation in the body, and only when these are stripped away does true individual variation remain (PMID 38878117). Whether the drug worked for that patient who felt less short of breath three months later cannot be known from that single patient's course alone. The single line to take away from this section is this: What a trial measures is the average of a group, and an individual's effect is never directly visible in any trial.

Who Was Included in the Trial — External Validity

Looking at a systematic review pooling 80 acute coronary syndrome trials published between May 2007 and May 2009 with 68,016 participants, 29.7% of the trials used age as an explicit exclusion criterion. Among the participants, 13.8% were aged 75 or older and 27.7% were women, whereas around the same time among adults in the US who experienced this disease, 41.9% were aged 75 or older and 41.5% were women (PMID 21361882). The authors noted that such exclusions may block evidence-based decisions for this high-risk population (PMID 21361882).

External validity is the extent to which trial results provide acceptable evidence for generalization to other populations, other clinical settings, and other treatment modalities and outcome measures. This wording translates a definition agreed upon by 84 panelists through three rounds of online surveys, and the same consensus also produced 14 criteria for judging the external validity of trials included in systematic reviews (PMID 38146072). There is a reason this consensus arrived late. In a systematic review gathering 28 instruments measuring external validity, 61% had no evidence at all for their measurement properties, and the authors concluded that no fully recommendable tool exists yet for measuring the external validity of a trial (PMID 35387582). Even without a measuring tool, who gets left out is quite clear, as the numbers above show.

Exclusion does not happen only in the age column. Looking at phase 3 trials supported by the US National Institutes of Health between 1965 and 2015 that addressed conditions causing major hospitalizations in older adults, 33% had an arbitrary upper age limit, and 67% had a mean or median participant age younger than people afflicted with the disease. Even in trials without explicit age limits, comorbidity criteria such as polypharmacy and concomitant medications (37%) or heart problems (30%) implicitly filtered out older adults (PMID 30693958). The history of patients with multiple chronic conditions being left out of trials is well documented, and it has been proposed that the evidence to support the care of multimorbid patients must be built separately through trials designed targeting those people (PMID 29090144).

Some exclusions are missed when looking only at criteria documents. An analysis examining trials from 2010 to 2025 found that exclusion criteria setting upper age limits were rare, and the exclusion criteria of 41 trials showed no bias toward preferentially removing patients over 80 years old. Even so, compared to the age distribution of people suffering from the disease, older adults entered less than expected, and the reasons more frequently cited by older adults themselves for not making it to randomization were the patient's own preference and the physician's recommendation (PMID 41231225). This means that even if a criteria table looks fair, the roster of participants can turn out differently.

Patients outside trials are usually sicker. In a review covering bloodstream infection trials, across five studies comparing people enrolled from the same source population to those not enrolled, the two groups differed significantly, and the non-enrolled side had a greater burden of comorbidities and consistently worse outcomes, including mortality (PMID 37633330). In a Japanese rheumatoid arthritis cohort, patients meeting all eligibility criteria across 19 biologic therapy trials amounted to a mere 2.3% (cohort participants at the time of each trial) and 7.6% (new users in 2016). In the same cohort, the proportion of trial-ineligible patients reaching response criteria compared to the trial was lower (ACR20 0.38-fold) or higher (remission 1.28-fold) depending on the index, and the authors concluded that there was a difference in treatment outcomes between trials and routine clinical care; this is a comparison of response rates without a control group (PMID 34918127).

Why the prognosis of patients outside trials differs can also be looked at separately. What is being compared here is mortality—that is, prognosis—and not treatment effect. In an analysis comparing 5 heart failure trials with reduced ejection fraction (16,922 participants) and 2 registry datasets (46,914 participants), 56% of the registered patients were eligible to enter the trial. The 1-year mortality rate varied greatly: 7% for trial participants, 12% for eligible registered patients, and 26% for ineligible registered patients. However, simply matching for age and sex made the overall mortality rates of trial participants and eligible registered patients similar (standardized mortality ratio 0.97, with ineligible patients not included in this comparison), and the authors noted that age and sex were the main factors making the difference. Nonetheless, cardiovascular death remained about 30% higher on the trial side even after all adjustments (standardized mortality ratio 1.28) (PMID 34596659).

We need to go a step further from here. "Being different from trial participants" and "having a different effect" are not the same thing, and whether the effect differs can be examined through calculation. By reweighting trial participants to match the composition of the target population, the effect expected in the target population can be estimated. Participants in a low-dose CT lung cancer screening trial were younger (median age 60 vs 63 years), had less heart disease (12.8% vs 17.9%), and were more educated than a nationally representative sample of US adults eligible for screening trials. The relative reduction in lung cancer death recalculated to reflect those differences was 18% (95% confidence interval 1 to 33%), similar to the trial's unadjusted 21% (9 to 32%), and the authors noted that they were similar particularly on the relative scale. Uncertainty grew. Estimates for the target population had wider intervals, which the authors read as a result reflecting the large differences between trial participants and the target population (PMID 38289605). In an analysis extending Swedish coronary intervention trials to all trial-eligible individuals and all actual treatment candidates, hazard ratios were nearly identical to the trial (0.92 and 0.93), but the absolute risks of the two treatment groups were 0.8 to 1.2 percentage points higher than in the trial (PMID 42081803). The target population in both analyses was people who met trial eligibility or became candidates for that treatment in routine care. Because what can be adjusted through calculation is only measured characteristics, these two analyses alone do not confirm whether relative effects remain unchanged for patients filtered out by trials.

Such calculations come with conditions. When the distribution of effect-modifying characteristics differs between the trial and the target population, trial estimates do not transfer as they are, and the methods for transferring them rely on the assumption that if measured characteristics are matched, the two populations would be exchangeable even if swapped, an assumption that is often uncertain or controversial (PMID 42237885). The one-line takeaway of this section is this. The mere fact that your patient is different from trial participants does not mean you can say the effect is different; what you must ask is whether that difference is a characteristic that modifies the effect, and how different that person's risk is.

Are Effects Different for Different People? — Subgroup Analysis and ICEMAN

The 74 stroke trials with public protocols reported a total of 647 subgroup effects, averaging eight or nine per trial (PMID 36988330). These are the cells in the figures at the back of trial papers that lay out effects by sex, age, or disease severity. What those cells ask is whether the effect differs from person to person.

Treatment effect heterogeneity is when the size or direction of a treatment effect varies beyond chance according to certain characteristics of the patients. This definition comes with a single caveat. Heterogeneity is measured on a defined scale for a defined clinical outcome (PMID 31711134). In randomized trials, this heterogeneity is usually explored through subgroup analyses (PMID 31711134). Subgroup analysis is an analysis that divides participants by one characteristic at a time, such as sex or age, and compares the effect across the resulting groups.

Effect modification is the phenomenon where a treatment effect varies depending on other variables like age or disease severity, and is also called subgroup effect or interaction. Most randomized trials and their meta-analyses look into effect modification, but it is difficult to judge whether an apparent effect modification is real (PMID 32778601). Conventional analyses that split by one variable at a time frequently lack statistical power and tend toward false positives (PMID 36036907). Researchers often claim subgroup effects even when they are unlikely to be genuine effect modifications (PMID 34688589).

Tools have therefore been created to evaluate such claims. ICEMAN (Instrument to assesS Credibility of Effect Modification Analyses) is a tool whose items were determined by literature reviews and expert consensus and refined through reviews by trialists, systematic review authors, and journal editors; it asks 5 core questions for randomized trials and 8 for meta-analyses, grading credibility on a scale from "very low" to "high" (PMID 32778601). The backbone of the questions is this: Did the researchers set up a hypothesis before analysis, including its direction? Do prior data support that hypothesis? Is the likelihood that it is explained by chance low? Is the number of tested subgroup hypotheses small? (PMID 34688589).

Applying this yardstick to actual trials yields poor results. Of the 647 subgroup effects reported by the aforementioned 74 stroke trials, 65% had "low" credibility on ICEMAN (PMID 36988330). In a review examining 46 claims of subgroup differences in phase 3 hematologic malignancy trials, only 25% of 44 claims regarding the primary result had clearly predefined hypotheses, and 77% met 4 or fewer out of 10 credibility criteria. The authors noted that even when such claims were expressed strongly, there was insufficient evidence to trust them (PMID 33270263). Across 30 pulmonary arterial hypertension treatment trials, 12 out of 13 subgroup effect claims met 4 or fewer criteria, and the authors concluded that clinicians should treat subgroup effect claims with skepticism (PMID 35743648).

ICEMAN scores do not end in yes or no but are graded into levels. In an individual patient data analysis of two trials comparing ICU oxygen targets, 90-day mortality showed no difference between the two groups at 40.4% and 40.9% (risk ratio 0.99). Among 14 pre-specified subgroups, the finding that COVID-19 patients in the lower target group had more days alive without life support was graded as "moderate" credibility, and the finding that cancer patients in the higher target group had lower mortality was graded as "low" (PMID 38990335). A review combining 11 secondary analyses that split effects based on molecular subtypes in acute respiratory distress syndrome graded a single combination of a liberal fluid strategy and the hyperinflammatory subtype as "high" credibility. At the same time, it concluded that evidence of subtypes consistently altering treatment response is rare and generally of low credibility, and noted that moving toward subtype-guided treatment requires prospective trials with pre-specified interaction hypotheses and methods to classify subtypes in real-time at the bedside (PMID 42730647). Both cases are secondary analyses re-slicing data from completed trials, and the respiratory distress syndrome papers include post-hoc analyses. An ICEMAN rating of "high" or "moderate" is a grading of how believable an effect modification claim is, and that rating alone does not constitute a recommendation to change treatment.

Let us apply this yardstick to a single scenario. Imagine an 82-year-old patient as you open the subgroup figure of a trial paper, and suppose that only in the group aged 75 and older is the point estimate of the effect close to 1 with the confidence interval crossing 1. Asking the ICEMAN questions in turn goes like this: Was this subgroup a pre-specified hypothesis with a directional guess, or was it one of a dozen or more boxes like age, sex, region, and diabetes status? Did other previous trials also show effects diverging by age? Was the difference by age tested directly as an interaction between the two groups, or did you merely observe the fact that the interval for one side alone crossed 1? The more cells there are, the more likely it is that one cell has gone astray purely by chance, and in the previous reviews, claims backed by pre-specification and prior evidence were a minority (PMID 33270263). One must also bear in mind that the cell for those aged 75 and older is likely to have a wide interval due to fewer participants. Before passing through these questions, there is no basis to consider that a single cell tells us more about this patient than the overall average.

"High" is rare. Most reports that a subgroup yielded a different result are hard to trust, and you cannot place those numbers ahead of the overall average simply because that subgroup resembles your patient. The one-line summary of this section is this: Effects can differ from person to person, but subgroups split by one variable at a time rarely show it.

Same Relative Effect, Different Absolute Benefit — Baseline Risk

Suppose there is a treatment that reduces risk by 25%. For a person with a 20% probability of an event occurring in the future, this treatment lowers it from 20% to 15%, subtracting 5 percentage points; for a person with a 4% probability, it lowers it from 4% to 3%, subtracting 1 percentage point. What separates the two people is baseline risk. Baseline risk is the probability that an outcome will occur in an individual without treatment. Even if the relative effect is the same, differing baseline risks lead to differing absolute benefits. This arithmetic and the number needed to treat were covered in What Does "50% Lower Risk" Mean in People? — Reading Relative Risk, Absolute Risk, and NNT. The question here is how much baseline risk actually scatters within a single trial.

A study that reanalyzed individual participant data from 32 published large randomized trials and built 39 risk prediction equations measured that magnitude (PMID 27375287). Within the same trial, the event incidence rates of the quarter with the highest risk and the quarter with the lowest risk differed by a median of 4.3-fold, and widely exceeded 50-fold. The distribution of risk is skewed to one side, so the risk of a patient standing at the center was 0.86 times (median) the trial mean. The authors noted that most patients have a risk lower than the trial mean contained in the summary results. Among 18 comparisons where the overall effect was significant, the relative effect varied according to baseline risk in only 1, yet the difference in absolute risk reduction between the highest-risk quarter and lowest-risk quarter had a median of 5.1 percentage points and a range of -3.2 to 28.3 percentage points (PMID 27375287). This means that while there was typically no evidence that the relative effect varied by risk, the absolute benefit differed across risk strata. The authors concluded that risk-stratified analyses can be clinically useful, especially when outcomes are predictable and rare (PMID 27375287).

Let us transfer these numbers to a hypothetical trial. Assume that the overall trial reduced events by 25% and that the relative effect is similar across risk strata. Let us set the event incidence rate of the lowest-risk quarter at, say, 4%, and set the highest-risk quarter at roughly 4.3 times that, or 17%. The 4.3-fold is the median across 32 trials, and ranged from 1.8-fold to 50.7-fold across trials (PMID 27375287). Then the same 25% subtracts 1 percentage point in the lower stratum, and roughly 4 percentage points in the upper stratum. A single absolute benefit written in the summary results is an average combining these strata, and most trial participants stand below that average. In effect, events are reduced by about one per hundred people in the low-risk stratum, and four per hundred in the high-risk stratum. This is the stratum average, and how much a single person within the stratum benefits remains invisible. The opposite side of the scale, the harms of treatment, cannot be assumed to be identical across strata either. As criteria for dividing participants, not only the risk of the primary outcome but also the risk of serious harms from treatment are used (PMID 30530757), and in the aspirin analysis below (preprint), the relative increase in bleeding was greatest in the low-risk group (PMID 42326811). Because both benefits and harms can vary across strata, the point where the scale tips must be assessed separately for each stratum.

The document that organized this perspective into a standard for analysis is the PATH Statement from 2020. Created through a multidisciplinary panel of experts, literature review, and simulation, this document recommends "predictive heterogeneity analysis"—where, instead of subgroup analyses looking at one variable at a time, patient characteristics are considered all at once to estimate for each patient the outcome risk both with and without treatment—rather than conventional subgroup analyses (PMID 31711134). The approach branches in two directions. A risk model incorporates multiple patient characteristics at once to predict each participant's baseline risk, which is used to divide trial participants into strata of predicted risk and observe effects within each stratum. An effect model incorporates treatment assignment as well as interactions between treatment and patient characteristics to directly predict each patient's treatment effect. Both approaches can be used to predict differences in absolute effect, which is the measure most useful for clinical decisions (PMID 31711134). An accompanying guidance document also outlined the limitations of these methods and remaining research tasks (PMID 31711094).

A scoping review that cited the PATH Statement and gathered 65 reports (162 trials) that built prediction models using trial data followed up on this landscape. Reports identifying reliable and clinically important heterogeneity accounted for 37%. Among reports claiming heterogeneity, the proportion meeting reliability criteria split into 87% (20 out of 23 papers) for risk models and 32% (10 out of 31 papers) for effect models, where effect models were mostly exploratory and whether they were validated in external data split their reliability (PMID 40694348). In the same review, reports from 19 trials that showed overall benefit predicted that 5 to 67% of participants would have no benefit or would even suffer harm, and in 5 trials that showed no overall benefit, 25 to 60% were predicted to benefit (PMID 40694348). These numbers are predictions generated by models, not a count of what actually happened to those individuals one by one.

A single case illustrates this picture concretely. An analysis re-analyzing a trial of epidural steroid injections for lumbar spinal stenosis using a risk model showed that disability scores improved more—by 3.7 and 3.3 points (point estimates)—only in the upper two quartiles predicted to have greater future limitation in back function, with no effect in the lower two quartiles (PMID 36036907). This was a secondary analysis of a single trial, and the p-value for the difference across quartiles was 0.03.

Heterogeneity also changes its appearance depending on the scale. In a preprint analysis that built a bleeding risk model using data from an elderly aspirin trial (19,114 participants), the relative increase in bleeding was greatest in the group with the lowest predicted bleeding risk (subdistribution hazard ratio 2.26), but viewed as an absolute risk difference, heterogeneity between groups was not significant. The preprint authors noted that this result supports an individualized, risk-based approach to aspirin decisions in older adults (PMID 42326811). There is also a theoretical result that the assumption that relative effects transport and the assumption that absolute differences transport generally cannot hold together unless one accepts assumptions where treatment effects are non-existent on average or stronger (PMID 38724763). This is why one must always ask together on which scale "identical" is being claimed.

There are also cases where nothing is found. An analysis re-examining two trial cohorts of 248 patients aged 60 and older with proximal humeral fractures using the PATH method found no heterogeneity between surgery and non-surgery, and the authors concluded that non-surgical treatment should remain the standard (PMID 41067717). When a trial comparing dopamine and norepinephrine in shock (1,679 participants) was re-analyzed using risk-based and effect-based methods, no subgroup for which dopamine was better emerged either (PMID 39256813). The single line of this section is this: Even when relative effects remain similar, absolute benefits grow and shrink following baseline risk, and where my patient's risk lies relative to the trial mean largely dictates that person's absolute benefit. However, the model itself that assesses that risk is also an estimate requiring validation.

Efficacy and Effectiveness

In an analysis comparing survival in routine care with the trials that served as the basis for approval across 20 publicly funded cancer drugs and 29 indications in Ontario, Canada, 29 out of 29 showed worse survival in routine care. The median difference in median overall survival was 5.2 months, the hazard ratio for survival in routine care versus trials was 1.58, and the rate of hospitalization during treatment was a median of 14 percentage points higher than the rate of severe adverse events reported in trials (PMID 31913522). The same drug produced different results in a trial versus everyday practice. Efficacy is the effect a treatment produces under well-controlled trial conditions, and effectiveness is the effect the same treatment produces in routine clinical practice. There is often a gap between the two. In the Netherlands, the median survival of 166 patients with BRAF-mutated metastatic colorectal cancer who received the same combination therapy as in the pivotal trial was 6.7 months, which was shorter than the 9.3 months seen in the trial, and 36% of these patients would not have been eligible to enter the original trial. Among those treated outside of trial eligibility criteria, patients with symptomatic brain metastases (5.0 months) and those with a WHO performance status of 2 or worse indicating impaired ability to perform daily activities (3.9 months) had the shortest median survival. The authors interpreted that in patients with brain metastases and/or a performance status of 2 or worse, neither efficacy nor effectiveness has ever been proven, making the likelihood of a survival benefit negligible (PMID 38769450). This is an author interpretation derived from an observational cohort in a country without an untreated comparator group, and stating that a benefit has not been proven is not the same as stating that there is no benefit.

The pathways leading to this gap, as outlined by a review using multiple myeloma as an example, are as follows. Trials are typically composed of patients who are younger, healthier, and have fewer comorbidities, trial results are primarily replicated in patients who resemble trial participants, and in older, frail patients with comorbidities, effectiveness is reduced due to increased toxicity, dose reductions, and early discontinuation. This review notes that frailty rather than age itself appears to be the primary determinant of this difference (PMID 42425530). Whether this mechanism identified in multiple myeloma plays an equivalent role in the gaps seen with other treatments must be verified separately. There are also conditions created by the trial itself. Trial activities that differ from routine care, such as participation compensation or frequent follow-up visits, can alter how consistently patients adhere to treatment (PMID 41144860).

Caution is also required when measuring the gap. Differences in results between routine care studies and trials may not reflect a true efficacy-effectiveness gap. When the designs of the two studies differ substantially or the estimation in the routine care study itself is biased, the estimation of that gap is also biased (PMID 38413070). This is because a comparison between those who received treatment and those who did not in everyday practice is a comparison of people who were different to begin with, and that issue is covered separately in What "Adjusted for Age, Sex and BMI" Actually Did — How to Read Confounding and Adjustment.

On the trial design side, there are also attempts to narrow this gap. Pragmatic trials are randomized trials whose purpose is to aid decision-making in clinical settings, and an extension of trial reporting guidelines recommends reporting eight items in greater detail—such as participants, interventions, outcomes, and generalizability—so that readers can judge how well the results fit their clinical conditions (PMID 19001484). The bottom line of this section is this: The gap between efficacy and effectiveness is not in itself proof that a trial was wrong; instead, it prompts us to first ask who was treated outside the trial, at what dose, and for how long.

Measuring Directly in One Person — The n-of-1 Trial

There is a narrow side door in the wall that says we cannot see individual treatment effects. An n-of-1 trial is a crossover study that gives one patient two or more treatments in a randomized order over multiple periods, comparing the outcomes within that single person. It is usually run double-blind in chronic disease patients, with the goal of finding the treatment that works best for that specific person (PMID 34551122). Whereas typical designs that compare groups yield an average effect while masking differences between individuals, this design builds evidence for that particular person (PMID 35705532). Because it can also be used to decide on reducing polypharmacy, consecutive n-of-1 trials are underway comparing continuing versus stopping beta-blockers in heart failure patients with preserved ejection fraction (PMID 35705532).

The evidence is still small. In a preliminary randomized trial that assigned 49 hypertensive patients aged 10 to 22 either to n-of-1 trials to select medications or to usual care, the probability that the n-of-1 trials increased blood pressure control after 6 months was 69% with a skeptical prior distribution and 74% with a neutral prior distribution, and the interval for the odds ratio of 1.24 ranged broadly from 0.51 to 2.97. The authors wrote that larger trials are needed to judge whether it should be used in actual clinical practice (PMID 36227203). Only three n-of-1 trials have been identified in inflammatory bowel disease, and designs repeating crossover periods can be perceived as a burden by both patients and researchers (PMID 39303799). Even though the design looks simple, strong assumptions may be needed at the analysis stage (PMID 41501684). This trial is rarely performed outside of research settings and has not yet established itself in general clinical practice (PMID 34551122).

The places where this design can be used are also narrow. This design is mostly used for chronic diseases, and a new use has also emerged to test custom-made treatments for a single patient with a rare disease (PMID 34551122). Because comparing treatments interchangeably requires that the effect wears off when the treatment is stopped and that outcomes can be measured repeatedly over short periods, treatments that prevent once-and-for-all events like death or stroke are difficult to compare using this design. The n-of-1 trial is a narrow tool for recurring chronic problems and some rare diseases, and it is not a general solution that replaces the trial average.

The Number Before a Patient — Closing the Series

We place the numbers this series has traveled through back down. The sensitivity and specificity in "What a Test Actually Tells You — An Introduction to Sensitivity, Specificity, and Pretest Probability" were the proportions counted among groups of people with and without the disease. The pretest probability in "Where Does Pretest Probability Come From? — History, Physical Examination, and Clinical Prediction Rules" was the proportion of people among those with findings like this person who actually have the disease. The surrogate endpoints in "Better Numbers, Better Patients? — An Introduction to Surrogate Endpoints" were relationships verified across multiple trials, and the reference ranges in "A Line Painted on a Slope — An Introduction to Reference Ranges and Diagnostic Thresholds" were lines drawn by gathering values from healthy people. Both the share of overdiagnosis counted in "Is Finding It Early Always Better? — An Introduction to Overdiagnosis" and the benefit of medicines and the time until that benefit appears weighed in "Why Medicines Pile Up — An Introduction to Polypharmacy and Time to Benefit" were numbers counted by gathering certain groups of people.

Those numbers are all numbers of groups. And the three questions of this article fit all of those numbers just as they are. Which people did these numbers come from, and does the person in front of me resemble those people? Even within that group, do the numbers vary from person to person? Where does the person in front of me stand in the risk distribution of that group? Returning to the "25% reduction" before an 82-year-old patient, we apply these questions. Just as patients outside trials were usually sicker, this person is likely to have a higher baseline risk than the trial participants. If the relative effect remains the same, the absolute benefit could be larger than the trial average. Yet, because patients like this person were rarely included in the trial, whether the relative effect remains the same is less verified to begin with, and conditions outside the trial—such as toxicity and early discontinuation pointed out by the multiple myeloma review, along with remaining life expectancy and the time until benefit appears—are weighed together. That weight is not decided by numbers alone.

The average of a trial cannot replace the answer for a single person, but it is the most reliable starting point from which to begin thinking about that person. If we discard the starting point, what remains is merely an impression of one person's clinical course, and that impression does not show individual treatment effects, exactly as we saw at the beginning of this article.

Where to Look Further

Two open questions remain. First, what validation must an effect model go through before its individual predictions can be used in clinical practice? While it is known that validation on external data separates reliability (PMID 40694348), this review did not address whether treating patients according to those predictions actually improves outcomes. Second, are characteristics that appear to create the largest gap between efficacy and real-world effectiveness—such as frailty—measured in the same way by trials and real-world practice? The multiple myeloma review notes that frailty is not measured consistently in either trial or real-world practice data (PMID 42425530).

How a meta-analysis handles differences between trials when pooling them is covered next in "A Meta-Analysis Does Not Take an Average — How the Diamond in a Forest Plot Is Built" and "I² Is Not the Size of the Scatter — How to Read Heterogeneity Statistics". What randomized trials guarantee and what they do not is in "Is the Randomized Controlled Trial Really the 'Gold Standard'?", and a different field's version of the same question asking who entered the study is in "Who Was Being Measured — Who "People" Maned When Psychology Said It". The problem of transferring screening trial results to other populations is good to read alongside "Lung Cancer and Low-Dose CT Screening — Reading a Small Spot in the Picture Over Time".

Source Papers

• "Personalized evidence based medicine: predictive approaches to heterogeneous treatment effects." (2018) — PMID 30530757 — Judging individuals using trial evidence is reference population prediction, and personalized medicine aims to narrow that reference group; the limits of subgroup analysis examining one variable at a time, and risk models versus effect models.
• "External validity of randomised controlled trials: 'to whom do the results of this trial apply?'." (2005) — PMID 15639683 — (Narrative review) The authors state that clinicians' concerns that external validity is poor, particularly in some pharmaceutical-company trials, have led to the underuse of effective treatments; reporting is usually inadequate, and a checklist for clinicians is provided.
• "Interpreting Randomized Controlled Trials." (2023) — PMID 37835368 — Trial samples are convenience samples, the numbers in a single arm have their meaning limited to the population value, and comparative inference also requires a transportability assumption.
• "Understanding and misunderstanding randomized controlled trials." (2018) — PMID 29331519 — Estimates apply only to the sample, and generalization to other groups or individuals requires justification; the authors simultaneously conclude that demands for external validity diminish the contribution of trials.
• "N of 1: Optimizing Methodology for the Detection of Individual Response Variation in Resistance Training." (2024) — PMID 38878117 — To observe individual response variation, sample noise, measurement error, and biological variation must be stripped away.
• "Consensus on the definition and assessment of external validity of randomized controlled trials: A Delphi study." (2024) — PMID 38146072 — 84 panelists reached consensus on 14 external validity definitions and assessment criteria through a three-round Delphi study.
• "Identification of tools used to assess the external validity of randomized controlled trials in reviews: a systematic review of measurement properties." (2022) — PMID 35387582 — Out of 28 tools, 61% have no basis in measurement properties, and no tool is fully recommendable.
• "Exclusion of older adults and women from recent trials of acute coronary syndromes." (2011) — PMID 21361882 — 80 acute coronary syndrome trials involving 68,016 patients from 2007 to 2009; 29.7% excluded by age, 13.8% aged 75 or older, and 27.7% women (compared with 41.9% and 41.5% of US patients).
• "Adequacy of Inclusion of Older Adults in NIH-Funded Phase III Clinical Trials." (2019) — PMID 30693958 — An upper age limit in 33%, 67% of participants younger than the disease onset age, and implicit exclusion via polypharmacy/concomitant medications in 37% or heart problems in 30%.
• "Improving the external validity of clinical trials: the case of multiple chronic conditions." (2013) — PMID 29090144 — The exclusion of patients with multimorbidity from trials is well documented, and trials and reporting targeted at multimorbidity need improvement (Editorial).
• "Representation of Older Adult Patients in Clinical Trials." (2025) — PMID 41231225 — Upper age limit exclusions were rare, but older adults were under-enrolled relative to the prevalence age, with patient preference and physician recommendation cited as reasons for non-allocation.
• "Are we enrolling the right patients? A scoping review of external validity and generalizability of clinical trials in bloodstream infections." (2023) — PMID 37633330 — Non-enrolled patients carry a greater comorbidity burden and consistently have worse outcomes, such as mortality.
• "Differences in patients' population and efficacy/effectiveness of biologic disease-modifying antirheumatic drugs between randomized controlled trials and real-world settings in patients with rheumatoid arthritis - using the IORRA cohort." (2022) — PMID 34918127 — An average of 2.3% to 7.6% of cohort patients met all inclusion criteria across 19 Japanese trials; the response rate among ineligible patients was 0.38 times (ACR20) and 1.28 times (remission), leading to the conclusion that a difference in efficacy and effectiveness exists between trials and clinical practice.
• "Generalizability of randomized controlled trials in heart failure with reduced ejection fraction." (2022) — PMID 34596659 — 56% of enrolled patients were trial-eligible, with 1-year mortality rates of 7%, 12%, and 26%; overall mortality was similar after age and sex adjustment (SMR 0.97), while cardiovascular mortality was higher on the trial side (1.28).
• "Comparing Lung Cancer Screening Strategies in a Nationally Representative US Population Using Transportability Methods for the National Lung Cancer Screening Trial." (2024) — PMID 38289605 — The relative reduction in lung cancer mortality re-weighted to the target population was 18% (1–33%) versus 21% (9–32%) in the trial, showing similarity particularly on the relative scale alongside increased uncertainty.
• "Extending Inferences from a Randomized Trial to Trial-eligible and Treatment-candidate Target Populations: Examples of Generalizability and Transportability." (2026) — PMID 42081803 — Risk ratios of 0.92 and 0.93 matched the trial in trial-eligible and treatment-candidate target populations, while absolute risk was 0.8 to 1.2 percentage points higher.
• "Global Sensitivity Analysis for Studies Extending Inferences From a Randomized Trial to a Target Population." (2026) — PMID 42237885 — If the distribution of effect modifiers differs, results do not transport directly, and the exchangeability assumption of transport methods is frequently uncertain and contentious.
• "The Predictive Approaches to Treatment effect Heterogeneity (PATH) Statement." (2020) — PMID 31711134 — The definition of treatment effect heterogeneity, predictive heterogeneity analysis instead of one-variable-at-a-time subgroup analyses, risk models versus effect models, and absolute effect as the most decision-relevant measure.
• "The Predictive Approaches to Treatment effect Heterogeneity (PATH) Statement: Explanation and Elaboration." (2020) — PMID 31711094 — The intent and rationale behind each recommendation, along with methodological limitations and research agenda.
• "Development of the Instrument to assess the Credibility of Effect Modification Analyses (ICEMAN) in randomized controlled trials and meta-analyses." (2020) — PMID 32778601 — The difficulty of judging effect modification, 5 core trial questions and 8 meta-analysis questions, on a scale from very low to high.
• "Estimating individualized treatment effects using a risk-modeling approach: an application to epidural steroid injections for lumbar spinal stenosis." (2023) — PMID 36036907 — One-variable-at-a-time analyses suffer from low statistical power and false positives; using a risk model, only the top two quartiles showed a 3.7- and 3.3-point improvement ($P = 0.03$).
• "Evidence-based Urology: Subgroup Analysis in Randomized Controlled Trials." (2021) — PMID 34688589 — Researchers frequently make inappropriate claims about subgroup effects; a summary of ICEMAN criteria.
• "The credibility of subgroup analyses reported in stroke trials is low: A systematic review." (2023) — PMID 36988330 — 65% of 647 subgroup effects across 74 stroke trials had low credibility.
• "Subgroup analysis in haematologic malignancies phase III clinical trials: A systematic review." (2021) — PMID 33270263 — Of 44 primary result claims, 25 were pre-specified, and 77% met 4 or fewer out of 10 criteria.
• "Subgroup Analysis in Pulmonary Hypertension-Specific Therapy Clinical Trials: A Systematic Review." (2022) — PMID 35743648 — Across 30 trials, 12 out of 13 subgroup claims met 4 or fewer criteria, leading to the conclusion that clinicians should be skeptical.
• "Lower or higher oxygenation targets in the intensive care unit: an individual patient data meta-analysis." (2024) — PMID 38990335 — 90-day mortality was 40.4% versus 40.9% (risk ratio 0.99), with low credibility for COVID-19 intermediate and cancer subgroups.
• "Heterogeneity of Treatment Effect by Molecular Endotype in Acute Respiratory Distress Syndrome: A Systematic Review." (2026) — PMID 42730647 — Across 11 secondary analyses, one hyperinflammatory phenotype with liberal fluids showed a high effect, but overall evidence is sparse and of low credibility.
• "Risk and treatment effect heterogeneity: re-analysis of individual participant data from 32 large clinical trials." (2016) — PMID 27375287 — The median risk ratio for extreme quartiles was 4.3, most patients were at lower risk than the trial average, risk interaction for relative effects appeared in 1 of 18 trials, and the median absolute reduction difference was 5.1 percentage points.
• "Predictive Modeling of Heterogeneous Treatment Effects in RCTs: A Scoping Review." (2025) — PMID 40694348 — Out of 65 reports, 37% showed reliable heterogeneity, with risk models showing 87% credibility versus 32% for effect models, and 5% to 67% of patients predicted to receive no benefit or harm in benefit trials.
• "Heterogeneity of Treatment Effect of Aspirin and Clinically Significant Bleeding in Older Adults." (2026, preprint) — PMID 42326811 — The relative increase in aspirin-related bleeding was greatest in the low-risk cohort (sHR 2.26), while heterogeneity on the absolute scale was not significant.
• "Learning about treatment effects in a new target population under transportability assumptions for relative effect measures." (2024) — PMID 38724763 — The transportability assumption for relative effects and the transportability assumption for difference effects are generally incompatible (theoretical).
• "Prediction of patient-reported outcomes after proximal humerus fractures in elderly patients does not appear to be a credible option : a secondary analysis of two randomized controlled trials." (2025) — PMID 41067717 — 248 patients showed no heterogeneity, leading to the conclusion that non-surgical treatment should remain the standard.
• "Exploration of different statistical approaches in the comparison of dopamine and norepinephrine in the treatment of shock: SOAP II." (2024) — PMID 39256813 — Re-analysis of 1,679 patients found no subgroup where dopamine was superior.
• "Assessing the efficacy-effectiveness gap for cancer therapies: A comparison of overall survival and toxicity between clinical trial and population-based, real-world data for contemporary parenteral cancer therapeutics." (2020) — PMID 31913522 — Real-world survival was worse in 28 out of 29 indications, with a median survival difference of 5.2 months and a hazard ratio of 1.58.
• "Efficacy-effectiveness analysis on survival in a population-based real-world study of BRAF-mutated metastatic colorectal cancer patients treated with encorafenib-cetuximab." (2024) — PMID 38769450 — In a Dutch observational cohort without a comparator arm, median survival was 6.7 months versus 9.3 months in the trial, with 36% outside trial eligibility and those with brain metastases or a performance status of 2 or worse showing the shortest survival (interpreted by the authors as negligible potential for benefit).
• "Bridging Randomized Trial Efficacy and Real-World Effectiveness in Multiple Myeloma: Integrating Clinical Trials and Real-World Evidence for Individualized Care." (2026) — PMID 42425530 — Trial results are replicated in patients resembling trial participants, whereas older and frail patients experience reduced efficacy due to toxicity, dose reduction, and discontinuation, with frailty appearing to be the primary determining factor (Review).
• "Transporting Results from a Trial to an External Target Population When Trial Participation Impacts Adherence." (2026) — PMID 41144860 — Trial activities such as compensation and follow-up frequency alter adherence.
• "[Progress in methodological research on bridging the efficacy-effectiveness gap of clinical interventions (1): to improve the validity of real-world evidence]." (2024) — PMID 38413070 — Biases stemming from design differences or real-world studies may cause gap estimates to diverge from the true gap.
• "Improving the reporting of pragmatic trials: an extension of the CONSORT statement." (2008) — PMID 19001484 — Definition of pragmatic trials and expansion of 8 reporting items for assessing applicability.
• "A Useful and Sustainable Role for N-of-1 Trials in the Healthcare Ecosystem." (2022) — PMID 34551122 — Blinded, randomized crossover trials in chronic disease patients; rare outside of research and unintegrated into routine care.
• "N-of-1 trials to facilitate evidence-based deprescribing: Rationale and case study." (2022) — PMID 35705532 — Group comparisons average out results and obscure individual differences; an n-of-1 trial is underway for beta-blocker deprescribing in heart failure with preserved ejection fraction.
• "N-of-1 Trials vs. Usual Care in Children With Hypertension: A Pilot Randomized Clinical Trial." (2023) — PMID 36227203 — A pilot trial of 49 patients showed a 69% probability of improved blood pressure control (OR 1.24, 0.51–2.97), indicating a need for a larger trial.
• "Personalized (N-of-1) Clinical Trials for Inflammatory Bowel Disease: Opportunities and Challenges." (2025) — PMID 39303799 — 3 inflammatory bowel disease n-of-1 trials and the burden of the crossover period.
• "Individual-centric N-of-1 trials: a case study assessing the effect of alcohol abstinence on mood levels." (2026) — PMID 41501684 — Even a simple design may require strong assumptions for analysis.