Where Does Pretest Probability Come From? — History, Physical Examination, and Clinical Prediction Rules

"What a Test Actually Tells You — An Introduction to Sensitivity, Specificity, and Pretest Probability" ended with why a single number on a test report cannot be an answer on its own. A test only determines how much it shifts the probability, and where you start is determined by the person themselves. Pretest probability is the probability that a person has the disease before taking the test. This starting point is not written anywhere on the test result sheet.

Then where is the pretest probability written?

The answer is as old as it is mundane. Pretest probability comes from the medical history, physical examination, and the path the person took to get here. When they arrive at the laboratory door, half of the judgment is already done. However, since this half is filled by humans, it inherits the exact nature of human work. It wavers from person to person and leans to one side. That is why tools become necessary.

This article uses several terms from the previous piece as they are. Sensitivity is the proportion of people who actually have the disease whom the test catches as positive, and specificity is the proportion of people who do not have the disease whom the test catches as negative. Both are properties belonging to the test.

Likelihood ratio is a value representing how many times more likely a given result is to occur in a person with the disease compared to a person without the disease. If the two likelihoods are equal, the likelihood ratio is 1, and it does not shift the probability at all. If it is greater than 1, it pushes the probability toward the disease side; if smaller, toward the opposite side. Posttest probability is the probability after reflecting the test result, in other words, the probability after the test. Pretest probability and likelihood ratio together create the posttest probability.

Why You Need to Know — The Hand Setting the Starting Point Shakes

Even when given the same patient case, clinicians vary widely in the pretest probabilities they assign. This was shown by a postal survey sent to 819 internists and general practitioners in Australia and the United Kingdom, asking them to write down pretest probabilities for three identical scenarios (ischemic heart disease, deep vein thrombosis, and stroke). The "correct answer" used as a comparison standard was the value calculated using a validated clinical prediction rule. A clinical prediction rule is a tool that combines history, physical examination, and basic test findings to assign a patient's probability.

The proportion of respondents who fell within 20 percentage points of the correct answer was about 55% for the heart disease and stroke scenarios, and 6.7% for the deep vein thrombosis scenario. Among demographic characteristics, practice type, and educational background, none clearly predicted accuracy. Pretest probability estimation scatters widely even among experienced clinicians. The authors concluded without blaming the clinicians, pointing instead to tools. They argued that clinical prediction rules must be developed and disseminated to support practical judgment (PMID 15115422).

Pretest probabilities do not just scatter; they also lean to one side. Multicenter data from 840 patients who presented with chest pain and shortness of breath whose diagnoses were not revealed by electrocardiogram and remained unclear provide an example. Pretest probabilities assigned by clinicians by eye (gestalt) were consistently higher than those from a validated computerized method. They were 17% versus 4% for acute coronary syndrome, and 12% versus 6% for pulmonary embolism (PMID 24070658).

How well the two methods separated patients was compared using the area under the curve. The area under the curve is a value that summarizes how well a test separates people with and without a disease into a single number, with larger values indicating better separation. This value was similar between the two methods for pulmonary embolism (0.81 versus 0.84), but lower for the gestalt method in coronary syndrome (0.64 versus 0.78) (PMID 24070658). This means that while the gestalt method set probabilities higher, its power to separate patients for pulmonary embolism was as accurate as the computerized method.

The same pattern appears in prognosis estimation. When 257 chronic kidney disease patients and their attending nephrologists were followed together, 13% reached end-stage renal disease within two years. The three sets of estimates generated by the patients, the physicians, and a risk calculator all had good ability to rank who was at higher risk (c-statistic exceeding 0.8). The c-statistic is a value used interchangeably with the area under the curve in such predictions. However, when compared against actual outcomes, both physicians and patients overestimated risk, and the one that best matched the actual incidence rate was the calculator (PMID 30630859).

What these studies show is that estimating pretest probability is inherently difficult. The direction of this point is opposite to saying that doctors cannot be trusted. Because the difficulty of estimating pretest probability was recognized, research arose to weigh the importance of medical history and physical examination, leading to prediction rules that combine them. The rest of this article follows those tools.

The Big Picture — There Are Three Ingredients to Pretest Probability

There are three ingredients to pretest probability. The first ingredient is prevalence. Prevalence is the proportion of people in a given population who have the disease. Looking at the population as a whole, the prevalence is the pretest probability.

The value combining these three becomes the starting point of the next test. Depending on how you estimate the pretest probability, everything changes: whether or not to do additional tests, which tests to do, how to read the results, and what to do afterwards (PMID 34740785).

Among the three ingredients, prevalence—the first one—is forgotten most often and moves the most widely.

Temporal arteritis shows this contrast as it is. The prevalence of this disease in the general population is less than 1%. Yet, looking only at patients referred to receive a temporal artery biopsy, 39% were biopsy-positive (synthesis of 21 core studies) (PMID 11754714). It is the same disease and the same test, but the starting points are nearly 40 times apart. Between the two numbers lies the filter of referral, meaning the fact that someone has already filtered them once and sent them.

This filter is everywhere. All the values below are values that the study observed in that setting.

• Among 71 adults referred by a general practitioner who suspected pneumonia and prescribed antibiotics, pneumonia confirmed by chest X-ray was 15% (PMID 3387706).
• In adults coming to the emergency room with a sudden pain in a single joint, the prevalence of non-gonococcal septic arthritis was estimated to be roughly 27% (PMID 21843213).
• Among patients coming to the emergency room with acute wrist trauma, the actual fracture was 39% (PMID 26682537).
• In 279 consecutively enrolled people with knee problems from orthopedic and family medicine clinics and the community, it was the same people, but the prevalence varied wildly with each diagnosis. Symptomatic meniscal tear was 28.7%, symptomatic knee osteoarthritis was 46.2%, and patellofemoral pain was 26.9% (PMID 29111463, PMID 29781110, PMID 29128344).

As seen in the last item, even within the same patient population, the pretest probability changes entirely depending on which diagnosis you are asking about. Pretest probability is not a single number attached to a patient. It is a value determined together by these three: this patient, this setting, and this question.

The filter of referral is also smeared onto numbers published in papers. In an overview meta-analyzing physical examination findings for shoulder pain, all included studies examined patients referred to specialists. In high-quality studies, the prevalence of rotator cuff disease was scattered between 33% and 81%. The authors therefore write themselves that it is unknown whether these results fit unreferred populations (PMID 23982370).

Where then do prevalence numbers come from either? There is still an unfilled hole here. Among 59 Cochrane diagnostic accuracy reviews (308 meta-analyses), 148 analyses actually used a pretest probability, and the sources of those values were all over the place: studies included in the review, external data, and author consensus. When the same pretest probability was imposed on all to compare multiple meta-analyses of the same disease side by side, that value fell within the prevalence range of the analyses covering that disease about half of the time (PMID 32299367). The authors suggest that review authors should examine whether to impose the same pretest probability and how that value changes the interpretation. Therefore, the sentence "the pretest probability of this disease is X%" is usually a single representative value chosen in some way from some data. This is why pretest probability must be read as a range rather than a point.

Easy Basic Terms

Medical history and physical examination findings also have likelihood ratios. Likelihood ratios do not belong only to blood test numbers or imaging. There is a likelihood ratio even in a single question like, "Have you ever had to stop chewing because your jaw hurt?" This might be the most important sentence in this article.

The likelihood ratio when a finding is present is called the positive likelihood ratio, and the likelihood ratio when a finding is absent or a test is normal is called the negative likelihood ratio. A positive likelihood ratio shifts the probability significantly toward disease the larger it is, while a negative likelihood ratio shifts it significantly away from disease the smaller it is.

The power of medical history and physical examination is frequently underestimated. A review comparing clinical evaluation and additional tests side by side using likelihood ratios argues as follows. Modern medicine tends to emphasize sophisticated tests and imaging, pushing aside medical history and physical examination rather than supplementing them. Yet, when actually measured, the discriminative power of clinical evaluation and additional tests is often similar, and clinical evaluation frequently provides more information (PMID 9666947).

Let us look at a few actual values. All of them are values observed by those studies in those patients, and what kind of patients they were is as important as the values themselves.

• In patients referred for temporal artery biopsy, among medical history items, only jaw claudication (positive likelihood ratio 4.2) and diplopia (3.4) clearly raised the probability. Jaw claudication is a symptom where chewing hurts and forces one to stop. On physical examination, changes where the temporal artery feels like a string of beads (4.6), prominence (4.3), and tenderness (2.6) raised the probability. A normal erythrocyte sedimentation rate lowered the probability significantly (negative likelihood ratio 0.2). Conversely, a finding that the temporal artery has no abnormalities merely lowered the probability slightly (0.53) (PMID 11754714).
• In influenza, chills (7.2), a combination of fever and visiting within 3 days of onset (4.0), and sweating (3.0) raised the probability. The absence of systemic symptoms, cough, inability to perform daily activities, and bed rest lowered the probability (0.36 to 0.50). The authors' conclusion was that the value of individual signs and symptoms is limited, and rules that systematically combine symptoms may be a more useful strategy (PMID 15014046).

Likelihood ratios do not need to be limited to just positive and negative. When results come out as numbers, likelihood ratios can be assigned to each range of values. In septic arthritis, the range-specific likelihood ratios for synovial fluid white blood cell count were 0.33 in the low range, 1.06 in the middle range, and 3.59 in the range above that (PMID 21843213). Values in the middle range shift nothing at all. The same test oscillates between ruling out, neutral, and ruling in depending on the value.

Findings that clinicians believe are important and findings that actually separate diagnoses can be different. There is a study that directly measured this discrepancy. Fifty-one musculoskeletal physicians were surveyed on how valuable 83 signs and symptoms were, respectively, for separating hip osteoarthritis and lumbar spinal stenosis. Then, the 32 symptoms and 13 physical examination items chosen as most valuable were applied to actual patients to calculate positive likelihood ratios. There were 77 patients with hip osteoarthritis and 79 with spinal stenosis.

Among the 32 symptoms chosen by physicians, 11 actually separated the two diagnoses. Symptoms supporting hip involvement were groin pain (4.9), pain that decreases while walking (3.9), and so on. Symptoms supporting spinal stenosis were pain below the knee (2.3), and leg numbness or paresthesia (2.7). The likelihood ratio for the finding that pain is less when pushing a shopping cart was 1.0, shifting nothing. The same was true for lower back pain (1.1), leg weakness (1.1), and buttock pain (1.2).

Physical examination was different. As physicians expected, 7 of the 13 physical examination items strongly supported hip involvement (inability to bear weight on the painful leg 10, observed antalgic gait 9). Four neurological deficits supported spinal stenosis (PMID 30708114).

Which findings shift probability is not determined by belief, and can only be known by measuring. This does not mean physical examination is better than history taking.

Bundling several such findings together is a clinical prediction rule. Earlier, we called this rule a tool that bundles findings to assign probabilities. It is also called a clinical decision rule. More precisely, it is a tool that quantifies the contribution of medical history, physical examination, and basic tests to diagnosis, prognosis, and treatment response respectively, dividing patients according to probability. The Users' Guides to the Medical Literature describe the rule as an attempt to formally test, simplify, and increase the accuracy of a clinician's intuitive sense. Rules are created through three steps (PMID 10872017).

Derivation is the step of finding predictors in one group to make a rule, and validation is the step of re-measuring that rule in a different group and a different setting. Validation performed in hospitals and patients outside the data used to make the rule is called external validation.

Impact analysis is the step of seeing whether practice and outcomes change when the rule is actually made to be used. The three steps answer different questions.

The Users' Guides consider a rule that has even completed validation in a new environment as level 1, and deem this most suitable for adoption (PMID 10872017). A paper that separately outlines the methodology of impact analysis speaks more strongly on this point. Most studies stop at the derivation stage, a minority go up to validation, and a tiny fraction go up to impact analysis. Yet, impact analysis is the most efficient method to evaluate whether putting a rule into the decision-making process actually improves patient care (PMID 21999201).

Let us look at an actual example of a rule. The Amsterdam Wrist Rules is a rule to select patients who should undergo X-ray imaging among those visiting the emergency department with acute wrist trauma. It was created using eight variables such as age, swelling, and tenderness in 882 patients across five hospitals (487 for derivation, 395 for external validation). In external validation, the area under the curve was 0.81. Sensitivity was 98%, hardly missing any fractures, but specificity was low at 21%. The negative predictive value, meaning the probability that a person whom the rule called negative truly does not have a fracture, was 90%. Explicitly noting that it failed to reach 100% sensitivity and that specificity is low, the authors position this rule as a screening tool (PMID 26682537).

The combination of 98% sensitivity and 21% specificity comes from where you draw the threshold. As seen in "What a Test Actually Tells You", lowering the threshold catches more at the cost of more false alarms. This tool tilted its values toward not missing things. What you want to avoid more determines the shape of the tool.

Authors writing for themselves "not yet" regarding their own rule is the mark of a good rule paper. Papers in the validation and impact analysis stages that we will look at later also state their limitations first.

When evaluating prediction rules, people often look only at performance. But performance has two different faces, and the two complement each other (PMID 34882175).

• Discrimination is the ability to correctly order people with and without the disease. It is summarized by the area under the curve.
• Calibration is the degree to which the numbers produced by a model match reality. Calibration is good if the event actually happens to about 30 out of 100 people whom the model told "30%".

These two can separate, and they separate very widely.

A representative study externally validated 43 mortality prediction models in intensive care units across two cohorts. Originally, only 11 out of 43 models (26%) had the information necessary for replication and could be validated (PMID 36378565). Among them, some had good discrimination (area under the curve up to 0.83), but most had poor calibration. Only after re-tuning the intercept, which determines the overall height of predicted probabilities, and the slope, which determines how extreme predictions spread, to fit the new data—in other words, only after recalibration—did they become moderate. The title of the paper summarizes it directly. Discrimination was preserved, calibration was poor.

There is an example closer to rehabilitation. A model predicting long-term prognosis of aphasia after stroke was externally validated in data from 131 patients from a different trial. Discrimination was good (area under the curve 0.87). However, at 6 months, 86 out of 124 people (68%) showed good outcomes, whereas the model predicted 88%. It ordered well, but the scale was shifted toward the higher side. The authors note that the model needs to be recalibrated, renamed, and subjected to further external validation (PMID 30322381).

The same thing happened in a fracture risk tool. This tool was applied to 63,632 white women and 1,703 Asian women from Canadian bone density registry data. In white women, the observed fracture rate rose sequentially as the risk stratum was higher. In Asian women, particularly in the middle and high-risk strata, the observed fracture rate was generally lower than predicted. The tool had overestimated risk in this group. The authors diagnosed this as poor calibration and pointed out that risk overestimation can influence decisions made together with patients (PMID 36201065).

A model can order well while its scale is wrong. And the number used in front of an individual is a scale, not an order.

If you have estimated the pretest probability, the next question is this. So, will you do the test?

Concepts that give a formal shape to this question are the testing threshold and the treatment threshold. The testing threshold is the point below which probability drops and you stop testing and also do not treat, and the treatment threshold is the point above which probability rises and you start treatment without further testing. This is because when probability is very low, testing itself causes more harm than good, and when it is very high, further testing does not change the decision. The space between the two thresholds is the zone where tests do their work.

A review on septic arthritis performs this calculation in practice. Incorporating the sensitivity of 56% and specificity of 90% for synovial fluid white blood cell count, along with best estimates for the risks associated with diagnosis and the risks and benefits of treatment, the testing threshold for arthrocentesis came out to 5%, and the treatment threshold to 39% (PMID 21843213). The authors specify that this value is an example calculation derived from assumptions they entered, and the purpose is to provide a table where one can try out such calculations. If the assumptions entered change, the thresholds change too.

How It Actually Shows Up

First, we look at whether rules or a clinician’s intuition are better. The results are not neat.

In a meta-analysis (16 studies, 8,306 people) comparing eyeball estimates and prediction rules in pulmonary embolism, the actual rates of pulmonary embolism across three groups of low, intermediate, and high risk in 7 studies using eyeball estimates were 8–19%, 26–47%, and 46–91%, respectively. In 10 studies using prediction rules, they were 3–28%, 16–46%, and 38–98%. The discriminative ability of the two was similar. The authors recommended the rules, and the rationale was not just accuracy. The rules were also confirmed to be accurate, plus less experienced clinicians could use them (PMID 14657070).

There are also data pointing in the opposite direction. A retrospective analysis of 1,038 patients suspected of having a pulmonary embolism in the emergency room (actual prevalence 31.3%) showed that the area under the curve for eyeball estimates was 0.81, which was higher than the Wells score (0.71) and the revised Geneva score (0.66). The Wells score and the revised Geneva score are two representative prediction rules used for pulmonary embolism. However, another result from the same paper is important. The agreement among the three methods was poor (all $\kappa$ values below 0.3). $\kappa$ is the agreement measured after subtracting the share that would match by chance. This means that for the same patient, each method placed them in a different probability bracket (PMID 23433653).

There is no total victory for either side. When comparing two prediction models and the attending physicians' evaluations in 244 hip fracture patients, there was no statistically significant difference in discriminative power (area under the curve 0.73–0.80). One model missed the majority of high-risk 30-day mortality cases (sensitivity 33%), and the other model systematically overestimated. Doctors found high-risk patients better (sensitivity 78%) but also tended to overestimate (PMID 35605101).

The picture is the same in the intensive care unit. 961 critically ill patients on mechanical ventilation were gathered from five European centers. Subjective 28-day survival estimates by nurses, residents, and attending physicians discriminated as well as a model gathering 62 objective indicators (c-statistic 0.74–0.78 versus 0.67–0.72). However, subjective estimates had poor calibration, overestimating mortality by about 20 percentage points in absolute terms among high-risk patients. The authors' conclusion is this. Subjective estimates are simple, cost nothing, and have similar discriminative power, but because of overestimation, there is a risk that potentially lifesaving treatments might be withheld. Therefore, they should be compared against objective tools, and if the two disagree, they must be interpreted with caution. As another result of this study, combining subjective estimates and objective indicators improved discrimination and reduced mortality overestimation (PMID 37076881).

Placing these data side by side reveals the place of tools. A prediction rule is a tool to be placed alongside a clinician's judgment and compared, and its usefulness increases when the two are combined.

Whether rules actually change practice must be measured separately from accuracy, and studies measuring that are rare. This is the widest gap in this field.

A registry of clinical prediction rules related to primary care shows the size of that gap. Gathering 745 papers and 989 studies from 1965–2009 literature, there were 434 unique rules that had even been derived. Among them, 54.8% went as far as validation, and 2.8% had their impact on clinical practice or outcomes analyzed (PMID 25024245).

The rehabilitation field is no different. In an 8-year-old updated version of a review collecting diagnostic rules for lower back pain, there were 11 rules. Five new ones that came out in the meantime were all still in the derivation stage, and three that had completed external validation were noted to require cautious use until impact studies come out (PMID 37807828).

An example shows what an impact study asks. A rule for choosing whether to take a chest CT in blunt trauma patients was analyzed using a cost-effectiveness model. Based on a hypothetical cohort of 1,000 people, estimates showed that using the rule would reduce chest CTs by 161, miss 0.08 more clinically significant injuries, and save 136,432 dollars in costs. Flip it around, and the calculation is that routine clinical practice requires taking 2,015 more CTs to catch 1 more significant injury (PMID 31303535).

What an impact study asks is how much the things people experience change. This example estimated that using a model instead of actual implementation data.

There are several reasons why performance, especially calibration, fluctuates in external validation. The model may have been overly tuned to the derivation data, and the patient composition, measurement methods, and baseline risk may differ. The aphasia model and ICU model we saw earlier were classic examples.

Machine learning models have an extra layer of problems. Most of them have not been externally validated in the first place. Among machine learning models predicting orthopedic surgery outcomes, out of 59 models that went through only internal validation—checking performance within the data they were created from—10 made it to external validation. In the 18 papers covering this, discriminative power remained good across the board, but only 3 studies reported other key performance indicators, making it difficult to evaluate overall performance (PMID 33870837).

Even rules that have completed validation do not easily make it into the clinical setting. A commentary summarizing this issue finds the stumbling block in the clinical workflow. The most likely place to put a rule is the electronic health record, but the infrastructure and organizational barriers are large. Therefore, it argues that the center of gravity for research should shift toward impact studies of already well-validated rules and practical implementation methods (PMID 24236419).

Human factors also play a role in deciding whether to use a rule. Cardiovascular prediction rules are widely used in primary care, and a qualitative study exploring how research evidence affects the decision to use a rule through in-depth interviews with 15 clinicians practicing there yielded two paths. Clinicians who were familiar with the research evidence and trusted it tended to look up the evidence directly, make a judgment, and decide to use the rule. Clinicians who were unfamiliar with the evidence, distrusted it, or found it difficult to use tended to encounter the evidence passively rather than judge for themselves. The authors think that targeting unfamiliarity, distrust, and difficulty of use—not just knowledge and skills—may be useful (PMID 37730553).

Common Misunderstandings

The first misunderstanding is that "rules replace clinicians." The data in the previous section is the answer itself. In pulmonary embolism, the discriminative power of eyeballing and rules was similar (PMID 14657070), in some emergency room data eyeballing was higher (PMID 23433653), and in hip fracture mortality prediction the discriminative power of models and doctors was not statistically different (PMID 35605101).

What rules actually do are three things. Rules bring starting points that varied by clinician to the same spot. They allow people with less experience to use them (PMID 14657070). And they become a benchmark to compare systematic bias remaining on the human side. In the studies seen in this article, that bias was toward viewing risk as higher (PMID 37076881, PMID 30630859). However, because a tendency for patients and doctors to view the possibility of survival as higher has also been reported, the direction of bias varies by study (PMID 34882175).

The second misunderstanding is that "if the area under the curve is high, it is a good model," and this is the misunderstanding most frequently missed. The aphasia model had a discriminative power of 0.87, but predicted good outcomes as 88% instead of 68% (PMID 30322381). Intensive care unit mortality models had respectable discriminative power but bad calibration (PMID 36378565). The fracture risk tool overestimated risk in one population (PMID 36201065).

Performance must be looked at with discriminative power and calibration together, and preferably in external validation data (PMID 34882175). Ranking things well and having the numbers' graduations match are different tasks.

The third misunderstanding is that "the performance written in the development paper will come out for my patient too." It is difficult to expect it to come out as is. In the external validations seen earlier, the power to rank was mostly maintained, but the graduations frequently mismatched (PMID 36378565, PMID 30322381). And the reason why that difference is not well revealed in papers has also been uncovered.

When evaluating the reporting completeness of 152 prediction model studies made with machine learning, the median of met items out of 22 reporting guideline items was 38.7%. In particular, information essential for judging whether or not to use this model frequently went missing. Studies reporting participant flow stopped at 3.9%, model specifications at 5.2%, and the model's predictive performance at 5.9% (PMID 35026997).

The interpretation itself is also inflated. Among 62 oncology prediction model studies, 56% used overly strong expressions somewhere in the title, abstract, results, discussion, or conclusion. What was cited as evidence when comparing models with each other was mostly discriminative power metrics (78%) (PMID 36935090).

Therefore, the numbers in a paper should be read as "values that those authors observed in that data and reported as such."

The fourth misunderstanding is that "rules do not have a scope of application." All rules have a tag attached saying from what kind of patient they were built. That tag is soon the rule's scope of application. The shoulder examination overview explicitly stated that it is not known whether its results fit populations that are not referred (PMID 23982370). The general practice pneumonia study explains part of its results as selection bias, namely a phenomenon where patients entering the study are picked to one side from the start, skewing the results (PMID 3387706).

What to put in as variables is also a choice. The original version of the ureteral stone prediction score contained a race item. When side effects of putting race, a socially constructed identity, into clinical algorithms were raised as an issue, the authors removed that item and remade the score. The performance of the revised version, which put gross hematuria in place of race, was practically the same as the original version. The areas under the curve were 0.85 versus 0.86, and misclassification rates were 0.23 versus 0.23 (PMID 39524039). What is contained inside a rule is also an object to look into separately.

Honest Unresolved Problems

In some settings, there are no numbers at all. More than 80% of dizzy patients are initially managed by general practitioners and are not referred to specialists. Yet the recommended examination methods lack empirical evidence, and no studies on the diagnostic accuracy of vestibular disorders have ever been conducted in general practice settings. This is the situation revealed in a research protocol attempting to fill that gap (PMID 38569697). Many of the likelihood ratios and rules we have are values derived from referred patients, in other words, people who have already been filtered.

The source of prevalence numbers is murky. The Cochrane review investigation we saw earlier remains unresolved just as it was (PMID 32299367). Even the primary ingredient of pretest probability is not precise.

Nearly half of the created rules have not been validated, and very few have even had their impact evaluated. Out of 434 rules, validation was 54.8%, and impact evaluation was 2.8% (PMID 25024245). Machine learning models rarely undergo external validation itself, and reporting is also insufficient (PMID 33870837, PMID 35026997). Even rules that have completed validation struggle to enter clinical practice due to barriers in workflow and infrastructure (PMID 24236419), and whether or not to use a rule can also depend on clinicians' unfamiliarity with and distrust of the evidence (exploratory study interviewing 15 people, PMID 37730553). The three problems are of different kinds, and there is no method yet to solve all three at once.

The most difficult thing is how to communicate uncertainty to people. Estimating pretest probability well does not automatically determine how to communicate it. When gathering studies that investigated what numbers words expressing probability are read as, there were only five expressions that were consistently interpreted by both laypeople and healthcare professionals. "Very likely" was read as 90%, "likely/probable" as 70%, "possible" as 40% (range 30–60%), "unlikely" as 20%, and "very unlikely" as 10%. The authors recommend presenting the corresponding numerical range together when using probability expressions, and giving numbers in the form of "how many out of 100 people" instead of percentages, odds, or fractions (PMID 38707974). The fact that the single word "possible" can be read anywhere between 30% and 60% directly shows the breadth of what is exchanged in the consultation room.

There is no correct answer here. However, the problem has a name.

To Go Deeper

• When reading a prediction rule paper — first look for four things. ① What kind of patients was it built from (what setting, what referral pathway, what was the prevalence in that group) ② What stage is it in now (derivation, has it gone to external validation, is there an impact assessment) (PMID 10872017, PMID 21999201) ③ Did they report both discrimination and calibration, and are those values from external data (PMID 34882175) ④ What did the authors themselves say it is not yet. The better the paper, the easier this sentence is to spot.
• When looking at history and physical examination — you have to measure to know which findings actually move the probability. A finding learned for a long time might have a likelihood ratio close to 1.0 (PMID 30708114), and a single seemingly minor item can push the probability significantly (PMID 11754714, PMID 15014046).
• If you leave with only one thing — the pretest probability is a range, not a point. What can be said honestly about a single person is not a single percentage number, but a sentence that among people with these findings in this setting, it is roughly between this much and that much. History, physical examination, and prediction rules are tools to narrow that range, and cannot turn the range into a point.

Source Papers

Design and cohorts are noted together. Numbers already in the main text are not repeated.

The Starting Point Shakes — PMID 15115422 (2004) Scenario survey of 819 Australian and British clinicians · PMID 24070658 (2014) Comparison of gestalt and computational methods in 840 patients with chest pain or shortness of breath · PMID 30630859 (2019) Three-way comparison among patients, physicians, and a calculation formula in 257 chronic kidney disease patients · PMID 34740785 (2022) Overview of clinical epidemiology.

The Environment Sets the Probability — PMID 11754714 (2002) Rational Clinical Examination on temporal arteritis · PMID 3387706 (1988) 71 pneumonia cases referred by general practitioners, with the authors self-reporting selection bias · PMID 21843213 (2011) Evidence-based diagnosis of adult septic arthritis (including example calculations of interval likelihood ratios and test/treatment thresholds) · PMID 26682537 (2015) Derivation and external validation of the Amsterdam wrist rules in 882 patients across 5 hospitals · PMID 29111463 · PMID 29781110 · PMID 29128344 (2018) Three papers querying three diagnoses in the exact same 279 people · PMID 23982370 (2013) Overview of shoulder rotator cuff disorders, entirely consisting of referred patients.

Where Prevalence Numbers Come From — PMID 32299367 (2020) Investigation of pretest probability sources and selection methods across 59 Cochrane diagnostic accuracy reviews.

The Weight of Medical History and Physical Examination — PMID 9666947 (1998) Overview placing likelihood ratios side-by-side with clinical evaluation · PMID 15014046 (2004) Systematic review on influenza · PMID 30708114 (2019) Comparison between a survey of 51 physicians and 156 actual patients.

The Three Stages of Prediction Rules — PMID 10872017 (2000) Users' guides to the medical literature, defining derivation, validation, impact, and level 1 evidence · PMID 21999201 (2011) Framework for impact assessment methodology.

Discrimination and Calibration — PMID 34882175 (2022) Core principles of prognostic model development and interpretation · PMID 36378565 (2023) External validation of 43 intensive care unit mortality prediction models · PMID 30322381 (2018) External validation of a post-stroke aphasia prognosis model in 131 patients · PMID 36201065 (2022) Canadian bone mineral density registry data, calibrated by racial group.

Rules and Intuition — PMID 14657070 (2003) Meta-analysis of 16 pulmonary embolism studies involving 8,306 patients · PMID 23433653 (2013) Retrospective analysis of 1,038 emergency department patients, with gestalt dominating · PMID 35605101 (2022) 244 hip fracture patients, model versus physicians · PMID 37076881 (2023) 961 mechanically ventilated intensive care patients across 5 European centers, comparing subjective estimates against a 62-variable model.

The Gaps Between Validation, Impact, and Adoption — PMID 25024245 (2014) Primary care prediction rules registry containing 434 rules · PMID 37807828 (2023) 8-year update of the low back pain decision rules review · PMID 31303535 (2020) Impact analysis (cost-effectiveness model) of NEXUS Chest CT · PMID 33870837 (2021) Availability and reporting quality of external validations for orthopedic machine learning models · PMID 24236419 (2012) Commentary diagnosing implementation failures within the workflow · PMID 37730553 (2023) In-depth interviews with 15 primary care clinicians.

How to Read the Numbers in a Paper — PMID 35026997 (2022) Completeness of reporting in 152 machine learning prediction models · PMID 36935090 (2023) Overinterpretation in 62 oncology prediction models · PMID 39524039 (2024) Reconstitution of a ureteral stone score with the race variable removed.

Remaining Holes — PMID 38569697 (2024) Protocol for a diagnostic accuracy study of dizziness in general practice (noting the absence of data from that setting) · PMID 38707974 (2024) Systematic review of the numerical meanings assigned to probability expressions.