Paperis.

© 2026 네오쿤스. All rights reserved.

공지사항이용약관개인정보처리방침환불 정책콘텐츠·제거 요청

상호: 네오쿤스 · 대표자: 김근태 · 사업자등록번호: 751-04-03501 [사업자정보 확인] · 통신판매업 신고: 제 2026-서울서초-3586호 (서초구청)

주소: 서울특별시 서초구 서초대로 266, B103-S36호(서초동) · Tel: 010-7254-2475 · 이메일: support@paperis.app · 고객센터 운영 시간: 평일 10:00–16:00 (한국 시간, 주말·공휴일 제외) · 호스팅 서비스 제공자: Vercel Inc.

Paperis.
Does AI Diagnose Better Than Doctors? — Between the Test Score and the Clinic

Paperis Articles

Does AI Diagnose Better Than Doctors? — Between the Test Score and the Clinic

On some controlled benchmarks and case vignettes, language models scored above generalist physicians, though pooled comparisons did not clearly separate them from clinicians. In trials that put the same AI beside doctors, physician scores rose little or modestly depending on the trial, while in real mammography screening the arm where AI triaged and flagged images for radiologists found more cancers. This piece follows why the three numbers differ.

🔥Debate✦AI-generatedMedicine · Nov 19, 2026

Your field's papers, as audio

Pick your fields and journals — Paperis summarizes the key papers and turns them into audio. Listen on your commute.

Start Paperis free →

Does AI Diagnose Better Than Doctors? — Between the Test Score and the Clinic

83.2% and 36.1%

A patient's brief initial finding is given first. You must choose yourself what else to ask and which tests to order, gathering information piece by piece to arrive at a diagnosis, and a cost is attached to each test. A study published in 2026 created a test that scores diagnosis narrowed down step-by-step in this manner, and had an AI system designed to mimic a conference where multiple doctors gather and share opinions solve it (PMID 42912144). This system got 83.2% right on clinical-pathological discussion cases from the medical journal The New England Journal of Medicine and 80.3% on a collection of emergency room admission records (MIMIC-IV-ED), while general practitioners who took the same test scored 36.1% and 46.8%, respectively (PMID 42912144).

The foundation of this system is a language model. Large language models are AIs trained on vast amounts of text to guess the next word, answering questions in sentences (How Guessing the Next Word Becomes a Conversation — The Minimum Foundation for Reading Language Models). When you type in symptoms as if asking a doctor, it returns possible disease names and their reasons in text.

83.2% and 36.1% are numbers from the same question paper. A benchmark is a standard test that scores and compares multiple AIs or humans using the same set of questions. Because they are made to solve the same questions the comparison is fair, and for that exact reason the results hold true only within that question paper. The researchers also wrote in the abstract that these results were obtained in a controlled benchmark setting, and concluded that atop that, they show the potential for language models to assist in more accurate and less costly diagnoses (PMID 42912144). This article follows the distance between the score inside that question paper and what happens when AI is brought into actual examination rooms and clinics.

Scores Move With the Test Sheet

On medical board-style question sets, leading language models get 84 to 90% right (PMID 41325597). In a systematic review pooling 39 medical benchmarks published through 2025, success rates were 45 to 69% on benchmarks mimicking actual clinical practice, 45 to 55% on diagnostic tasks, and 40 to 50% on tasks assessing safety judgments (PMID 41325597). The authors concluded that test scores are insufficient and misleading as surrogate indicators of readiness to enter clinical practice (PMID 41325597). Report cards vary this much depending on the format of the test sheet used for measurement.

When the stages of the diagnostic process are separated and scored, the differences become sharper. Differential diagnosis is the process of listing multiple candidate diseases that can explain a patient's symptoms, and narrowing them down one by one using findings and tests. This is the stage where dealing with uncertainty in front of a first-time patient carries high stakes. In a cross-sectional study evaluating 21 language models on 29 patient cases from a medical reference book (the MSD Manual), all models showed failure rates below 0.40 on final diagnoses, but failure rates above 0.80 on differential diagnoses (PMID 41973425). The authors concluded that while these models get final diagnoses right, they are weak compared to other reasoning stages in formulating differential diagnoses and navigating uncertainty, and that off-the-shelf models are not yet at a level where they can be safely used in clinical practice (PMID 41973425). A high score on a question with a single correct answer does not guarantee the ability to widen the pool of candidates.

When studies directly comparing humans are pooled, the picture does not tilt to one side. In a meta-analysis pooling 50 studies that excluded multiple-choice questions and required open-ended diagnosis, the proportion where the top-ranked diagnosis was correct was 0.89 times that of healthcare professionals for language models (confidence interval 0.79 to 1.00) (PMID 41652180). As the number of allowed diagnoses increased, the two grew closer; when up to ten candidates were permitted, it was 1.17 times (confidence interval 0.87 to 1.57) (PMID 41652180). Another meta-analysis pooled 30 studies and 4,762 cases, noting that most studies had a high risk of bias, the primary reason being that they used vignettes where the diagnosis was already known (PMID 40279517). In this meta-analysis, the primary diagnosis accuracy of the best model in each study was scattered from 25% to 97.8% (PMID 40279517). The authors concluded that while the accuracy of language models does not yet match that of healthcare professionals, they can become good assistive tools if used cautiously (PMID 40279517). A score that surges far ahead on one benchmark and results that do not clearly part ways with healthcare professionals when multiple studies are pooled exist side by side.

Left: share correct on a 2026 sequential diagnosis benchmark (controlled benchmark setting) — New England Journal of Medicine clinicopathological cases, AI system 83.2% vs generalist physicians 36.1%; emergency department records (MIMIC-IV-ED), AI system 80.3% vs 46.8%. Right: a meta-analysis of 50 studies, ratio of language models to clinicians in getting the diagnosis right — first-listed diagnosis 0.89 (CI 0.79–1.00), up to ten candidates 1.17 (CI 0.87–1.57), where 1 means the same as clinicians. The two panels use different measures

Figure: a diagram Paperis drew using only what the article says

The takeaway line is this: An AI's diagnostic score has meaning only when accompanied by a statement of what test sheet was used to measure it.

Putting the Same AI Next to a Doctor

If an AI outperforms doctors on a test, it is natural to expect that handing that AI to a doctor would raise the doctor's score. The method used to test this expectation is the randomized controlled trial. A randomized controlled trial is a test in which participants are randomly divided like a draw, with an intervention given to only one side and the results compared against the other side. Here, the intervention is making the AI available for use.

In late 2023, researchers at several US university hospitals randomly divided 50 doctors trained in family medicine, internal medicine, and emergency medicine (26 specialists, 24 residents) into two groups (PMID 39466245). One group could use a language model (GPT-4) in addition to their usual resources, the other group used only their usual resources, and both groups solved up to 6 case problems over 60 minutes (PMID 39466245; PMID 38559045). The case problems were written patient cases adapted from existing diagnostic reasoning tests (PMID 38559045), and grading checked whether the differential diagnosis was correct, whether supporting and opposing evidence were adequately provided, and what the next test should be (PMID 39466245). The median diagnostic reasoning score was 76% for the group that used the language model and 74% for the group that used only usual resources, and the adjusted difference was 2 percentage points, which was not statistically meaningful (PMID 39466245).

In an exploratory analysis of the same study, the language model was made to solve the problems alone, scoring 16 percentage points higher than the group of doctors who used only usual resources (PMID 39466245). The confidence interval was wide, ranging from 2 to 30 percentage points (PMID 39466245). The researchers noted that the AI alone scored higher than both doctor groups, and concluded that for doctor-AI collaboration to pull its weight, advancements are needed on both the technology and workforce fronts (PMID 39466245). In this test, the score of the doctor with the AI by their side did not rise as much as the score of the AI alone. This result came from a test where 50 US doctors solved written case problems, and was not measured in a clinic seeing actual patients.

Three scores — 2023 US randomized trial, median diagnostic reasoning scores of 50 physicians solving written case vignettes. Physicians with usual resources only: 74%; physicians with the language model alongside: 76% (difference of 2 percentage points, not statistically significant). For the language model alone there is no score bar: only its exploratory-analysis difference of 16 percentage points above the usual-resources group, with a confidence interval of 2 to 30 points

Figure: a diagram Paperis drew using only what the article says

A study that re-ran this test in the UK yielded slightly different numbers. It was a design where 22 UK doctors solved 4 case problems and were allowed to use a language model on only 2 of them, and when the language model could be used, the doctors' score was 74.3%, which was higher than the 65.7% when using only usual resources (PMID 42053372). Even so, the score of the language model alone was on average 21.3 percentage points higher than the doctors using the language model (PMID 42053372). In management reasoning asking for treatment and testing plans, in a randomized controlled trial where 92 doctors solved 5 species of case problems in a simulated setting, doctors using GPT-4 scored 6.5% higher, took 119.3 seconds longer per case, and there was no meaningful difference between doctors using GPT-4 and GPT-4 alone (PMID 39910272). The researchers noted that these results should be verified in actual clinical practice (PMID 39910272).

Pooling multiple individual tests reveals the doctors' benefit more clearly. In the aforementioned meta-analysis of 50 papers, healthcare professionals aided by a language model were 1.13 times (confidence interval 1.00 to 1.27) more likely to get their first-choice diagnosis right compared to unassisted professionals, and 1.11 times (confidence interval 1.01 to 1.23) more likely when including up to the third candidate (PMID 41652180). The authors of this meta-analysis concluded that due to methodological flaws in the included studies, rigorous evaluation in real-world environments is needed before introducing them into clinical practice (PMID 41652180). Individual trials alternate between "barely rose" and "rose slightly," but when aggregated, the data points in the direction of the assisted side coming out ahead.

There are also results where the AI clearly boosted doctor scores. In a study where 20 doctors read 302 difficult journal case reports with already known diagnoses, doctors aided by a language model tuned for diagnostic reasoning placed the correct answer within the top ten candidates 51.7% of the time, which was higher than 36.1% without help and 44.4% for doctors who used a search engine (PMID 40205049). In the same study, the language model alone scored 59.1%, which was higher than the 33.6% of doctors solving without help in the same comparison (PMID 40205049). On the other hand, another meta-analysis pooling 10 studies on human-AI collaboration noted that results were uncertain because only two studies dealt with diagnostic accuracy, and in studies comparing three groups, the combination of human and AI was not always better than AI alone (PMID 41606089).

Overlapping multiple tests paints a single picture. Doctors with an AI by their side generally performed similarly to or better than when they were alone (with the spread varying widely across studies), and in some case problem tests, fell short of the score of the AI alone.

Not Asking, and Following Along

In a British replication study, doctors paired with an AI asked the language model directly in 30% of the case questions (PMID 42053372). The researchers interpreted doctors' underutilization of the AI as a contributor to the score gap with the AI alone, and reported that the magnitude of the benefit gained by individual doctors varied widely when they were allowed to use the language model (PMID 42053372). Their conclusion was that doctors should be trained to slot this tool into their own train of thought, and that systems should be designed so that using it in this manner remains the default rather than an optional choice (PMID 42053372). If you do not ask the AI sitting next to you, the AI's proficiency does not transfer over to the doctor's score.

There is a problem in the opposite direction, too. Automation bias is the tendency to lean excessively on the answer produced by a machine and follow even the wrong answers. A randomized controlled trial held in Pakistan in 2025 divided 44 doctors who had received 20 hours of training in critically evaluating AI into two groups (PMID 42932681). Both groups solved six case problems with the option to view diagnostic suggestions from a language model (ChatGPT-4o) whenever they wished; one group was given error-free suggestions, while the other was given suggestions with deliberate errors introduced in three out of the six cases (PMID 42932681). The diagnostic reasoning accuracy for the group given suggestions mixed with errors was 73.3%, which was 14.0 percentage points lower by adjusted difference than the 84.9% of the group given error-free suggestions, and the proportion of correctly identifying the single most likely diagnosis was 76.1% versus 90.5% (PMID 42932681). The researchers concluded that the doctors exhibited substantial automation bias even under conditions where they received AI training and consulted voluntarily, viewing this as a safety risk that requires rigorous verification systems and regulatory safeguards before widespread adoption in clinical practice (PMID 42932681).

Automation bias also appears when the AI is helpful. In an experiment where 40 clinicians diagnosed cruciate ligament tears across 200 knee magnetic resonance images, accuracy rose from 87.2% to 96.4% with the AI's help, but 45.5% of the remaining mistakes in readings aided by the AI were errors where they followed the AI, and this phenomenon appeared in clinicians of all experience levels (PMID 37561535). In a wound diagnosis experiment involving 210 participants, the more useful the system was perceived to be, the higher the rate of agreeing with incorrect AI recommendations, and those without specialized training in that field were more vulnerable to this bias (PMID 39234734). The authors of this study wrote that those who stand to gain the most from AI are also those most vulnerable to this bias (PMID 39234734).

The tendency to follow the AI pays off when the AI is right. In a study where 30 readers found lung cancer nodules in 120 chest X-ray images, readers given a high-accuracy AI improved their detection performance more than those given a low-accuracy AI, and the proportion of changing their original reading after seeing the AI's suggestion was also higher at 67% (224 out of 334 cases) compared to 59% (229 out of 386 cases) (PMID 37367443). In a study simulating screening pathways using Australian mammography data, automation bias degraded performance in systems where multiple readers read together, but actually improved performance for solo readers (PMID 39214982). Even if the tendency to follow along is the same, the direction of the outcome changes depending on how accurate the AI is and who is reading within what system.

How the AI presents its answers also changes the outcomes. In a randomized experiment where radiologists evaluated 2,020 reading cases, accuracy was 12.2% higher than the condition without AI help when they received step-by-step written explanations of the reasoning process the language model used to reach its diagnosis, 7.2% higher than receiving just the diagnosis name, and 9.7% higher than receiving a list of candidates (PMID 42026140). The authors stated that these results came from a controlled setting using case problems, and interpreted that explaining the reasoning allows doctors to spot and correct AI errors (PMID 42026140). Yet, in other studies targeting 12 knee magnetic resonance readers and 44 electrocardiogram readers, explanations showing the basis for the AI's judgment were ineffective or sometimes even harmful, and the sequence of showing the AI's answer first yielded higher accuracy than the sequence where humans judged first (PMID 36990586). Whether explanations help or hinder remains mixed across studies.

When stacking the studies in this section onto the desk of a doctor facing a single diagnosis, three forks in the road appear. First is the sequence. A doctor can establish a differential diagnosis on their own first or look at the AI's answer first; in one reading study, the sequence of looking at the AI's answer first was more accurate (PMID 36990586). Second is whether to ask or not. British doctors asked the AI for only 30% of their questions (PMID 42053372). Third is whether to follow the answer received. In the knee reading experiment aided by AI, accuracy rose from 87.2% to 96.4% (though 45.5% of remaining mistakes were errors following the AI) (PMID 37561535), and in case problems mixed with intentionally wrong answers, even doctors who received AI training saw their accuracy drop by 14.0 percentage points (PMID 42932681). Because each of the three forks was measured in different studies, how they intersect when all three happen together in a single doctor has not yet been measured.

Outcomes are shaped just as much by what a doctor asks the AI and how they accept that answer as they are by the AI's accuracy rate.

What Happened in the Screening Room

At four mammography screening centers in southwestern Sweden, 105,934 eligible women were randomly assigned to two arms between April 2021 and December 2022 (PMID 39904652). One arm was read by two radiologists according to standard practice, while in the other arm, when a woman’s images were taken, an AI first assigned a risk score from 1 to 10 across ten levels (PMID 37541274). Scores of 1 to 9 were read by a single radiologist, the highest score of 10 was read by two, and for images with scores of 8 to 10, the AI highlighted suspicious regions (PMID 37541274). The scores were visible to the radiologists reading all images (PMID 37541274). This AI is an image-reading program and is a different kind from the language models covered earlier. This trial is MASAI.

The cancers found in the AI-assisted arm were 6.4 per 1,000 people, which was higher than the 5.0 per 1,000 in the standard arm, representing a ratio of 1.29 (PMID 39904652). The recall rate for non-cancers did not increase significantly, and the number of reads performed by radiologists decreased by 44.2% (PMID 39904652). The additional cancers found included cancers that had spread to surrounding tissue as well as early-stage cancers that remained confined to where they first formed. These cancers numbered 68 versus 45, and about half of the increased share was on the higher-malignancy side (PMID 39904652).

Finding more cases in screening alone is not enough to demonstrate a benefit. We need to see whether cancers missed by screening come to light later. Interval cancers are cancers diagnosed after a screening is deemed normal, before the next scheduled screening arrives. The rate of interval cancers, which was the primary evaluation result of MASAI, was 1.55 per 1,000 people in the AI-assisted arm and 1.76 in the standard arm, showing that AI-assisted screening was not inferior according to a pre-specified margin (PMID 41620232). The ratio between the two arms was 0.88 and the confidence interval was 0.65 to 1.18 (PMID 41620232). The proportion of actual cancers caught by screening was 80.5% in the AI-assisted arm and 73.8% in the standard arm, and the proportion of cancer-free individuals deemed normal was 98.5% in both arms (PMID 41620232). The researchers interpreted that introducing AI-assisted mammography screening into clinical practice can be considered (PMID 41620232).

In this trial randomly dividing roughly 100,000 Swedish women, the arm where AI triaged the readings and highlighted suspicious regions before radiologists read them found more cancers, reduced the radiologists' reading volume, and showed a non-inferior interval cancer rate. Certain qualifications remain regarding the same evidence. A meta-analysis gathering literature from 2015 to 2025 noted that the cancer detection ratio combining three trials—one randomized trial (MASAI) and two matched prospective trials—was 1.13 (confidence interval 0.83 to 1.55), while looking at MASAI alone showed 1.29 (confidence interval 1.09 to 1.51) (PMID 42793830). The same meta-analysis noted that larger and more precise benefits came from studies without random assignment, which is likely an artifact of confounding, making randomized evidence the standard to rely on (PMID 42793830). This meta-analysis concluded that a decade of evidence supports using AI as a second reader and triage tool in organized screening, but does not support autonomous reading replacing radiologists (PMID 42793830). This meta-analysis gathered literature up to the end of 2025 and noted that interval cancer and mortality endpoints were still awaited, while the aforementioned MASAI interval cancer results appeared in 2026, and mortality endpoints have still not emerged (PMID 42793830; PMID 41620232). Benefits and qualifications stand side by side in colonoscopy as well. Adenomas are polyps that can grow into cancer, and in a meta-analysis pooling randomized trials where the control group's adenoma detection baseline was already high, AI's lesion-highlighting cut missed adenomas by nearly half (0.473-fold pooled across 6 studies) and raised the proportion of exams finding one or more adenomas to 1.137-fold pooled across 28 studies, but the benefit of finding more advanced adenomas was not statistically confirmed (PMID 43050091).

The domains where AI showed benefits in randomized trials involving real people were tasks with a narrow scope and clear boundaries for the correct answer, such as finding lesions within images. A 2026 review article similarly summarized that the most reliable advances occur in tasks with clear endpoints and narrowly defined problems (PMID 42825240).

The Day the Machine Was Turned Off

Four endoscopy centers in Poland introduced AI to find colon polyps in late 2021, and afterwards, colonoscopies with AI turned on and off were randomly assigned depending on the examination date (PMID 40816301). The researchers looked back at the records of 1,443 patients who underwent colonoscopies without AI during the 3 months before introducing AI and the 3 months after (PMID 40816301). The rate of finding at least one adenoma dropped from 28.4% before the introduction of AI (226 out of 795 people) to 22.4% after (145 out of 648 people) (PMID 40816301). The researchers interpreted that continued exposure to AI could drop performance in exams done without AI, suggesting a negative effect on the behavior of the endoscopist (PMID 40816301).

This phenomenon has a name. Deskilling is when a person's original skill becomes dull while relying on automation. However, the Polish study is an observational study looking back at what already happened, and observational studies lack random assignment, making it impossible to confirm the cause. A 2026 review examining the evidence that AI can drop a doctor's performance noted that the evidence came from single studies or studies with design limitations, and it remains unknown how large and long-lasting the effect is (PMID 42984954).

Deskilling is a story about doctors who already have the skill. Concerns have also been raised that doctors who rely on AI from the early stages of training may fail to acquire that skill in the first place, but the authors themselves who put forward this perspective stated that direct evidence from medical training does not yet exist (PMID 42174254). A doctor's performance with AI turned on and their performance with it turned off are different numbers.

AI Facing Patients, and What We Still Don't Know

AMIE, a language model-based AI built to converse directly with patients and reach a diagnosis, first went through an exam where it chatted in text with actors playing the role of patients before ever standing before a real patient (PMID 40205050). In a randomized, double-blind crossover study comparing it with 20 primary care physicians using 159 cases gathered from Canada, the UK, and India, this AI had higher diagnostic accuracy and outperformed physicians in 30 out of 32 items evaluated by specialists (PMID 40205050). The authors noted the limitation that physicians consulted via real-time text chat, which is unfamiliar in clinical practice, and stated that the results should be interpreted with caution (PMID 40205050).

In the next stage, 98 patients who booked primary care appointments exchanged symptoms with the same AI up to five days before the visit and then met a physician (PMID 42849491). In this single-center prospective study without a control group, a physician safety supervisor watched all conversations, and there was never an instance where a conversation had to be stopped according to pre-determined criteria (PMID 42849491). The supervisor recorded one instance of fabricating facts and supplemented clinical information five times (PMID 42849491). Out of 44 cases where the primary care physician read the conversation transcript in advance, the physicians answered that it helped them prepare for the consultation in 33 cases (PMID 42849491). Funding was provided by Alphabet, and the research team defined this result as a first step confirming the safety, quality, and user acceptance of AI-assisted conversations under supervision, noting that further research is needed (PMID 42849491). The primary outcomes measured by this study were conversation safety, conversation quality, and patient and physician experience, while diagnostic accuracy was not on that list (PMID 42849491).

Looking at the clinic as a whole, more things are unmeasured than measured. A 2026 review summarized that the latest language models achieve performance comparable to or surpassing physicians in simulated diagnostic reasoning tasks, while also summarizing that less than 5% of AI and machine learning devices approved by the US Food and Drug Administration have received peer-reviewed evaluations, and prospective randomized trials remain rare (PMID 42825240). The same review noted that the latest models continue to fail in gauging how confident they should be in their own answers, human-AI collaboration is not yet optimal, and the risks of automation bias and deskilling are coming to light (PMID 42825240). A review examining 9 primary studies selected by an author evaluated that among them, only one randomized trial compared physicians using AI versus not using AI in open diagnostic reasoning using case vignettes (PMID 42984954).

Even if a test score is high, its limitations remain. In 150 emergency room appendicitis case vignettes intentionally packed with difficult cases, the top-performing AI system scored 95.3%, slightly exceeding the 92.7% consensus of general surgeons, and in systems coupled with web searches, unique failures such as fabricated answers brought on by the search were also observed (PMID 42500998). The authors of this study wrote that vignette-based comparisons should not be interpreted as evidence of readiness for clinical use, and that rather than leaving emergency triage entirely to AI, it should be evaluated by diagnosis under supervision (PMID 42500998). One review concluded that the benefit cannot be inferred from the accuracy of AI models alone, and that the performance of physicians using AI must be measured directly against those not using it (PMID 42984954).

Returning to the initial question, the answer becomes visible when the question is divided into two. Do AIs score higher than doctors on tests? Some benchmarks and vignette exams say yes, but meta-analyses pooling studies did not clearly separate the two. Does an AI-adopted screening room find more cancers? One randomized trial of mammography screening says yes. Whether that screening even reduces mortality, and whether patients receive more accurate diagnoses in the broader clinic, still awaits measurement. The score of the AI alone, the score of the physician using the AI, and the score of the physician when the machine is turned off are three different numbers, and the answer in the clinic emerges only when all three are measured.

Where to Look Next

How scores inside the test hall collapse outside it is covered next in "How AI Gets Things Wrong — The Test-Taker Who Stumbles Outside the Exam Hall". What language models are trained on and how they build sentences can be seen from the ground up in "How Guessing the Next Word Becomes a Conversation — The Minimum Foundation for Reading Language Models". What the proportion of catches and the proportion seen as normal in screenings actually tell us is continued in "What a Test Actually Tells You — An Introduction to Sensitivity, Specificity, and Pretest Probability", and whether the early-stage cancers that AI screenings find more of are always beneficial is followed up in "Is Finding It Early Always Better? — An Introduction to Overdiagnosis". The power and limitations of the trial design this article leans on are covered separately in "Is the Randomized Controlled Trial Really the 'Gold Standard'?".

Source Papers

  • "Sequential diagnosis with language models." (2026) — PMID 42912144 — AI systems in sequential diagnosis benchmarks: 83.2% (NEJM cases) and 80.3% (MIMIC-IV-ED) vs general practitioners at 36.1% and 46.8%, with authors themselves noting it is a controlled benchmark setting.
  • "Knowledge-Practice Performance Gap in Clinical Large Language Models: Systematic Review of 39 Benchmarks." (2025) — PMID 41325597 — Knowledge exams 84–90% vs practical tasks 45–69%, diagnosis 45–55%, safety 40–50%, showing that exam scores are insufficient and misleading proxy metrics.
  • "Large Language Model Performance and Clinical Reasoning Tasks." (2026) — PMID 41973425 — 21 language models and 29 cases, final diagnosis failure rate below 0.40 vs differential diagnosis above 0.80.
  • "Independent and collaborative performance of large language models and healthcare professionals in diagnosis and triage." (2026) — PMID 41652180 — 50-study meta-analysis, first diagnosis for language model/healthcare professional 0.89 and top ten 1.17, healthcare professional with language model help first diagnosis 1.13 times (1.00–1.27) and up to third 1.11 times (1.01–1.23), highlighting the need for real-world evaluations.
  • "Comparing Diagnostic Accuracy of Clinical Professionals and Large Language Models: Systematic Review and Meta-Analysis." (2025) — PMID 40279517 — 30 studies and 4,762 cases, many with high risk of bias (main cause: cases where the diagnosis was already known), showing accuracy is not yet on par with healthcare professionals.
  • "Large Language Model Influence on Diagnostic Reasoning: A Randomized Clinical Trial." (2024) — PMID 39466245 — 50 U.S. physicians randomized, 76% vs 74% (2%p difference, not significant), with an exploratory analysis showing language model alone +16%p (2–30).
  • "Influence of a Large Language Model on Diagnostic Reasoning: A Randomized Clinical Vignette Study." (2024) — PMID 38559045 — Preprint of the same trial, using GPT-4, 60 minutes, up to 6 cases.
  • "Human-AI collaboration in clinical reasoning: a UK replication and interaction analysis." (2026) — PMID 42053372 — 22 UK physicians, 74.3 vs 65.7% with language model use, language model alone 21.3%p higher, only 30% of questions asked directly, requiring training and design.
  • "GPT-4 assistance for improvement of physician performance on patient care tasks: a randomized controlled trial." (2025) — PMID 39910272 — 92 physicians and 5 simulated care-management cases, GPT-4 use +6.5%, taking 119.3 seconds longer per case, no difference from GPT-4 alone.
  • "Towards accurate differential diagnosis with large language models." (2025) — PMID 40205049 — 20 physicians and 302 cases, supported physicians top-ten correct answer rate 51.7% vs unsupported 36.1% and search 44.4%, model alone 59.1%.
  • "Human-large language model collaboration in clinical medicine: a systematic review and meta-analysis." (2026) — PMID 41606089 — 10 studies, uncertainty due to only 2 diagnostic accuracy studies, human + AI not always better than AI alone in three-group comparisons.
  • "Automation Bias in Large Language Model-Assisted Diagnostic Reasoning among Physicians Trained in AI Literacy - A Randomized Clinical Trial." (2026) — PMID 42932681 — 44 Pakistani physicians (20 hours of AI training), error-suggestion group 73.3% vs 84.9% (−14.0%p), first diagnosis 76.1 vs 90.5%, substantial automation bias.
  • "Artificial intelligence suppression as a strategy to mitigate artificial intelligence automation bias." (2023) — PMID 37561535 — 40 physicians and 200 knee MRIs, accuracy 87.2 to 96.4%, 45.5% of remaining errors due to automation bias, regardless of experience.
  • "Automation Bias in AI-Decision Support: Results from an Empirical Study." (2024) — PMID 39234734 — 210 wound diagnosis cases, perceived usefulness ↑ → incorrect agreement ↑, non-experts most vulnerable.
  • "The effect of medical explanations from large language models on diagnostic accuracy in radiology." (2026) — PMID 42026140 — 2,020 radiologist evaluations, step-by-step reasoning explanations +12.2% over unsupported, in a controlled case-vignette setting.
  • "Rams, hounds and white boxes: Investigating human-AI collaboration protocols in medical diagnosis." (2023) — PMID 36990586 — 12 knee MRI and 44 ECG cases, explanations ineffective or harmful ("white-box paradox"), showing higher accuracy when AI is shown first.
  • "Screening performance and characteristics of breast cancer detected in the Mammography Screening with Artificial Intelligence trial (MASAI)..." (2025) — PMID 39904652 — 105,934 randomized in Sweden, cancer detection 6.4 vs 5.0/1000 (1.29 times), no significant increase in false positives, reading volume 44.2% ↓, in-situ early cancers 68 vs 45.
  • "Artificial intelligence-supported screen reading versus standard double reading in the MASAI..." (2023) — PMID 37541274 — Safety interim analysis, classifying single/double reading by AI risk score and highlighting suspicious areas.
  • "Interval cancer, sensitivity, and specificity comparing AI-supported mammography screening with standard double reading without AI in the MASAI study..." (2026) — PMID 41620232 — Interval cancer 1.55 vs 1.76/1000 non-inferior (0.88, 0.65–1.18), sensitivity 80.5 vs 73.8%, specificity 98.5% identical, making adoption worth considering.
  • "Ten Years of Artificial Intelligence in Screening Mammography: A Systematic Review and Meta-Analysis..." (2026) — PMID 42793830 — Sum of 3 randomized and paired prospective studies 1.13 (0.83–1.55), MASAI alone 1.29 (1.09–1.51), large gains stem from non-randomized studies (confounding patterns) making randomized evidence the benchmark, functioning as a second reader rather than a replacement, awaiting interval cancer and mortality metrics based on literature through late 2025.
  • "Does computer-aided detection still add value in high-performing colonoscopy?..." (2026) — PMID 43050091 — 28 randomized trials adenoma detection 1.137 times, advanced adenoma benefit unconfirmed in high-detection settings.
  • "State of clinical AI in 2026." (2026) — PMID 42825240 — Less than 5% of FDA-cleared AI devices peer-reviewed, prospective RCTs rare, failure to calibrate confidence, poor collaboration, most reliable progress in narrow tasks.
  • "Endoscopist deskilling risk after exposure to artificial intelligence in colonoscopy: a multicentre, observational study." (2025) — PMID 40816301 — Observation across 4 centers in Poland, adenoma detection rate during exams without AI dropped from 28.4 to 22.4%, suggesting a negative impact.
  • "Augmentation or erosion? Artificial intelligence and the transformation of clinical reasoning: a critical narrative review." (2026) — PMID 42984954 — Randomized trials of open diagnostic reasoning limited to 1 case study, solutions based on single-study design limitations, model accuracy cannot infer benefits—direct measurement required.
  • "AI-induced never-skilling in medical education." (2026) — PMID 42174254 — Concerns about residents "failing to acquire skills from the start," though direct evidence in medical training is lacking.
  • "Conversational diagnostic artificial intelligence in ambulatory primary care: a prospective feasibility study." (2026) — PMID 42849491 — Single center, single arm, 98 patients, 0 safety stops, 1 fabricated detail, help in 33 out of 44 cases where chat logs were read beforehand, Alphabet research funding, requiring further research.
  • "Diagnostic Ambiguity in Appendicitis: Comparing Failure Modes of Large Language Models and Surgeons." (2026) — PMID 42500998 — Case-vignette comparisons must not be interpreted as evidence for clinical readiness.
  • "Effect of Human-AI Interaction on Detection of Malignant Lung Nodules on Chest Radiographs." (2023) — PMID 37367443 — 30 readers and 120 chest X-rays, high-accuracy AI boosted performance further and led readers to change answers based on AI suggestions 67% vs 59% of the time.
  • "Comparison of AI-integrated pathways with human-AI interaction in population mammographic screening for breast cancer." (2024) — PMID 39214982 — Australian simulation data, automation bias reduces performance in multi-reader systems while improving it for single readers.
  • "Towards conversational diagnostic artificial intelligence." (2025) — PMID 40205050 — Text consultation with standardized patients, 159 cases and 20 primary care physicians, AMIE showed higher diagnostic accuracy and superior performance in 30 out of 32 items, limited by unfamiliar text-chat format.
Sources (30)
  • Sequential diagnosis with language modelsPMID 42912144
  • Knowledge-Practice Performance Gap in Clinical Large Language Models: Systematic Review of 39 BenchmarksPMID 41325597
  • Large Language Model Performance and Clinical Reasoning TasksPMID 41973425
  • Independent and collaborative performance of large language models and healthcare professionals in diagnosis and triagePMID 41652180
  • Comparing Diagnostic Accuracy of Clinical Professionals and Large Language Models: Systematic Review and Meta-AnalysisPMID 40279517
  • Large Language Model Influence on Diagnostic Reasoning: A Randomized Clinical TrialPMID 39466245
  • Influence of a Large Language Model on Diagnostic Reasoning: A Randomized Clinical Vignette StudyPMID 38559045
  • Human-AI collaboration in clinical reasoning: a UK replication and interaction analysisPMID 42053372
  • GPT-4 assistance for improvement of physician performance on patient care tasks: a randomized controlled trialPMID 39910272
  • Towards accurate differential diagnosis with large language modelsPMID 40205049
  • Human-large language model collaboration in clinical medicine: a systematic review and meta-analysisPMID 41606089
  • Automation Bias in Large Language Model-Assisted Diagnostic Reasoning among Physicians Trained in AI Literacy - A Randomized Clinical TrialPMID 42932681
  • Artificial intelligence suppression as a strategy to mitigate artificial intelligence automation biasPMID 37561535
  • Automation Bias in AI-Decision Support: Results from an Empirical StudyPMID 39234734
  • The effect of medical explanations from large language models on diagnostic accuracy in radiologyPMID 42026140
  • Rams, hounds and white boxes: Investigating human-AI collaboration protocols in medical diagnosisPMID 36990586
  • Screening performance and characteristics of breast cancer detected in the Mammography Screening with Artificial Intelligence trial (MASAI)PMID 39904652
  • Artificial intelligence-supported screen reading versus standard double reading in the MASAIPMID 37541274
  • Interval cancer, sensitivity, and specificity comparing AI-supported mammography screening with standard double reading without AI in the MASAI studyPMID 41620232
  • Ten Years of Artificial Intelligence in Screening Mammography: A Systematic Review and Meta-AnalysisPMID 42793830
  • Does computer-aided detection still add value in high-performing colonoscopy?PMID 43050091
  • State of clinical AI in 2026PMID 42825240
  • Endoscopist deskilling risk after exposure to artificial intelligence in colonoscopy: a multicentre, observational studyPMID 40816301
  • Augmentation or erosion? Artificial intelligence and the transformation of clinical reasoning: a critical narrative reviewPMID 42984954
  • AI-induced never-skilling in medical educationPMID 42174254
  • Conversational diagnostic artificial intelligence in ambulatory primary care: a prospective feasibility studyPMID 42849491
  • Diagnostic Ambiguity in Appendicitis: Comparing Failure Modes of Large Language Models and SurgeonsPMID 42500998
  • Effect of Human-AI Interaction on Detection of Malignant Lung Nodules on Chest RadiographsPMID 37367443
  • Comparison of AI-integrated pathways with human-AI interaction in population mammographic screening for breast cancerPMID 39214982
  • Towards conversational diagnostic artificial intelligencePMID 40205050

This article is AI-generated from research papers for general information and does not replace professional advice. Spotted an error? Let us know

© 2026 Neokuns (Paperis). Sharing links is welcome; republishing the full text requires prior permission.