From Biostatistics to AI: A Gentle Bridge for Clinicians

Here is something that most biostatistics courses never mention: the concepts doctors learned to understand diagnostic tests are almost exactly the concepts used to evaluate AI models. The terminology differs. The context differs. But the underlying questions — how often does this tool correctly identify disease? How often does it raise false alarms? How should its performance be weighted against prevalence in the population? — are the same questions clinicians have been answering since the first diagnostic test was validated.

This matters because the gap between “understanding statistics” and “understanding AI” is far smaller than it appears from the outside. A clinician who has read a diagnostic accuracy study, interpreted a receiver operating characteristic curve, and thought carefully about how prevalence affects the positive predictive value of a test already has most of the conceptual vocabulary they need to read an AI validation paper.

To make this concrete, consider a single clinical scenario that will run through the entire article: an AI system that analyses chest X-rays and outputs a probability score indicating whether pneumonia is present. Call it a chest X-ray pneumonia detection tool. When it performs well, it flags patients who need further evaluation and antibiotic treatment. When it performs poorly, it either misses cases or raises false alarms that send clinicians chasing infections that are not there. How should a clinician evaluate whether this tool is worth trusting? The answer begins with concepts they already know.

The Concepts You Already Know — and Their AI Equivalents

Start with sensitivity. In diagnostic test terminology, sensitivity is the proportion of patients with the disease who test positive — the tool’s ability to catch true cases. A highly sensitive test rarely misses disease; a low-sensitivity test lets cases slip through.

In AI terminology, the same concept is called recall (sometimes also called the “true positive rate”). For the chest X-ray pneumonia tool: of all the chest X-rays that genuinely show pneumonia, what proportion does the AI flag as positive? A recall of 90% means the AI catches nine out of ten true pneumonia cases. A recall of 60% means it misses four in ten — which, for a condition that kills patients who are not treated, is a serious clinical concern.

The terminology change is superficial. Sensitivity and recall are the same calculation from the same two-by-two table: true positives divided by the sum of true positives and false negatives.

Specificity — the proportion of patients without disease who test negative — maps onto a concept AI literature sometimes calls selectivity, though “specificity” itself is also used in clinical AI papers. For the chest X-ray tool: of all the X-rays that do not show pneumonia, what proportion does the AI correctly identify as negative? Low specificity means the AI raises frequent false alarms — patients without pneumonia are flagged as positive, leading to unnecessary antibiotics, patient anxiety, and clinician time spent chasing a diagnosis that is not there.

Positive predictive value (PPV) and negative predictive value (NPV) carry over entirely unchanged. PPV is the probability that a patient the AI flags as positive actually has pneumonia. NPV is the probability that a patient the AI flags as negative genuinely does not. And critically, both values are dependent on prevalence in exactly the same way they are for any diagnostic test. A chest X-ray pneumonia tool evaluated in a tertiary hospital respiratory unit — where a large proportion of patients presenting with respiratory symptoms genuinely have pneumonia — will show very different PPV and NPV values when deployed in a primary care setting where most patients with cough have viral infections.

This is not a novel insight for clinicians. It is the same prevalence-dependence that governs the PPV of any screening test. The lesson for clinical AI is identical: performance metrics reported in a high-prevalence study population may be substantially optimistic when the tool is deployed in a lower-prevalence setting.

The ROC curve is, quite literally, the same graph in clinical statistics and AI. The receiver operating characteristic curve plots sensitivity on the y-axis against one-minus-specificity (the false positive rate) on the x-axis across all possible operating thresholds. The area under the curve (AUC) is a summary measure of overall discriminative performance: a perfect classifier achieves an AUC of 1.0; a random classifier achieves 0.5. The chest X-ray pneumonia tool’s AUC tells a clinician how well it distinguishes pneumonia from non-pneumonia across all possible decision thresholds — before any specific operating point has been chosen.

AI papers report AUC exactly as diagnostic test papers do, and the interpretation is the same. An AUC of 0.95 is impressive. An AUC of 0.75 should raise questions about whether the tool offers meaningful clinical advantage over clinical assessment alone. And the limitations of AUC as a single summary metric — which can look good even when the tool performs poorly at the operating point that matters clinically — are the same limitations that have been discussed in diagnostic test literature for decades.

The p-value, familiar from clinical research, appears less frequently in AI papers than it does in traditional clinical statistics. The trend in AI evaluation is toward reporting confidence intervals on performance metrics directly — which is arguably more informative. A chest X-ray AI paper that reports AUC 0.93 (95% CI 0.90–0.96) is telling the clinician more than one that reports p<0.001 for the AUC being greater than 0.5.

The table below summarises these equivalences:

The two-by-two table that generated every sensitivity, specificity, PPV, and NPV a clinician has ever calculated is the same two-by-two table that underpins the performance metrics of the chest X-ray pneumonia AI. The calculation is identical. Only the names have changed.

The New Concepts — Explained Through Clinical Analogy

There are concepts in AI that do not have a direct one-to-one mapping to clinical statistics, but which can be made intuitive through clinical analogies. Each one is worth understanding before a clinician reads an AI validation study.

Training data is the dataset on which the AI learned. Think of it as the study population — the patients whose chest X-rays, along with their confirmed pneumonia diagnoses, were used to teach the AI to recognise the pattern. For the chest X-ray tool: the AI was shown tens of thousands of X-rays, each labelled as “pneumonia” or “no pneumonia” by radiologists. It learned to associate certain pixel patterns with the “pneumonia” label.

The same biases that affect a study population affect training data. If the training X-rays were all acquired on a single type of imaging equipment, the AI may not generalise to different equipment. If the training population was predominantly one demographic group, the AI may perform worse on patients from other groups. If the pneumonia cases in the training set were mostly severe (because they came from a tertiary hospital respiratory unit), the AI may miss mild or atypical presentations that appear more commonly in primary care.

CheXNet, a chest X-ray AI system developed at Stanford and published in 2017, claimed performance exceeding radiologists at detecting pneumonia on the ChestX-ray14 dataset. Subsequent independent evaluations demonstrated that the system’s performance dropped substantially when tested on X-rays from external institutions with different patient populations and imaging equipment. The training data had shaped the model’s capabilities more than its published AUC suggested.

Overfitting is what happens when an AI model performs exceptionally well on its training data but fails to generalise to new patients. The model has, in effect, memorised the training data rather than learning the underlying pattern. The clinical analogy is a drug that showed excellent efficacy in the original trial population but failed in post-marketing real-world use — the trial had enrolled such a homogeneous, highly selected population that the drug’s performance in those patients told clinicians little about its performance in the full spectrum of patients who would ultimately receive it.

Validation set and test set are the AI equivalents of external validation studies. When building the chest X-ray AI, researchers set aside a portion of their labelled X-rays — the validation set — that the model was never shown during training. After training is complete, they measure the model’s performance on the validation set to check whether it has generalised or overfit. Some studies go further, testing the final model on an entirely separate test set from a different hospital or time period — the equivalent of an external validation study, and the strongest evidence that the model’s performance is real rather than an artifact of the training data.

Data drift is a concept without a clean clinical statistics equivalent, but it has a clinical parallel that makes it intuitive. Imagine a reference range for a common laboratory test that was established in a cohort studied thirty years ago — when the general population had different body composition, different medication use, and a different prevalence of comorbidities. Applied to today’s patients, that reference range might misclassify a meaningful proportion of normal values as abnormal, or vice versa. Something similar happens to AI models over time: the patient population changes, disease prevalence shifts, imaging equipment is upgraded, clinical documentation practices evolve. The model was trained on data from a specific time and place, and as reality drifts away from that training distribution, its performance can degrade — silently, without any alert to the clinician using it.

Data drift is a documented problem in deployed clinical AI systems and a reason why ongoing performance monitoring, rather than one-time validation, is an appropriate expectation for AI tools in clinical use.

Where the Analogy Breaks Down — The Honest Part

The mapping between diagnostic test statistics and AI model evaluation is genuine and useful, but it has limits. Understanding where the analogy breaks down is as important as understanding where it holds.

AI models can degrade silently over time. A blood test’s biochemical mechanism does not change from year to year. A chest X-ray pneumonia AI trained in 2020 may perform differently in 2025 because the patient population has changed, because new pathogen variants are presenting differently, or because the imaging equipment has been upgraded. Diagnostic tests have fixed analytical characteristics that remain stable (barring changes in assay methodology). AI models have performance characteristics that are conditional on the population and context in which they are deployed, and that population can change.

AI performance is population-specific in ways that many laboratory tests are not. A serum creatinine value is interpretable across patient populations with reference to well-established relationships between creatinine clearance and estimated GFR. The chest X-ray pneumonia AI’s performance may vary substantially between a district general hospital in one country and a tertiary referral centre in another — because the case mix, disease severity spectrum, patient demographics, and imaging acquisition parameters all differ. This does not mean AI is uniquely unreliable. It means the question “was this validated on patients like mine?” matters for AI in ways that it does not always matter for established biochemical assays.

The “why” is often opaque. When a troponin assay gives a high value, the clinician knows the biochemical mechanism: myocardial cell membrane damage has released intracellular troponin into the circulation, detectable via immunoassay. The mechanism is understood, and that mechanistic understanding allows the clinician to reason about confounders — renal impairment, myocarditis, type 2 MI — intelligently. When the chest X-ray AI gives a high pneumonia probability score, the clinician may not be able to determine which features of the image drove that prediction. The model’s reasoning is internal and, in many modern deep learning systems, not fully recoverable even by the engineers who built it. This opacity means that anomalous AI outputs are harder to interrogate than anomalous test results, and that the clinician’s ability to exercise critical judgement about the output is more limited.

Clinical AI requires more scrutiny of the training population than most diagnostic tests, and ongoing performance monitoring in a way that established biochemical tests rarely require. The analogy between AI evaluation and diagnostic test evaluation is useful, but AI is not simply a new kind of diagnostic test. It is a different kind of tool that borrows the same evaluation framework and adds new responsibilities.

From Reading a Diagnostic Test Paper to Reading an AI Study

A clinician who knows how to read a diagnostic accuracy study can, with modest additional knowledge, read a clinical AI validation paper. The framework is the same. The additional knowledge concerns a few AI-specific reporting items and a few AI-specific red flags.

CONSORT-AI is the AI adaptation of the CONSORT checklist used for reporting randomised controlled trials. Published simultaneously in Nature Medicine, the BMJ, and The Lancet in 2020, it specifies what information authors must report in clinical AI studies to allow readers to evaluate the evidence critically. It covers items specific to AI studies that traditional CONSORT does not require: the characteristics of the training data, whether the algorithm was modified after validation, how missing data was handled, what the model’s operating point was and how it was chosen, and whether the comparison to clinical standard of care was fair. Clinicians who regularly read AI research should know that this standard exists, and should treat papers that do not meet it with appropriate scepticism.

When reading a clinical AI paper about, say, a new version of the chest X-ray pneumonia tool, the questions to ask map onto familiar concepts with a few additions:

Training population demographics. Who were the patients in the training set? What was their demographic profile, geographic origin, disease severity distribution, and imaging equipment? The more the training population resembles the patient population in which the clinician intends to use the tool, the more confident the clinician can be that the published performance will replicate.

Validation population. Was the model validated on the same data it was trained on (internal validation only — weak evidence), on a held-out set from the same institution (better), or on data from a different institution in a different country (strongest evidence of generalizability)?

Comparison to clinical standard. Was the AI compared to a genuine clinical comparator — the existing diagnostic pathway, including clinical assessment and existing imaging review — or only to a gold-standard that is not available in routine clinical practice?

Performance on subgroups. Did the study report performance separately for relevant clinical subgroups — different age groups, sexes, disease severity levels, comorbidity profiles? A model that achieves excellent overall AUC but performs poorly in elderly patients with atypical presentations may be clinically problematic even if its headline numbers look good.

Red flags for low-quality AI studies. The following combination of features should substantially reduce a clinician’s confidence in an AI validation result: retrospective design only, single-centre dataset, no external validation, performance reported only as AUC without specifying the operating point (sensitivity and specificity at the threshold used clinically), and no comparison to current standard of care. Each of these limitations alone is manageable. In combination, they describe an AI paper whose published performance may tell a clinician very little about how the tool will perform in their hospital.

The single most important question remains the one borrowed directly from clinical evidence appraisal: “Was this validated on patients like mine?” For the chest X-ray pneumonia tool, this means: was the validation population similar in age, comorbidity burden, and disease prevalence to the population in which the clinician intends to use the tool? Was the imaging equipment similar? Was the tool validated in a clinical workflow that resembles the one it will be deployed in? If the answer to these questions is “I don’t know” or “no”, the published AUC provides limited assurance.

The question that changes everything when reading a clinical AI paper is the same question that changes everything when reading any diagnostic test study: “Would these results apply to my patients?” In AI, asking that question requires paying close attention to the training population, the validation population, and the clinical context in which the model was tested.

Closing: You Already Have the Foundation

The bridge from biostatistics to AI is shorter than it looks. A clinician who understands why prevalence affects predictive value, who knows what an ROC curve represents, and who has ever asked “was the study population like my patients?” is already equipped to begin critically appraising clinical AI.

The additional concepts — training data, overfitting, validation design, data drift — are extensions of familiar ideas rather than foreign ones. They require the same intellectual habits that good clinical appraisal always requires: attention to how the evidence was generated, scepticism about results that seem too good, and the discipline to ask uncomfortable questions about external validity.

The next module in this series moves into the raw material of clinical AI: Types of Clinical Data explores the different kinds of data that AI systems learn from — structured records, imaging, clinical text, genomics — and what the properties of each type mean for what AI can and cannot do with them.

For readers who want an overview of the full series and all six modules, the AI Foundations for Clinicians series page provides a complete map.

This article is part of the AI Foundations for Clinicians series, produced by MedAI Collective.