Consider two AI tools. Both claim to predict hospital readmission risk. Both were trained on “clinical data.” The marketing materials for each look nearly identical. But when you examine what they actually learned from, they are fundamentally different systems.
The first was trained on structured electronic health record data: lab values, vital signs, medication lists, and diagnostic codes. The second was trained on free-text discharge summaries — the narrative documents that senior clinicians write when a patient leaves hospital. Each summarises what happened clinically, what was found, what was decided, and what the plan is going forward.
Both are “clinical data.” They contain entirely different information. They have entirely different strengths and blind spots. An AI system trained on one does not behave like an AI system trained on the other — even if both produce a readmission risk score in similar formats.
This is the central insight of this article: the question “does this AI tool work?” is inseparable from the question “what data did it learn from?” Every claim an AI system makes is a pattern extracted from a dataset. Understanding the types of clinical data — and what each type captures, misses, and distorts — is how a clinician evaluates those claims.
Structured Clinical Data
The easiest data for computers to work with is data that has already been converted into a consistent, queryable format. In clinical settings, this means laboratory values, vital signs, medication records, and coded diagnoses.
A patient’s haemoglobin result of 9.2 g/dL recorded in an electronic health record is structured data. It has a consistent format, a defined unit, a timestamp, and a reference range. A computer can immediately compare it to the patient’s previous result, calculate the trend, flag it against thresholds, and feed it into a predictive model. This is the kind of data that early clinical AI was built on, and it remains the backbone of many deployed systems.
Diagnostic coding adds another layer. ICD codes (the International Classification of Diseases) assign a standardised numerical code to each diagnosis — allowing different hospitals in different countries to compare diagnoses on a common basis. Procedure codes (CPT in the United States, OPCS in the UK, DRG-related codes elsewhere) do the same for clinical procedures.
The strengths of structured data are genuine. It is reliable in format, consistent across records, and easy to feed into algorithms. A vital signs monitoring model that ingests timestamped heart rate, blood pressure, SpO2, and respiratory rate from thousands of ICU patients can identify trajectory patterns that would be invisible to any individual clinician reviewing individual charts.
But structured data has a critical limitation that every clinician instinctively understands: what gets coded is not always what happened clinically. ICD codes are assigned by clinical coders, not always by the treating clinician — and they reflect billing logic as much as diagnostic truth. A patient with a complex presentation may be coded for their primary diagnosis while the secondary clinical findings that most concerned the team go uncoded. A coding audit at a large teaching hospital found that a meaningful proportion of codes were either inaccurate, incomplete, or reflected the coder’s interpretation of ambiguous clinical documentation rather than the clinician’s actual diagnosis.
This means an AI trained on ICD code data has learned from a dataset that is systematically filtered through billing and administrative logic. It may learn patterns that reflect hospital reimbursement practices as much as clinical reality.
Unstructured Clinical Data
The richest clinical information in any patient record is not the structured fields. It is the text.
Clinical notes contain the clinician’s actual observations, reasoning, and judgment. A discharge summary synthesises the entire episode of care — what was found, what was tried, what worked, what was uncertain, what the patient understood, and what the plan is. A radiology report explains not just the finding but the differential reasoning. A referral letter from a general practitioner to a specialist contains clinical context that may never appear in any structured field.
Estimates consistently suggest that approximately 80% of clinical data exists in unstructured form — text, images, and audio — rather than in the structured fields of electronic health records. This means that AI systems which can only read structured data are working from a small minority of available clinical information.
Accessing unstructured text requires Natural Language Processing, or NLP. NLP is software that reads text and extracts structured meaning from it: identifying that a phrase like “bilateral lower lobe infiltrates” refers to a radiological finding, or that “no chest pain” is a negation of a symptom rather than a report of its presence.
NLP has become genuinely capable for certain clinical text types. Radiology reports follow relatively consistent formats, contain domain-specific vocabulary, and have clear outputs (findings, impressions) — making them well-suited to automated information extraction. Large-scale NLP systems have been validated on radiology report data with performance approaching that of trained readers.
Clinical notes are considerably harder. They are written in highly variable styles, contain abbreviations that differ between clinicians and institutions, rely heavily on clinical context to interpret, and include the kind of nuanced reasoning that text-extraction software struggles with. For a deeper examination of why unstructured and missing data present such a fundamental challenge to clinical AI, see Unstructured and Missing Data: What Every Clinician Must Know.
Medical Imaging Data
If there is one domain where clinical AI has made the most visible and validated progress, it is medical imaging. Chest X-rays, CT scans, MRI, whole-slide pathology images, fundus photographs, dermoscopy images — the breadth of imaging AI in active development and deployment is substantial.
The reason for this progress is worth understanding. Medical images have several properties that make them well-suited to the deep learning approaches that underpin modern AI.
First, images have consistent format. A chest X-ray is a chest X-ray — a two-dimensional greyscale representation of thoracic anatomy, acquired with relatively standardised equipment and technique. The variation between images from different hospitals, while real, is bounded in ways that clinical notes are not.
Second, images have a relatively clear “right answer” in the form of the radiologist or pathologist report. When training an image-recognition AI, the labels used to train the model are derived from expert human reporting — which provides a reasonable, if imperfect, ground truth.
Third, deep learning architectures are specifically suited to image pattern recognition. The neural network structures that power image AI — convolutional neural networks — are designed to identify spatial patterns at multiple scales, from pixel-level texture to anatomical structure-level features. They were not designed specifically for medical images, but they apply remarkably well.
The result is that imaging AI validation studies have reported performance metrics for specific tasks — detecting intracranial haemorrhage, classifying diabetic retinopathy, identifying pulmonary nodules — that are genuinely impressive and, in some studies, comparable to specialist-level performance on defined tasks in controlled conditions.
The important caveat is that performance in controlled validation conditions does not always translate to performance in real-world deployment. Variables including imaging equipment differences, patient population differences, and the open-ended nature of real clinical presentations (where the AI may encounter findings outside its training distribution) all affect real-world performance in ways that controlled studies may not capture.
Wearable and Monitoring Data
A continuous cardiac monitor generates a data point every few milliseconds. A glucose monitor records readings every five minutes. A wearable ECG patches transmits a rolling rhythm strip. The volume of data generated by monitoring devices is orders of magnitude larger than any other clinical data type.
This creates both opportunities and challenges for AI. The opportunities are significant: continuous monitoring data captures physiological dynamics that discrete clinical observations miss. A blood pressure that is entirely stable during morning rounds but shows a progressive downward trend overnight is visible in continuous monitoring data and may not be apparent in any other way. AI systems that can identify meaningful patterns in high-frequency monitoring data — distinguishing deterioration signals from normal physiological variation, identifying arrhythmias in long-term cardiac recordings — are genuinely valuable.
The challenge is noise. High-frequency monitoring data is full of artefact: motion artefact, equipment artefact, electrode displacement, patient movement. An AI model that has not been trained to distinguish artefact from signal will generate false alarms at a rate that quickly erodes clinical trust. The “alarm fatigue” problem in ICUs — where clinical teams habituate to the volume of alerts and begin ignoring them — is, in part, a consequence of monitoring systems that have not solved the signal-to-noise problem.
Genomic and Molecular Data
Genomic data represents the most complex and highest-dimensional data type in clinical medicine. A whole-genome sequence generates approximately three billion base pairs of information — far more than any other data type. Proteomics (the study of proteins expressed by a cell or organism) and microbiome data (the genetic profiles of microbial communities in and on the body) add further layers.
Clinical AI applications using genomic data are most advanced in oncology, where tumour genomic profiles increasingly inform treatment selection — identifying driver mutations that predict response to targeted therapies, or genomic signatures that suggest likely prognosis. Pharmacogenomics, the study of how genetic variation affects drug metabolism and response, is another active domain: AI models that predict how an individual patient will respond to a specific drug based on their genetic profile are in development and early clinical deployment.
These applications are real and clinically meaningful. But genomic AI is the least mature of the domains discussed here for current clinical deployment, and it remains concentrated in specialised settings — tertiary oncology centres, specialist genetics services — rather than in general clinical practice.
The NIH Precision Medicine Initiative has been one of the largest efforts to build the kind of richly annotated genomic and clinical data resource that AI development in this domain requires.
The Question Every Clinician Should Ask
For any AI tool presented to a clinician — in a product demonstration, in a validation paper, or in active deployment — there is a set of data questions that constitute due diligence.
What type of data did this AI learn from? If it is based on structured EHR data, the clinician should understand that it is working from a subset of clinical reality filtered through coding and administrative systems. If it is based on clinical notes via NLP, the clinician should understand both the potential richness and the known limitations of that approach. If it is imaging AI, the clinician should ask about the source and quality of the training images.
Was that data collected in settings like mine? A model trained at a large academic medical centre in one country has learned the patterns of that patient population, with that institution’s equipment and workflow. Clinicians working in different settings — different health systems, different patient demographics, different imaging equipment — should treat validation from a single institution with appropriate caution.
How complete was the data? What clinical information was missing from the training set? Social determinants of health — income, housing, diet, occupation — are rarely documented in EHR systems but profoundly affect clinical outcomes. An AI trained on EHR data without social history fields is predicting outcomes in a partial picture of clinical reality.
What wasn’t captured? This is the most important and least often asked question. An AI readmission predictor trained on hospital data does not have information about what happens to patients after they leave. It does not know about community resources, family support, medication adherence, or the dozen social factors that determine whether a patient ends up back in hospital. Its predictions are bounded by the data it has seen.
Data as the Raw Material of Clinical Judgment
Understanding clinical data types is not a technical skill. It is a clinical one.
The analogy to drug trials is exact. When evaluating a new medication, a clinician asks: what population was this trial conducted in? Were the patients like mine? Was the intervention as delivered in the trial comparable to how it would be used in my setting? These questions do not require statistical expertise — they require clinical judgment applied to the design of the evidence.
The same judgment applies to AI. What data was this model trained on? Was that data like the clinical reality of my patients? What was missing from it? A clinician who can ask and answer those questions is in a position to use AI tools as informed professionals rather than passive recipients of algorithmic outputs.
Data is the raw material of AI. Understanding data types is understanding the raw material. And that understanding is firmly within clinical reach — not because it requires coding, but because it requires the same disciplined questioning that medicine has always asked of its evidence.
For a comprehensive introduction to clinical AI, see Why Every Clinician Needs AI Literacy. For a deeper examination of the specific challenges posed by unstructured and missing data, see Unstructured and Missing Data: What Every Clinician Must Know.