Introduction#

Medical imaging, including radiography, computed tomography (CT), magnetic resonance imaging, optical coherence tomography (OCT), and color fundus photography, is at the heart of how diseases are detected and staged, but the process of interpreting the images is costly, time-consuming, and prone to considerable variation between different readers.
The present report looks at one specific aspect of this field: the use of deep learning for disease detection, by which is meant the making of an image- or patient-level diagnostic decision (such as present/absent, referable/non-referable, or malignancy risk) directly from a diagnostic image. Tasks that are closely related and which also make use of deep networks in connection with medical images, such as anatomical segmentation, image reconstruction and denoising, registration, and automated report generation, are not covered by this report, any more than non-imaging methods such as electronic health records or genomics are.
In this area, the field has followed a clear course of development. It started out with two-dimensional convolutional neural networks (CNNs) which had been pre-trained on natural images and then tested using panels of expert assessors [3]. It then progressed to three-dimensional architectures that take in the whole three-dimensional acquisition from start to finish [1]. It was later obliged to deal with evidence that the high accuracy figures often merely reflect artifacts of the dataset rather than actual pathology [2]. Recently, it has shifted its focus to self-supervised, transformer-based foundation models aimed at improving label efficiency and achieving cross-task generalization [4].
The three imaging techniques are represented by the four papers described below, and together they enable us to consider not only what deep learning is capable of detecting but also what kind of evidence should be required before a detection claim is accepted.

Key Points in the Papers#

Gulshan et al. (2016): Diabetic Retinopathy in Fundus Photographs [3]#

Contributions#

Gulshan et al. [3] fine-tuned an Inception-v3 convolutional neural network, which had been pretrained on ImageNet, using 128,175 fundus photographs that had been assessed by 54 US-licensed ophthalmologists and senior residents, each of the images being given several separate ratings.
With regard to referable diabetic retinopathy, the model achieved an AUC of 0.991 on EyePACS-1 and 0.990 on Messidor-2. The authors also give two operating points, one adjusted for specificity and the other for sensitivity, demonstrating that one trained model can be set up for different screening applications.

What I Found Interesting#

What interested me was that the reference standard is behind the innovation, not the network; the architecture is off-the-shelf, and the transfer recipe is standard.
Since there are multiple adjudicated grades per image, label noise is reduced below the level observed with a single reader, and therefore the model is assessed against a performance ceiling that is better than that of any individual clinician.
This means that the bottleneck in medical AI shifts from model capacity to the quality of the annotations.

Limitations#

There are some limitations.

• The study is retrospective and only includes gradable images.
• Although ungradable photographs are excluded from the main analysis, real-world screening systems must still deal with them.
• Only agreement between the graders is assessed, not actual clinical outcomes.
• The model provides a score without any explanation.
• Diabetic retinopathy is particularly suitable because lesions are visible in a single 2D image and an established grading scale already exists, making it difficult to generalize the findings to other diseases.

Ardila et al. (2019): Lung Cancer Risk from Low-Dose Chest CT [1]#

Contributions#

Ardila et al. [1] propose a complete three-dimensional model that accepts the entire low-dose CT volume together with a previous scan when available, producing a malignancy risk score rather than relying on a separate nodule detection phase.
The model achieved an AUC of 94.4% on 6,716 National Lung Screening Trial cases and showed similar performance on an independent clinical dataset containing 1,139 cases.
In a reader study performed without prior examinations, all six board-certified radiologists performed at or below the level of the model.

What I Found Interesting#

The most interesting design choice was conditioning predictions on earlier CT scans.
Radiologists routinely rely on changes in nodule size over time, and the model's advantage over human readers was greatest when those earlier scans were unavailable to the radiologists.
This suggests that claims of superhuman performance often result from unequal access to contextual information rather than fundamentally better perception.

Limitations#

Some important limitations include:

• The National Lung Screening Trial population differs from the general population because of eligibility criteria such as age and smoking history.
• The independent validation dataset originated from a single institution.
• Ground truth depended on biopsy-confirmed diagnosis within a defined follow-up period, mixing detection performance with lead time.
• No interpretability analysis similar to that of DeGrave et al. [2] was performed.

DeGrave et al. (2021): Shortcut Learning in COVID-19 Chest Radiograph Classifiers [2]#

Contributions#

Using explainable AI, DeGrave et al. [2] demonstrate that many COVID-19 chest radiograph classifiers make predictions based on confounding factors rather than genuine pulmonary pathology.
Examples include:

• Laterality markers
• Image annotations
• Patient positioning
• Framing of the shoulders and diaphragm

The authors attribute these failures to dataset construction, where positive and negative examples originated from different repositories.
Although the models performed well internally, their performance deteriorated substantially when evaluated on data from different hospitals.

What I Found Interesting#

Beyond saliency maps, the authors used generative image modification to determine what changes would actually flip a model's prediction.
This transforms explainability from a visualization tool into an experimental method for discovering causal model behavior.
It also argues that explainable AI should become a prerequisite for deployment rather than merely an optional feature.

Limitations#

The paper also has limitations.

• The evaluated models were developed rapidly during the COVID-19 pandemic.
• They likely represent a worst-case scenario rather than typical medical AI systems.
• Saliency methods themselves have known reliability issues.
• The paper analyzes failures thoroughly but validates mitigation strategies only briefly.

Zhou et al. (2023): RETFound – A Retinal Foundation Model [4]#

Contributions#

Zhou et al. [4] introduce RETFound, a Vision Transformer pretrained using masked autoencoder self-supervised learning on 1.6 million unlabeled retinal images from fundus photography and OCT.
After fine-tuning, the model outperformed supervised and alternative self-supervised methods across numerous ophthalmic tasks.
It also showed promising performance in predicting systemic diseases such as:

• Heart failure
• Myocardial infarction

The pretrained weights were released publicly.

What I Found Interesting#

The most compelling result was not the headline accuracy but the label-efficiency curves.
The model's advantage became larger as fewer labeled examples were available, directly addressing one of the major challenges in medical AI.
The ability to infer systemic disease risk from retinal images without ophthalmic labels was also particularly impressive.

Limitations#

Several limitations remain.

• The pretraining dataset primarily represents certain populations and imaging devices.
• Generalizability is therefore uncertain.
• Evaluation remains retrospective and based on curated datasets.
• No shortcut audit similar to DeGrave et al. [2] was conducted.
• Self-supervised learning does not inherently prevent learning site-specific artifacts.
• The computational requirements for pretraining remain prohibitively expensive for most organizations.

Discussion#

Gulshan et al. [3] and Ardila et al. [1] share a common validation philosophy.
They define a clinical task, establish a strong reference standard, compare model performance with expert clinicians, and evaluate performance using AUC. Their primary difference lies in architectural design: two-dimensional transfer learning versus end-to-end three-dimensional modeling with temporal information.
DeGrave et al. [2] take an entirely different perspective. Rather than asking whether models achieve high accuracy, they investigate why those accuracy values occur and demonstrate that dataset shortcuts can produce misleadingly impressive results.
Zhou et al. [4] shift attention again, asking how little labeled data a new task requires instead of focusing solely on single-task accuracy.
The strongest tension exists between the two most recent papers.
RETFound argues for broadly generalizable foundation models, yet evaluates them using the same retrospective metrics that DeGrave et al. criticize as insufficient.
One could argue that self-supervised pretraining may even amplify shortcut learning because reconstructing masked image regions encourages learning predictable scanner signatures and annotation artifacts.
All four studies remain retrospective, and none provide prospective randomized evaluations measuring actual patient outcomes.
Furthermore, the barrier to entry has increased rather than decreased.

• Gulshan et al. required 54 clinical graders.
• Zhou et al. required 1.6 million retinal images and enormous computational resources.

Future work should focus on three directions.

1. Robustness reporting should become standard practice, including multi-site validation and explicit shortcut audits.
2. Foundation models should expand beyond single-organ datasets toward cross-modality CT, MRI, and vision-language pretraining using radiology reports.
3. Evaluation should move from retrospective agreement with expert labels toward prospective measurement of patient outcomes while incorporating calibrated uncertainty, allowing models to defer difficult cases to clinicians.

References#

1. Ardila, D., Kiraly, A. P., Bharadwaj, S., Choi, B., Reicher, J. J., Peng, L., Tse, D., Etemadi, M., Ye, W., Corrado, G., Naidich, D. P., & Shetty, S. (2019). End-to-end lung cancer screening with three-dimensional deep learning on low-dose chest computed tomography. Nature Medicine, 25(6), 954–961. https://doi.org/10.1038/s41591-019-0447-x
2. DeGrave, A. J., Janizek, J. D., & Lee, S.-I. (2021). AI for radiographic COVID-19 detection selects shortcuts over signal. Nature Machine Intelligence, 3(7), 610–619. https://doi.org/10.1038/s42256-021-00338-7
3. Gulshan, V., Peng, L., Coram, M., Stumpe, M. C., Wu, D., Narayanaswamy, A., Venugopalan, S., Widner, K., Madams, T., Cuadros, J., Kim, R., Raman, R., Nelson, P. C., Mega, J. L., & Webster, D. R. (2016). Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. JAMA, 316(22), 2402–2410. https://doi.org/10.1001/jama.2016.17216
4. Zhou, Y., Chia, M. A., Wagner, S. K., Ayhan, M. S., Williamson, D. J., Struyven, R. R., Liu, T., Xu, M., Lozano, M. G., Woodward-Court, P., Kihara, Y., Altmann, A., Lee, A. Y., Topol, E. J., Denniston, A. C., Alexander, D. C., & Keane, P. (2023). A foundation model for generalizable disease detection from retinal images. Nature, 622(7981), 156–163. https://doi.org/10.1038/s41586-023-06555-x