Purpose

Though many studies have documented performance improvements with mammography AI, little data has been reported from large scale deployment. Such data would likely help radiologists better understand whether AI will generalize to their clinical practice. The purpose of this study was to evaluate performance changes of radiologists in a large-scale deployment of categorical AI used in screening mammography.

Materials and Methods

An FDA-cleared categorical AI product was deployed at 147 clinical sites across the USA. The AI provides lesion localization and categorical levels of suspicion for cancer on digital breast tomosynthesis screening mammograms. Data was collected from sites with three months of use of the AI, and on screening mammograms that have at least 30 days of follow-up (n=200,342). Data included the AI’s suspicion categories, radiologists’ BIRADS assessments, and available biopsy outcomes. (The suspicion categories are Minimal, Low, Intermediate and High, corresponding to ~25%, ~50%, ~20% and ~5% of a screening population.) For comparison, mammograms from the 14-month period prior to AI deployment were also retrospectively evaluated by AI. Performance metrics of cancer detection rate (CDR), abnormal interpretation or recall rate (RR) and positive predictive value (PPV1) from 185 radiologists were compared before and after deployment. Given the AI deployment was recent, there was limited follow-up time for cancer diagnosis so,for consistency, only cancers identified within 30 days of screening were counted. Significance was calculated using a Chi-squared test.

Results

Radiologists’ CDR increased by 33% after deployment, from 1.5 to 2.0 per 1000 mammograms (p< 0.05).Given the short 30 days follow-up period, the low CDRs were expected. The increase was primarily driven by a higher CDR in the High suspicion category going from 22.9 to 28.9 per 1000 mammograms. There was also a slight increase in RR of 11.8% to 12.5% (p< 0.001) with a slight increase in PPV1 (1.3% before vs. 1.6% after deployment).

Conclusion

These initial results show that radiologists are detecting more cancers soon after the deployment of an AI tool that provides suspicion categories as well as lesion localization. An increase in recall rate was expected as radiologists become accustomed to the product, similar to what has been observed when other new technologies such as DBT have been adopted. Our results provide early encouraging evidence that radiologists in clinical practice will benefit at scale, though further monitoring is ongoing and will be reported.

Clinical Relevance

AI for mammography using categorical suspicion levels and lesion localization can help radiologists detect more cancers.

Figure
Radiologist performance measures by AI categories in 147 clinical sites. Black lines indicate the 95% confidence intervals. CDR overall increases post deployment, primarily due to an increase in CDR for the High suspicion exams.