August 29, 2018

ACR DSI Urges AI Standardization, Interoperability and Reportability at NIH Workshop

ACR DSI Chief Medical Officer Bibb Allen, MD, FACR 

American College of Radiology Data Science Institute™ (ACR DSI) Chief Medical Officer Bibb Allen, MD, FACR, co-chaired the recent National Institute of Biomedical Imaging and Bioengineering (NIBIB) Workshop on Artificial Intelligence (AI) in Medical Imaging. Proceedings from the workshop will be published as a research roadmap for health care and scientific professionals.

ACR DSI presenters stressed the need for standardized AI use cases to not only ensure that the relevant clinical questions are answered, but to promote interoperability and reportability for ongoing quality improvement — including that: 

  • The AI market is dependent upon both the development of AI algorithms and integration with current digital solutions (PACS, reporting systems, etc.)
  • The DSI is working with various clinical experts, academic radiology departments and other radiology professional organizations to create standardized use cases for developing AI algorithms that will solve the most important clinical problems radiologists encounter
  • Compared to algorithms developed by industry developers or at single academic institutions, AI solutions that follow DSI AI use case standards will be able to be readily integrated into clinical practice
ACR DSI Chief Science Officer Keith Dreyer, DO, PhD, FACR 


ACR DSI chief science officer Keith Dreyer, DO, PhD, FACR, specifically outlined the Lung-RADS® Assist: Advanced Radiology Guidance, Reporting and Monitoring use case that was recently chosen as a pilot project by the FDA-funded National Evaluation System for Health Technology Coordinating Center (NESTcc). Dreyer outlined that the Lung-RADS® standardization provided a basis for this standardized use case that will:

  • Utilize existing ACR resources with data from multiple institutions to demonstrate the ability to validate algorithm performance prior to FDA clearance
  • Facilitate interoperability between reporting and AI developers to generate standardized data in a real-world setting
  • Capture real-world data in a national registry to monitor the performance of AI algorithms in clinical practice and enable both facility-level and developer reporting to ensure the algorithm performs as expected in clinical practice
ACR Senior Director of Quality Management Programs Judy Burleson, MHSA 

ACR Senior Director of Quality Management Programs Judy Burleson, MHSA, reinforced the need for algorithm reportability and monitoring. She recommended that use cases for AI algorithms specify relevant information in the form of common data elements that can be submitted to data registries — such as the ACR National Radiology Data Registries (NRDR®). 

Burleson urged that algorithms capture not only information about the algorithm performance, such as the percentage of time that doctors disagreed with the algorithm recommendation, but also relevant metadata — such as the exam parameters, equipment manufacturer and patient demographics. This information would be valuable for developers to help them update their algorithms and ensure unintended bias.

Attendees urged that imaging datasets for training and testing Ai algorithms be collected, anonymized and made available to help researchers, data scientists and physicians work together to develop and improve AI tools that are representative of clinical practice and free of unintended biases. Many agreed that this should eventually happen — but only after a standardized method of protecting patient privacy and information is created.

The American College of Radiology Data Science Institute™ (ACR DSI), the Radiological Society of North America and the Academy of Radiology and Biomedical Research cosponsored the event — held at the National Institute of Health in Bethesda, MD.

The workshop clarified the needs in foundational and translational research for machine learning in medical imaging.

The two-day event was streamed online. You can watch the recorded event at the links below: