What is the history of the ACR Appropriateness Criteria®?

The ACR created the Task Force on Appropriateness Criteria (AC) and appointed panel chairs in late 1993. By 1994, development of nationally accepted, scientifically based guidelines had begun. The Appropriateness Criteria were developed to support the ordering of appropriate imaging for given patient clinical conditions by referring physicians. This groundbreaking effort provided the College’s perspective on how to best use limited health care resources.

In 2000, the task force became the Committee on Appropriateness Criteria under the ACR Commission on Quality and Safety. Together, panel leaders and the chair of the Committee on Appropriateness Criteria act as a steering committee to oversee the activities of consensus panels.

The task force’s processes incorporated attributes for developing acceptable medical practice guidelines used by the Agency for Healthcare Research and Quality (AHRQ) as designed by the Institute of Medicine. From the beginning, the methodology relied on a combination of evidence, and when data from well-controlled trials were insufficient, expert consensus. Importantly, the methodology continues to employ the input of physicians from other medical specialties to provide relevant clinical perspectives.

Are the ACR Appropriateness Criteria evidence-based?

The ACR Appropriateness Criteria® are evidence-based guidelines to assist referring physicians and other providers in making the most appropriate imaging or treatment decision for a specific clinical condition. By employing these guidelines, providers enhance quality of care and contribute to the most efficacious use of medical imaging.

Expert panels led by the ACR in diagnostic imaging and interventional radiology have developed the guidelines. Each panel includes leaders in radiology and other specialties.

What does it mean if a procedure is appropriate for a specific clinical scenario?

The ACR adopted the definition of appropriateness mentioned in the RAND/UCLA Appropriateness Method User’s Manual (Fitch 2001) where “the expected health benefit (e.g., increased life expectancy, relief of pain, reduction in anxiety, improved functional capacity) exceeds the expected negative consequences (e.g., mortality, morbidity, anxiety, pain, time lost from work) by a sufficiently wide margin that the procedure is worth doing, exclusive of cost" (Brook et al., 1986; Park et al., 1986).

What is the rating scheme for appropriateness of the ACR AC?

The ACR methodology of rating appropriateness is based on the RAND / UCLA Appropriateness Method (Fitch 2001). The appropriateness rating scale is an ordinal scale from 1-9 grouped into three categories: 1, 2 or 3 are in the category “usually not appropriate”; 4, 5 or 6 are “may be appropriate”; and 7, 8 or 9 “usually appropriate.”

The ratings provide only the method to determine if the panel has disagreement about their recommendation. The rating category provides the recommendation. A group median rating of 1, 2, or 3, is the same rating category providing the recommendation that the procedure is usually not appropriate for the specific clinical condition. There is no intended preference between a procedure rated 1, or 2, or 3.

The additional category of “disagreement” indicates when the panel’s individual ratings vary too greatly from the panel’s median rating. This is described in the RAND / UCLA Appropriateness Method (Fitch 2001) as the IPRAS calculation. The calculation accounts for asymmetry in the panel’s ratings and for variations in the number of panel members

A more detailed explanation of the complete rating process can be found under Supporting Documents at www.acr.org/ac.

How is the evidence categorized?

ACR designed a systematic way to identify quality studies. The ACR uses a methodology based on GRADE to assess the quality of individual studies cited as evidence within the text of the AC document; largely due to the unreliability of the authors and journals to categorize accurately the study types of an article.

The first assessment is whether the study is primary evidence or secondary evidence. Primary evidence describes original studies that are conducted to answer a research question or hypothesis. Secondary evidence often combines or reviews original studies through some type of analysis or discussion and provides a conclusion. The study quality examines the key study elements that contribute to introducing bias into a study. More detailed description of the study quality can be found under Supporting Documents at www.acr.org/ac.

Study quality categories descriptions:

  • Category 1: The study is well designed and accounts for common biases
  • Category 2: The study is moderately well designed and accounts for most common biases
  • Category 3: There are important study design limitations
  • Category 4: The study is not useful as primary evidence. The article may not be a clinical study or the study design is invalid, or conclusions are based on expert consensus.

Is the ACR AC content transparent to the public?

The ACR AC are available on the ACR website for individual use such as for personal education or research purposes. Also, the National Guidelines Clearinghouse (NGC), one of the initiatives of AHRQ, is a public resource for evidence-based clinical practice guidelines. The ACR AC topics are submitted, reviewed and posted on the NGC site.

The ACR believes that imaging CDS systems that are used routinely to deliver health care best serve the public when they are widely available, backed by evidence and transparent development processes.

Is there any federal certification of the quality of the evidence?

No. There are various government agencies that cite the Institute Of Medicine’s (now National Academy of Medicine) Standards for Trustworthy Guidelines as the benchmark for demonstrating adherence to an evidence-based development process for clinical practice guidelines. The ACR is recognized as a qualified provider led entity by CMS which authorizes ACR to specify AUC content. To be recognized as a qPLE, ACR’s evidence based processes were reviewed. The AC process also meets the guidelines inclusion criteria established by the Agency for Healthcare Research and Quality (AHRQ)which can then be published on the National Guideline Clearing house, which is an initiative of the AHRQ.

What happens when evidence is conflicting?

ACR AC engages experts on its panels to review of all relevant literature. Taking into account the quality of evidence for each journal article cited in the AC documents, the expert panel clarifies any discrepancies among the published evidence. All such clarifications are published in the AC documents.

How often is the content updated and revised?

Each ACR AC are reviewed annually and typically are revised every three years.

What is ACR Select and how does it differ from the ACR AC®?

 The ACR AC diagnostic imaging topics are translated into a digital knowledge base. These evidence based imaging recommendations are combined with gap recommendations (not currently covered by an AC topic) from other sources. These other recommendations are reviewed by the ACR and are incorporated into the digital knowledge base to create the basis for an imaging clinical decision support (CDS) system. This imaging CDS system is ACR Select. ACR Select comprises both the platform to manage and deliver an imaging CDS system, as well as the evidence based content.

ACR Select makes the ACR AC digitally consumable by health care providers, either directly or through integration with health care IT platforms (such as electronic medical records, clinical decision support systems or computerized physician order entry systems).

How is ACR Select delivered?

The ACR Select platform includes an integration layer such that the content can be incorporated into health care IT platforms and clinical workflows. The ACR Select platform includes a web portal to access the content and augment the workflows of health care IT platforms as required.

National Decision Support Company provides the licensing and technical support of the ACR Select platform into the market so that health care providers can improve the delivery of imaging services and access ACR Select in their clinical workflows.

Why does ACR Select use all available evidence rather than only evidence from controlled trials?

Medical studies are designed to answer specific questions. Typically, therapeutic studies require a degree of confidence that the treatment provided causes the expected outcome. Randomized controlled trials are the best design for answering these types of questions. For diagnostic accuracy or test performance, observational studies are preferred to assess the sensitivity and specificity of an imaging procedure compared with the definitive diagnostic test. While the change of introducing bias in an observational study is greater than a randomized controlled trial, a well-designed and well implemented observational study does produce high quality evidence. Because of the high costs and paucity of funding for imaging clinical trial, the evidence from RCT may be scarce for diagnostic imaging. If a high-quality study exists, it is unlikely to continue to be studied. Research may never be performed to study the value of examinations that are unlikely to be appropriate for given conditions.

The ACR AC scours the available medical literature for relevant evidence. The studies are evaluated using a quality assessment process based on GRADE and developed by ACR. Once the recommendations are determined, the studies associated with the recommendation are evaluated to determine the strength of the evidence using an ACR process based on the GRADE methodology. This information is available in the ACR AC appendices in the evidence tables and can be found in ACR Select.

Should a CDS system be able to provide expert consensus guidance as well as evidence-based guidance?

 All studies must be vetted by experts to interpret and translate the evidence so it can be used effectively. The ACR AC uses the strength of evidence assessment to define the amount of high quality evidence available for a specific recommendation. The strength of recommendation assessment informs the user of how strongly the panel feels about the recommendation. Therefore, a recommendation with low quality evidence still can be highly recommended but it would be based on expert opinion. A recommendation with high quality evidence may only have a weak recommendation if the experts think that another recommendation is a better clinical choice (even though the evidence for the other recommendation may be weaker). 

An imaging CDS system should provide the ordering clinician all information available at the time of order, as well as the source and quality of all evidence leading to the recommendation for imaging. This would include recommendations with high quality evidence and those based more on expert opinion. Experience has indicated that presenting the end user with the most complete guidance possible improves the physician experience with imaging CDS.

Why is ACR Select important to me as a radiologist?

When used in the CPOE workflow, ACR Select ensures the most appropriate imaging procedure based upon a structured set of clinical indications.

The recommendations are based on the most current medical literature and reflect more than 20 years of the ACR’s expertise and collaboration with other medical societies.

The health care delivery model is moving away from volume-based models towards value-based payment models. The reimbursement for imaging services continues to decline without regard to the value of imaging on patient care.

By using ACR Select to guide the ordering physician towards more appropriate imaging, radiologists can be at the forefront of the dialogue regarding how best to utilize imaging in the care cycle and collaborate on new value-driven models for the delivery of imaging services.

ACR Select allows radiology to demonstrate the value of properly delivered imaging services on improving the quality of care. CDS can help drive radiology as a consultative service when clinicians encounter moderate to low level of appropriateness scores.

Bibb Allen, Jr., MD, FACR, has published a presentation outlining the value proposition for radiologists to adopt clinical decision support.