Introducing Medical Students to Appropriateness Criteria

When COVID-19 sent the nation into lockdown, Tyler Pease was completing his third year at Rutgers Robert Wood Johnson Medical School (RWJMS). As universities across the country pivoted to online learning, it soon became clear that regular clinical rotations — during which medical students often receive the greatest exposure to radiology — would be unavailable to most medical students.

“My peers and I were wondering what was going to happen with our education,” Pease says. “We weren’t sure how we would complete our clinical requirements.”

In March of 2020, Judith K. Amorosa, MD, FACR, vice chair for faculty development and academic affairs at RWJMS and radiologist partner with University Radiology, received an urgent email from the dean’s office requesting ideas to address the necessary changes in educational instruction. After thinking about how radiology could rise to the challenge, she conceived a proposal to virtually introduce medical students to radiology and appropriate imaging ordering practices.

Drawing from her experience developing a hybrid class for students who were interested in radiology but unable to attend lectures, Amorosa created a self-directed radiology course to educate medical students about the ACR Appropriateness Criteria® (AC). ACR developed the AC to help referring physicians make appropriate image ordering decisions. This elective has allowed students to focus on the AC most relevant to their own subspecialty interests and receive credit for clinical hours during the pandemic.

Judith K. Amorosa, MD, FACR, vice chair for faculty development and academic affairs at Rutgers Robert Wood Johnson Medical School (RWJMS), developed a remote radiology course to introduce medical students to the ACR Appropriateness Criteria.

Twenty students have completed the requirements and gained considerable knowledge about the AC since the course began in April of 2020. “Without this course, I wouldn’t be able to say that I felt fluent in navigating and using the ACR AC,” says Pease, who was the first student to enroll in the class. “The course gave me a leg up when it came to gaining clinical experience in radiology.”

Addressing a Need

The idea for the elective grew from a previous program that predated COVID-19. “I received many calls from medical students asking me about radiology,” says Amorosa. “Many medical students couldn’t take the courses we offered because their schedules were too demanding. Instead, they would send scribes to take notes on lectures so that they could learn the material on their own. This gave me the idea to create a hybrid course that encouraged independent and self-directed learning.”

The original program, named Radiology on the Interview Trail, combined clinical experience with online instruction to educate busy medical students about imaging methodologies and basic radiology procedures. “It worked well because students were able to take advantage of radiology resources while also navigating specialty interviews and other demands on their time,” Amorosa says.

When the dean’s message went out about ways to pivot to virtual learning, Amorosa saw this existing virtual program as an opportunity to expand on an already successful course model. She immediately knew she wanted to incorporate the AC to help expand its use. She also added the AC project to Radiology on the Interview Trail, exposing more than 80 additional students to the AC guidelines.

Meigra M. Chin, MD, associate professor and assistant program director of emergency medicine at RWJMS, chaired the curriculum committee that approved the Remote Radiology course.

“I realized shortly after starting my first job as a radiologist that referring doctors often do not know which imaging modality is best for a clinical condition,” Amorosa says. “I still feel from my daily experience with many practitioners that they are not using the AC as much as they should, so when the call came to do something to teach medical students — our future practitioners — online, the AC came to mind.”

Leveraging the Guidelines

As Amorosa recognized, even though the ACR introduced the AC in 1993 to help minimize inappropriate image ordering, many ordering providers still do not rely on or even know about the resource. In 2009, she was part of a team that led a study to determine how often physicians use the AC. Of the 126 physicians surveyed, only three reported using the AC when ordering image studies.¹

Amorosa, who has served on panels to develop the chest thoracic AC, has seen firsthand how much time, effort, and care go into developing the guidelines. She knows how valuable this resource could be for medical students and the radiologists who work with them.

The AC’s value only increased in June of 2016 when the Centers for Medicare and Medicaid Services (CMS) named the ACR a qualified provider-led entity. The designation means that providers can consult the AC to meet the requirements of the Protecting Access to Medicare Act (PAMA). Under PAMA, providers must consult qualified appropriate use criteria through clinical decision support when ordering imaging for Medicare patients. This requirement is slated to take effect in 2023.

“This course helps prepare medical students to comply with PAMA,” says Amorosa. “These medical students will be familiar with the background and resources from the ACR as they become practicing physicians.”

Developing the Course

In March of 2020, Amorosa presented the proposed elective to the curriculum committee, which is chaired by Meigra M. Chin, MD, associate professor and assistant program director of emergency medicine at RWJMS. After several conversations about the course structure and how its objectives would provide effective substitutes for clinical hours, the committee approved it for the 2020 spring and summer terms.

“It was clear that students were going to need opportunities to progress in their education in this time without patient contact,” Chin says. “The committee was completely supportive of Dr. Amorosa’s elective course because we all recognize that radiology is an area in which students do not always receive guided and specific education.”

In less than a week, 20 students enrolled in the new Remote Radiology elective for April and May 2020. “It was a desperate time,” says Amorosa, “but there was great enthusiasm for the course, and I was encouraged to move forward.”

Implementing the Course

Amorosa partnered with Jasmine Allen, radiology course coordinator at RWJMS who coordinates the residency and medical student rotation program, to organize the course and maintain its content. Allen also helped roll out the course through the school's learning management system and track student success throughout the course.

During the course, students worked directly with Amorosa to learn about the AC. She asked students to develop presentations around the AC topics that aligned with their specialty areas. For Pease, this meant studying the AC that corresponded with orthopedics.

After a virtual kickoff meeting to introduce the course, Amorosa met with students individually through email, phone, and video conferencing to help them find appropriate resources and develop their presentations. She asked students to produce three presentations per week, but this number varied based on individual student needs and topics. For instance, Pease ultimately completed four presentations on the acutely limping child, scoliosis child, back pain child, and developmental dysplasia of the hip.

Students were required to include in-depth clinical and lab data along with appropriate imaging in their presentations. The research forced the students to become immersed in the AC for true learning, Amorosa notes. “Recognizing the students’ interests and future plans is key,” she adds. “Aligning AC guideline topics with their future careers brings them closer to this resource at their fingertips.”

 

Tyler Pease, a fourth-year student at RWJMS with a focus in orthopedics, was the first student to enroll in the Remote Radiology elective

To meet the course requirements, Pease knew that he would need a better understanding of MRI visualization techniques for his first presentation, which focused on the acutely limping child. “I sought out a number of lectures on YouTube about the physics of MRI,” he explains. “I wanted to understand MRI on a fundamental level. This opportunity to engage more with these techniques enhanced my understanding of what MRI is and how it works. I hadn’t been taught that in any of my pre-clinical experiences.”

Along the way, Pease received notes from Amorosa and Catherine S. King, MD, whom Amorosa consulted as a subject matter expert. King, who specializes in musculoskeletal radiology at Northwest Radiology, was then serving as chief resident at Rutgers Robert Wood Johnson University Hospital. Both Amorosa and King provided feedback about Pease’s ideas and materials and helped guide his presentation content by suggesting imaging modalities connected to specific pathologies. “Dr. Amorosa and Dr. King gave invaluable feedback that led me to relevant AC guidelines,” Pease says. “This helped me to identify appropriate imaging modalities.”

Presenting the Research

For each presentation, students delivered scripts and decks of 30-45 slides. No formal reports were required, but some students choose to present their materials to Amorosa and a consulting specialist, as well as their peers. Currently, the radiology department is in the process of uploading the learning materials they created to the department’s website.

Pease presented his final research as case studies. For example, in his presentation on the acutely limping child, he introduced a hypothetical four-year-old whose parents were concerned about him limping and favoring his right foot. Pease also developed additional context that the child had spent the morning with his maternal aunt, who is the mother of two similarly aged children. This helped Pease explore multiple possible reasons for the limp.

Pease discussed potential diagnoses for limping children based on different possible sub-scenarios, such as whether the child could identify the pain source. Using the AC for the acutely limping child, he discussed the advantages and disadvantages to different imaging techniques, such as MRI, ultrasound, and three-phase bone scan.

He also reviewed other important considerations for image ordering, such as radiation exposure, patient immobilization, and sedation, as well as parent and patient preparedness, and age-appropriate imaging selection. His presentation included appropriate images and citations for each option.

Amorosa was impressed with the quality of the materials that Pease and the other students developed. “Most of these students are better at making these kinds of presentations than I am,” she laughs. “They are very imaginative, and I am always impressed with how well they take their interests and the AC and make something incredible.”

Soliciting Student Feedback

Throughout the process, Amorosa encouraged her students to keep process and research journals. She asked the students to provide feedback to help improve the curriculum in the future.

Of the 20 students who participated in the Remote Radiology course and the students who evaluated their project experience with the AC in Radiology on the Interview Trail, 90% said they learned and found the experience “very useful/useful” for their future career.

Pease says he was extremely satisfied: “I thought the ACR AC was really excellent as an educational resource,” he says. “I didn’t know it existed before I took this course. I would definitely recommend that other medical students reference the AC when familiarizing themselves with radiology in their specialties.” Pease says that he plans to use his Remote Radiology presentations as educational materials if asked to explain appropriate image ordering protocol to colleagues or trainees in the future.

In the survey, another Remote Radiology student noted, “By creating the topic slides, I had to dive into the material. The process of creating the cases myself allowed me to deeply engage with the material and review a variety of different pathophysiologies. I found the process educational and learned valuable information that will greatly benefit me in clinical practice.”

Looking Ahead

Based on positive feedback like this, the curriculum has been regularly offered as an elective since April of 2020. In addition, Amorosa has introduced a similar project as part of the Radiology on the Interview Trail course, and she is already considering how it might be reconfigured when students are able to see patients again. She hopes to continue offering the elective alongside regular clinical rotation opportunities and to introduce additional AC resources, such as the ACR Appropriateness Criteria app, which was developed by a medical student and is not associated with the ACR.

“I would like to see some portion of this particular elective become a requirement for the fourth-year experience,” Chin says. “I think the AC is something medical students need to be educated about. Helping them determine the best and safest study for patients is something they should be prepared to do, and this course can help position them to make rational decisions when necessary.”

Amorosa says programs like this add tremendous value to the medical student experience without requiring much additional work from radiologists. “The amount of work to develop and monitor a course like this is not overwhelming for faculty,” Amorosa explains. “It is more of an advisory role. Students just need to know what resources are out there, and they can mostly take it from there.”

Amorosa encourages others to find ways to engage medical students with appropriate radiology ordering practices throughout the pandemic and beyond. “Ultimately, courses like these can boost student confidence in image ordering,” she says. “They can also help medical students recognize how appropriate orders help radiologists and ordering physicians work together to enhance patient care. As physicians, we have an obligation to pass this knowledge along to the next generation of care providers.”