Leveraging a memory-driven transformer for efficient radiology report generation from chest x-rays to establish a quantitative metric

Nathan Hadjiyski; Akhil Kasturi; Ali Vosoughi; Axel Wismüller

doi:10.1117/12.3027803

4 October 2024 Leveraging a memory-driven transformer for efficient radiology report generation from chest x-rays to establish a quantitative metric

Nathan Hadjiyski, Akhil Kasturi, Ali Vosoughi, Axel Wismüller

Author Affiliations +

Proceedings Volume 13118, Emerging Topics in Artificial Intelligence (ETAI) 2024; 1311804 (2024) https://doi.org/10.1117/12.3027803
Event: Nanoscience + Engineering, 2024, San Diego, California, United States

Abstract

This study utilizes a novel method for learning representations from chest X-rays using a memory-driven transformerbased approach. The model is trained on a low-quality version of the MIMIC-CXR dataset, utilizing 17,783 chest X-rays that contain at most 3 views. The model uses a relational memory to record crucial information during the generation process and a memory-driven conditional layer normalization technique to integrate this memory into the transformer's decoder. The dataset is divided into distinct sets for training, validation, and testing. We aim to establish an intuitively comprehensible quantitative metric, through vectorization of the radiology report. This metric leverages the learned representations from our model to classify 14 unique lung pathologies. The F1-score measures classification accuracy, indicating the model's viability in diagnosing lung diseases. We also have introduced the use of Large Language Models (LLMs) for evaluation of the generated reports accuracy. The model's potential applications extend to more robust performance in radiology report generation.

Conference Presentation

(2024) Published by SPIE. Downloading of the abstract is permitted for personal use only.

Citation Download Citation

Nathan Hadjiyski, Akhil Kasturi, Ali Vosoughi, and Axel Wismüller "Leveraging a memory-driven transformer for efficient radiology report generation from chest x-rays to establish a quantitative metric", Proc. SPIE 13118, Emerging Topics in Artificial Intelligence (ETAI) 2024, 1311804 (4 October 2024); https://doi.org/10.1117/12.3027803

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available