1.

Introduction

Computed tomography (CT) applications represent one of the most well-established diagnostic tools in current medical imaging; CT is capable of providing very detailed anatomical images of many biological tissues at one time due to its large dynamic range. With the widespread availability of CT equipment and the increasing number of patient examinations,¹ the issues of the quantification of risks related to X-ray exposure, and consequently the need for further optimization of CT protocols to fulfill the ALARA (“As Low As Reasonably Achievable”) principle, have arisen.^2,3 The main international organizations dealing with ionizing radiation protection and safety standards, the International Commission For Radiological Protection (ICRP) and the International Atomic Energy Agency (IAEA), have provided patient dose management recommendations and have identified lacunae in justification and optimization, thus providing guidance and improving practice.^4–7 In the Directive 2013/59/EURATOM,⁸ the European Union Council stated the need to develop and put into action optimization programs to achieve the best compromise between radiation dose and image quality, with the aim to reduce patients dose to the minimum level compatible with diagnostic accuracy.

The choice of the optimum dose level requires the evaluation of CT image quality, which can be measured by receiving operator characteristic (ROC) analysis in reader studies in which trained medical staff perform a specific clinical task. Such an approach is especially suitable in the case of iterative reconstruction techniques because the standard physical quantities are no longer suitable for a thorough image quality assessment.⁹ However, the extremely high number of different CT protocols in use even within small radiological facilities makes de facto the evaluation of all necessary ROC curves almost impracticable as it would require too much observation time to be provided by medical staff. In the recent past, this fundamental limitation has been addressed by replacing human observers with algorithmic approaches (i.e., model observers); in particular, the channelized Hotelling observer (CHO) model^10–12 demonstrated great potential, but it is still limited by poor generalization capability to different CT settings.^13,14 The appreciable results obtained through such algorithmic methods have encouraged researchers to proceed by employing artificial intelligence (AI) algorithms, which are seemingly more powerful than CHO.^15–21 Recently, the increasing availability of computational resources has driven the scientific research toward the use of the latter approach, which has shown remarkable effectiveness in mimicking the human observers’ performances in different diagnostic imaging evaluation tasks.^17,21–24 To take into account the inefficiency and variability of human responses, several strategies have been proposed. Previous adopted approaches consist of the introduction of an internal noise component in the output statistics of convolutional neural networks (CNNs)^13,16,23 and the use of human-labeled data for training.^17,21,25 When actual patient CT data are used, the dose level dependency is commonly studied by introducing appropriate noise into the images.^{16,21–23,26–28}

Within this context, our goal is to build a solid model observer (MO) framework based on CNNs that is capable of reproducing the performances of human observers in the identification of low contrast-to-noise ratio (CNR) objects in reference phantom CT images. Compared with the current state-of-the-art methods, our work is characterized by the concurrence of large dataset variability in terms of size and CNR of the imaged objects, CT acquisitions at eight different dose indices, two reconstruction techniques, and labels by a large group of 30 human observers, including 19 radiologists from four different radiological departments.

The use of a specifically designed phantom allowed for the collection of a large dataset of 30,000 images at various dose indices and under controlled acquisition conditions.

Two intrinsically different CNN architectures were optimized for the double task of localization and classification of low CNR objects within the phantom CT images to get insight into the relation between CNN behavior and architectures.

We performed an extended statistical analysis of the results to address the overall observers performances in terms of localization-area under curve (L-AUC), which is expected to be more accurate than the conventional AUC metric because it takes into account both localization and classification capabilities.²⁹

Several statistical indices and the accuracy metric were computed to obtain a better understanding of the CNNs response and the limitations of this AI approach.

The results are very promising: both approaches are capable of miming human detectability performance in phantom CT images. We believe that these CNN-based MOs, combined with specifically designed phantoms, may effectively support the optimization of CT protocols, avoiding the time-consuming limitations of medical staff evaluations.

2.

Materials and Methods

2.1.

Image Dataset

The annotated dataset used to train, validate, and test the proposed CNNs is a subset of the large dataset extensively described in our previous work.³⁰ The dataset consists of CT images of a specifically manufactured PolyMethyl MethAcrylate (PMMA) phantom (Fig. 1), containing 10 cylindrical inserts of different diameters (3, 4, 5, 6, and 7 mm); each couple of inserts with the same diameter provides two different contrast values (45 and 55 HU) with respect to the PMMA background obtained by filling the inserts with aqueous solutions of iodinated contrast media at two distinct concentrations. The phantom consists of three adjacent blocks, each with an ellipsoidal shape with a major axis of 31 cm, a minor axis of 21 cm, and a thickness of 7 cm: two blocks have five inserts each and the third, with no inserts, is finalized to obtain homogeneous background images.

Fig. 1

Lateral view (a) and top view (b) of one of the two blocks containing five inserts filled with iodinated contrast media.

Acquisition was performed with a 128 slice CT scanner (Somatom Definition Flash, Siemens Healthcare) at eight different volumetric CT Dose Index settings (${\mathrm{CTDI}}_{\mathrm{vol}}$ [mGy] = 4.4, 5.1, 6.0, 6.9, 7.8, 8.6, 9.6, and 10.2), with the following protocol for abdomen selected: 120 kVp, AEC on, helical mode, pitch = 1, beam collimator = 38.4 mm, and slice thickness = 2 mm).

Both filtered back projection (FBP) and Iterative Reconstruction (IR, SAFIRE force 3) image reconstruction techniques were applied to the acquired data, with convolution kernels B41s and IF41s, respectively. The CT image reconstruction FoV (RFoV) was chosen to be $5{\mathrm{cm}}^2$ ($512\times 512\text{pixels}$ per image) to produce reconstructed images containing one single insert each. Images without inserts were also similarly reconstructed and added to the dataset. To further increase the dataset variability, data augmentation techniques consisting of 90 degrees rotations and horizontal and vertical flips were applied on all images. Out of this large dataset, 30,000 images were selected with a tradeoff between having an adequate amount of training data for CNNs and an acceptable amount of time being spent to collect confidence scores and insert position coordinates (when detected) for each single image by visual inspection. On the basis of the knowledge acquired in a previous work,³⁰ the selected subensemble was chosen as described in Table 1. It is worth pointing out that the dataset is not balanced in terms of diameters and contrasts of the imaged objects: the abundance of imaged object types was selected according to their detectability (as quantified by human LAUC analysis in Sec. 3): the larger the detectability is, the smaller the subensemble is; moreover, images containing inserts of 6 and 7 mm in diameter at the higher CNR were excluded because the visibility of such objects was too elevated across the entire ${\mathrm{CTDI}}_{\mathrm{vol}}$ range. A reference subset was chosen to evaluate the observers performances; it consists of images containing 4-mm diameter objects, contrast $C=45\mathrm{HU}$, and the iterative reconstruction (IR) algorithm: the CNR computed on such a subset monotonically increased with ${\mathrm{CTDI}}_{\mathrm{vol}}$ from 1.9 to 3.1.

Table 1

Selected subensembles from the original image dataset characterized by insert (object) size and contrast.

No. images	object diameter	contrast
	d (mm)	C (HU)
10,000	Homogeneous images	—
3000	3	45
3000	3	55
3000	4	45
3000	4	55
2800	5	45
2800	5	55
1200	6	45
1200	7	45

For the human observers image visualization step and the subsequent steps of algorithm optimization, the dataset, originally reconstructed with a $512\times 512$ RFoV, was reduced to $256\times 256\text{pixels}$ per image to optimize computational resources, after testing to ensure that resizing the images did not affect the CNN performances.

Figure 2 shows an example set of reconstructed images of two inserts (4 and 7 mm diameter) at the lower contrast (45 HU) for different CTDI values. It is noticeable that the visibility of the insert decreases with ${\mathrm{CTDI}}_{\mathrm{vol}}$ due to the decrease of CNR.

Fig. 2

Example of reconstructed images (iterative reconstruction technique) with (a) 4 mm and (b) 7 mm inserts at the lower contrast (45 HU); it is noticeable that object visibility depends both on size and ${\mathrm{CTDI}}_{\mathrm{vol}}$.

2.2.

Confidence Scores Collection

To collect the labels to train the MOs, the detection task was represented as a multiclass ranking task: an ordinal score was assigned by the human observer, corresponding to the confidence attributed to the presence (or absence) of the object, in a range from 0 to 3 (0 = object surely not present; 1 = object unlikely to be present; 2 = object likely to be present; 3 = object surely present). At the same time, the operator was asked to identify the location of the object (if assigned score is not 0). A graphical Python-based interface was developed to automatically save the score and the coordinates assigned to each identified object by the human observers. A representation of the screen window generated by the software and presented to the operator for image evaluation is reported in Fig. 3.

Fig. 3

Screenshot of the software interface developed to collect the human observer response to CT images visual inspection.

A total of 30 human observers contributed with the visual examination of 1000 images each; following a strategy already proposed in previous works,^12,16 both radiologists (20) and medical physicists (10) were included as evaluators to get a larger variability of evaluation performances and make the CNN-MOs more reliable and robust. In consideration of the easy task and the simple content of the dataset, ingredients that risk inducing overfitting in CNN training, as well as the very time-consuming reader study, we decided to promote dataset size over multiple ratings of single images: there was no intersection between the subsets of 1000 images evaluated by each observers. However, for the same contrast, size, reconstruction technique, and ${\mathrm{CTDI}}_{\mathrm{vol}}$, a multitude of images with similar properties (noise pattern and signal detectability) were generated from different slices of the same CT scan, among which the signal location was varied by data augmentation (see also Sec. 2.1).

2.3.

Convolutional Neural Networks

Two specific CNNs were developed and optimized to perform the MO task: a U-Net-based architecture and a MobileNetV2-based architecture.

Both CNNs were trained from scratch on the training dataset, which consisted of noisy images that previously underwent visual inspection, labeled with the corresponding confidence scores and coordinates assigned by the human observers. Despite a few recent applications,²⁵ this labeling choice represents an alternative training strategy with respect to most of the MO algorithms reported in the literature,^16,16,24,31 often based on an a posteriori correction of the output statistics of CNNs, trained with impartial labels representing the actual presence and location of the object within the images.

To ensure a robust statistical analysis of results, a fivefold cross validation procedure was applied: five training experiments were carried out using randomly assigned train and test subsets (80% and 20%, respectively, for each experiment).

In the following, the developed CNNs are described in detail.

2.3.1.

UNet-based architecture

The first architecture was designed for the MO tasks by customizing a UNet-based CNN, previously developed by the authors³⁰ for denoising and segmentation of phantom CT images. The double-task strategy, successfully employed in the cited work, was implemented in this context to achieve object localization and confidence score prediction at the same time.

The UNet is a CNN based on an autoencoder architecture already well documented in the literature^32–42 that, in particular, has been employed for segmentation and localization tasks in the postprocessing of medical images and has already demonstrated elevated performances as an MO.⁴²

The original architecture, consisting of a combination of max pooling, convolution, and fully connected layers, was reduced to a total of nine layers and four skip connections. A scheme of the UNet used is reported in Fig. 4 (architecture details and layers sequence are reported in Fig. S1 in Supplementary Material). At the end of the encoder stream, a dense layer is connected to produce a scalar output representing the confidence score prediction (implemented as a multiclass classification task). A mean square error loss ${\text{Loss}}_{\mathrm{MO}}$ is implemented to let the CNN learn the scores given by the human observers (used as score labels for training).

Fig. 4

Schematic illustration of the developed UNet-based CNN architecture.

The decoder stream is fully devoted to the localization task. The idea behind the implementation of the localization task originated from the CNN architecture proposed by Newell et al.,⁴³ in which concatenated autoencoders were used to estimate the pose of a human body through the generation of a series of heatmaps, one for each identified body joint. In our case, a single autoencoder is implemented to generate the heatmap corresponding to the object identified within the image. The heatmap is a $256\times 256$ matrix with a maximum that is assumed to be the prediction of the object center. Following Newell et al.,⁴³ two additional losses are implemented and devoted to the localization tasks. The first one is a Kullback–Leibler divergence loss (${\text{Loss}}_{\mathrm{KLD}}$)^44–48 between the heatmap produced by the network and the ground truth, represented by a 2D Gaussian (normalized to unity and with FWHM equal to the object diameter) centered in the coordinates picked by the human observers. In the case of images classified as “object absolutely not present” by the human observer (score = 0), the ground truth consists of a matrix filled with zeros. The second loss consists of a mean square error loss (${\text{Loss}}_{\mathrm{LOC}}$) between the predicted coordinates and those actually picked by the human observers (the latter being used as coordinate labels for training). The contribution of this loss was set to zero for images classified as “object absolutely not present” by the human observer.

A weighted sum of the three losses was tuned during the optimization procedure for the final training as

Eq. (1)

$${\mathrm{LOSS}}_{\mathrm{UNet}}=({\text{Loss}}_{\mathrm{MO}}+100·{\text{Loss}}_{\mathrm{KLD}}+0.1·{\text{Loss}}_{\mathrm{LOC}}).$$

It is worth noting that a specific function based on the softmax operation was built to compute the maximum of the heatmap by means of a differentiable function, an essential requirement for backpropagation to occur properly during CNN training.⁴⁹

The batch size is 48, the learning rate is 0.0001, and the Adam algorithm is employed as the optimizer.

2.3.2.

MobileNetV2-based architecture

The second strategy is based on the MobileNetV2 architecture,⁵⁰ the complexity of which was reduced. We used a MobileNetV2 architecture with fewer convolution layers than the original architecture, i.e., we only used the first 11 layers up to the layer called block_3_depthwise_relu during the optimization procedure to limit overfitting. The MobileNetV2 has already been exploited in the medical imaging field, mostly for classification and detection of lesions,^51–56 and recently it was successfully implemented for COVID-19 diagnosis.^57–60

Two different MobileNetV2-based CNNs are implemented for prediction of the confidence score (represented as a multiclass classification task) and of the object coordinates; their architectures are reported in Figs. 5 and 6, respectively. Two distinct CNNs were built as their architectures are not exactly identical but differ for the final two layers and the input data in the training phase have different sizes in the two cases.

• 1. The CNN devoted to the classification task (Fig. 5) takes as input a CT image and, after the 11th layer of the original MobileNetV2, ends with a global average pooling layer followed by a densely-connected layer, consisting of four units and a softmax activation function to predict the confidence score of human observers. The sparse categorical cross-entropy function is used as the loss during the training phase.

• 2. The CNN devoted to the localization task (Fig. 6) is trained using $48\times 48$ images, obtained by cropping the original $256\times 256$ images around the coordinates picked by the human observers (or random coordinates when the assigned score is 0). The crop size is chosen to be large enough to include the largest insert diameter ($7\mathrm{mm}=36\text{pixels}$). After the 11th layer of the original MobileNetV2, an average pooling layer (pool size 4, strides 1, no padding) followed by a convolutional layer (1 filter, $3\times 3$ kernel size and linear activation function) produces a real number as output, which is then approximated to the closest integer number. In this training phase, a mean squared error loss is used to predict the confidence score.

  Once the training is completed, a convolutional implementation of the sliding window approach⁶¹ is implemented in the test phase to predict the object coordinates: the trained CNN takes as input the original $256\times 256$ images and produces a $27\times 27$ heatmap as output. Each pixel of the heatmap corresponds to a delimited region of the input test image and represents the probabilities that the object is located in the center of that region: the position of the probability maximum provides the predicted coordinates.

Fig. 5

Schematic illustration of the MobileNetV2-based CNN architecture used for the classification task.

Fig. 6

Schematic illustration of the MobileNetV2-based CNN architecture developed for the localization task.

The batch size is 32, the learning rate is 0.001, and the Adam algorithm is employed as the optimizer.

3.

Results

As a preliminary analysis, the performance of the human observers in the task of identifying low-contrast objects within the images was evaluated in terms of the LAUCs reported in Fig. 7, as a function of ${\mathrm{CTDI}}_{\mathrm{vol}}$, in the case of the two reconstruction techniques (FBP top panel, IR bottom panel), for the different contrast values $C$ expressed in terms of the HU difference from the PMMA background ($C=45\hspace{0.17em}\mathrm{HU}$ left panel, $C=55\mathrm{HU}$ right panel). Within each panel in Fig. 7, different curves refer to images with inserts of different diameters.

Fig. 7

Human observer performances quantified by LAUC versus ${\mathrm{CTDI}}_{\mathrm{vol}}$, for different object sizes, contrasts (left: 45 HU, right: 55 HU), and reconstruction techniques (a) FBP and (b) IR.

As expected, the human observer performance improves as ${\mathrm{CTDI}}_{\mathrm{vol}}$ increases, due to CNR increasing at larger radiation dose values. The detectability of the smallest objects (3 mm diameter) is poor for both contrasts $C$ and remains below 90% even at high ${\mathrm{CTDI}}_{\mathrm{vol}}$. The noise in the CT images is correlated, which means that the noise in any point of the image is affected to some extent by the noise values of the neighboring points. Small objects are affected the most by noise correlations. The calculated correlation distance for the highest ${\mathrm{CTDI}}_{\mathrm{vol}}$ images in our dataset, following Refs. 75 and 76, is $\sim 0.7\mathrm{mm}$, which can significantly change the appearance of the 3 mm inserts (1.5 mm radius) and thus make them difficult to be detected even at high radiation doses.

The curve related to the 4 mm diameter objects shows a significant increase with ${\mathrm{CTDI}}_{\mathrm{vol}}$, and saturation of the human observer performances, corresponding to LAUC approaching 1, is reached above 6 to 7 mGy of ${\mathrm{CTDI}}_{\mathrm{vol}}$, though this slightly depends on the contrast and reconstruction technique: objects in IR reconstructed images are better recognized than in FBP reconstructed images.

In the case of inserts with diameters $>4\mathrm{mm}$, saturation of the human observer performance occurs even at low ${\mathrm{CTDI}}_{\mathrm{vol}}$, with LAUC values always above 80%. This result justifies the preliminary selection of the number of images for each contrast and size, as reported in Table 1, in which the more populated subsets are those relative to the inserts of 3 and 4 mm diameters.

Additional statistical analysis was performed to evaluate the difference among the human observers and between the two professional categories that took part in the visual inspection of the CT dataset: radiologists and medical physicists. The LAUC computed for the two classes, reported in Fig. S3 in the Supplementary Material, shows slightly higher performances of the radiologists, especially in the case of the less visible inserts (corresponding to 3 mm diameter inserts). This difference would be, of course, much more significant in the case of complex images, but the scope of this work lies outside the usage of diagnostic images: we aim to exploit the advantages given from using a simple phantom, which can be acquired under different user-defined CT settings (such as protocols and ${\mathrm{CTDI}}_{\mathrm{vol}}$) and in different CT scanners.

Given the above assertions, to achieve the optimization of CNNs, which are notoriously affected by overfitting and biases due to limited data selection, the increased dataset and label variability can be considered an added value, provided that the significant results of this research originates from the analysis of the inserts $>3\mathrm{mm}$ (and, in particular, of the reference dataset).

The performances of trained convolutional neural networks were quantified by computing LAUC as well. The comparison of LAUCs extracted from the whole test dataset among the three observers (two CNNs and the human observer) is shown in Fig. 8, with associated standard errors, in the case of different reconstruction techniques (FBP left panel, IR right panel). A very good agreement is noticeable between the two CNNs and the human observer, especially in the case of IR reconstruction. It can be noticed that at the highest ${\mathrm{CTDI}}_{\mathrm{vol}}$ (10 m Gy) LAUC values of MobileNet show a decrease of performance, an anomalous trend that is present also for some LAUC curves of the human observer in the case of signal with 3 mm diameter (see Fig. 7). This trend is under investigation, and further analysis are under way which include CNR evaluation of the images dataset. Our first hypothesis is that it can be related to the MobileNet noise modeling.

Fig. 8

Overall MO and human observer performances quantified by LAUC versus ${\mathrm{CTDI}}_{\mathrm{vol}}$, for the two reconstruction techniques (a) FBP and (b) IR, with associated standard errors.

According to the previous consideration on the human observer performances, a reference LAUC was selected to evaluate the agreement between human observer and MO, as the one extracted from images containing 4 mm inserts with a lower concentration ($C=45\mathrm{HU}$) was more explicative of the human observer behavior as a function of ${\mathrm{CTDI}}_{\mathrm{vol}}$. In addition, the iterative reconstruction technique, being the most common algorithm used by clinicians in CT protocols, was selected as the reference.

Figure 9 shows the LAUC values for the three observers as a function of ${\mathrm{CTDI}}_{\mathrm{vol}}$ in the case of the reference images subset (4 mm insert, $C=45\mathrm{HU}$, IR reconstruction). The LAUC comparison in the case of the full dataset, i.e., at different diameters, contrasts, and reconstruction techniques, is reported in Figs. S4 and S5 in the Supplementary Materials. A very good agreement between the observers is qualitatively noticeable.

Fig. 9

Comparison of human observer and MO LAUCs versus ${\mathrm{CTDI}}_{\mathrm{vol}}$ for the images with an object size of 4 mm, $C=45\mathrm{HU}$, and IR reconstruction, with associated standard errors.

In order to address the utility of the full training dataset, we performed two additional CNN experiments by using reduced dataset, excluding the 6–7 mm inserts and the 3 mm inserts, respectively. The results of these new experiments are reported in the Supplementary Materials (Figs. S8–S10) and they support the evidence that the dataset variability and numerousness is essential, and it contributes to the success of the training in its totality.

In the following, the level of agreement among the observers is quantitatively addressed by means of appropriate metrics and statistical indices. The MAPE evaluated between the LAUC of the human observer and the LAUC of the two MOs is summarized in Table 3 in the case of the full IR dataset, the full FBP dataset, and the reference subset. Excellent agreement is found between the trained CNNs and the human observer, with an MAPE below 2% in the case of IR reconstruction, slightly above 2% in the case of FBP reconstruction, and in general below 5% when considering all of the image subsets, each related to a different relevant parameter (diameter, contrast, and reconstruction technique), as reported in Tables S2 and S3 in the Supplementary Material. An exception is represented by the 3 mm inserts that are barely recognizable by either the human observer or the MO.

Table 3

MAPE between human observer and MO LAUCs for full IR and FBP datasets and a representative case (Fig. 9).

The accuracy metric, as defined in Sec. 2.4, was evaluated separately for the localization and score prediction tasks: the analysis results as a function of the different variables (inserts diameters, contrast $C$, ${\mathrm{CTDI}}_{\mathrm{vol}}$, and reconstruction technique) are reported in Figs. 10 and 11, respectively.

Fig. 10

MOs localization accuracy metric versus each of the independent parameters, from left to right: insert diameter, insert contrast, ${\mathrm{CTDI}}_{\mathrm{vol}}$, and reconstruction techniques.

Fig. 11

MOs score prediction accuracy metric versus each of the independent parameters, as in Fig. 10.

By looking at Figs. 10 and 11, it is noticeable that the UNet is able to localize slightly better than the MobileNetV2, whereas the latter classifies with a slightly higher overall accuracy than the UNet. This behavior appears consistent with the intrinsic character of the two CNNs, which were initially designed, as reported in the literature, for localization/segmentation and classification tasks, respectively.

Naively expected trends can be observed: the accuracy, in both tasks, increases with object size (insert diameter), contrast ($C$), ${\mathrm{CTDI}}_{\mathrm{vol}}$ (radiation intensity), and IR reconstruction.

In general, the localization accuracy is above 80%, and score prediction accuracy is well above 50%; once again an exception occurs for those images containing the 3 mm inserts.

Furthermore, other common multiclass statistical indices were computed to address the inter-rater agreement in the score prediction task. The values of Cohen kappa, S-statistics, Krippendorff’s Alpha, and ICC, evaluated for the whole images dataset, are summarized in Table 4. The Cohen and S-statistics⁷⁷ indices show a fair to good agreement between the MOs and human observer score predictions, whereas Alpha and ICC indices show a good to excellent agreement (see also Table S1 in the Supplementary Material).

Table 4

Human-model inter-raters statistical indices over the entire dataset.

Moreover, consistent with the previous LROC analysis, a strong correlation between the S-statistics and ${\mathrm{CTDI}}_{\mathrm{vol}}$ is found for both CNNs, as shown in Figs. S6 and S7 in the Supplementary Material. These plots indicate that, when increasing ${\mathrm{CTDI}}_{\mathrm{vol}}$ and the insert diameter, the ability of the CNNs to predict the scores in agreement with the human observer increases as well. A very low value of the S-statistics and no correlation with ${\mathrm{CTDI}}_{\mathrm{vol}}$ are found in the case of the 3 mm insert: the very poor CNR in those images is such that the CNNs are mistaken because they cannot learn from the human observers answers, which are rather imprecise (see also Fig. 7). If the images containing 3 mm inserts are ruled out from the dataset to compute the S-statistics, index values are found to be 0.64 for the UNet and 0.66 for the MobileNnet, indicating a substantial agreement.

4.

Discussion

In this work, we developed and characterized MOs based on artificial intelligence for automatic quality evaluation of phantom CT images. Two CNNs were trained to mimic human observer images assessment in terms of object detection and localization in CT images acquired on a specifically designed and manufactured phantom.

First, we collected a big dataset of phantom CT images containing objects of different sizes and contrasts, acquired at different ${\mathrm{CTDI}}_{\mathrm{vol}}$ settings and reconstructed by means of different techniques (FBP and IR). The dataset was initially submitted to the visual evaluation by human observers to collect the labels necessary for the algorithm training and testing. In this way, the labels fully reflect the human observers’ interpretation of the CT images, regardless of the correctness of human images interpretation, and no internal noise component is necessary to calibrate the CNN-MO on the average human performance.

To verify the viability of our ultimate goal, which is the possibility of CT protocol optimization by means of CNN-MOs, in an almost independent way from the chosen CNN, we implemented two different architectures. UNet and MobileNetV2 were originally built and optimized for the tasks of segmentation and classification, respectively. To the above mentioned purpose, the relation between the CNN architecture and performance was also investigated. We found that both models performed quite similarly, suggesting that there are no critical aspects preventing MO application of them.

In the case of human observers, the confidence score (classification task) and the localization task are intrinsically interconnected and cannot be disentangled in the image evaluation process, whereas CNNs need to be specifically trained to carry out the two tasks, which are partially independent, using different loss functions. The accuracy metrics (Figs. 10 and 11) and the inter-rater agreement statistics (Table 4) show a trend in accordance with the above observation: the two CNNs have different performances in the two tasks of classification (MobileNet is slightly superior) and localization (UNet is slightly superior), as expected. However, the predictions of the two tasks, when combined together, for both CNN-MOs achieve very good overall performances, measured in terms of the LAUC metric. This result supports the robustness of the proposed approach and its being fairly independent from the CNN used. The quality of the trained CNNs was quantified by several statistical indices describing the inter-rater agreement between the MO and human observer in the confidence score task. The statistics computed on the full dataset, ruling out the 3 mm inserts with human detectability that is affected by strong noise correlation, give values of the robust S-statistics above 0.64 for both CNNs, indicating good general CNN performances.

The evaluation of the overall performance of the proposed algorithms in reproducing the human observer response was carried out by computing LAUC, a more accurate metric than AUC because it takes into consideration both localization and classification capability.²⁹ The MAPE calculated between LAUCs extracted from human observer and MO responses was found to be below 2.5%, with slightly higher performances in the case of IR reconstructed images. In addition to the LAUC averaged on the full dataset, we chose a reference subset of images (with a 4 mm insert, $C=45\mathrm{HU}$, and IR reconstruction) reflecting a significant trend as a function of ${\mathrm{CTDI}}_{\mathrm{vol}}$: the LAUC extracted from human observer data of the reference subset (Fig. 9) covers a wide range of values, showing poor detectability performances at low ${\mathrm{CTDI}}_{\mathrm{vol}}$ and then rising until saturation. This curve is a suitable starting point for developing an optimization strategy for the current CT protocol: the value of 6 mGy can be considered the optimum ${\mathrm{CTDI}}_{\mathrm{vol}}$, above which there is no increase in detectability performances, and thus it is reasonable to expect a plateau of the diagnostic accuracy also.

In this work, we demonstrated the viability of an image quality assessment approach based on phantom acquisitions and CNN-MOs, which has a remarkable potential for improvements toward the final goal of dose optimization.

There are several pitfalls and perspectives to consider. We acknowledge several limitations in this study that we plan to address in future works. To finally achieve and implement a CT optimization program, a much more variable CT image dataset, acquired by different CT scanners with well defined setting parameters of a chosen CT protocol is needed. From this perspective, the proposed algorithms need to be retrained on the new dataset, after collection of new human-labeled data. However, we believe that, given the potential of the deep learning methods, the above mentioned effort, representing the next step of the ongoing research, enriched with elevated generalization capability of the algorithms, will be able to avoid the need to repeat the time-consuming reader studies for each protocol. Other limitations that we plan to address in future studies are the decrease of MobileNet performances at the highest ${\mathrm{CTDI}}_{\mathrm{vol}}$, which can be correlated to the CNR and/or noise modeling by the CNN, and the optimization of the phantom design in terms of CNR, which in turn affects the objects detectability.

5.

Conclusion

In this work, we have developed and investigated the applicability of two MO algorithms based on CNN-MOs trained to mimic human observer performances in the phantom CT image detection task. We have demonstrated that two very different AI algorithms are both able to achieve very good results, thus indicating that the proposed approach is robust and fairly independent from the CNN used.

The positive results encourage continuing the exploitation of the proposed methodology toward an automatic image quality assessment based on the evaluation of CT images acquired on a specifically designed phantom; this should foster a systematic optimization, and possible standardization, of the large number of CT protocols currently used in radiological facilities, with the final goal of reaching the best tradeoff between radiation dose and image quality, which is an issue of utmost relevance in diagnostic radiology as emphasized by international organizations (ICRP, IAEA, EURATOM) focused on ionizing radiation risk and radiological protection.