1.

Introduction

Medical images are routinely used to identify lesions within the human body. These images are read by radiologists, who generally make the final judgment about the presence of a lesion. Image quality is crucial for the radiologist to make the best possible judgment and may be improved by optimizing system designs and acquisition protocols. However, it is expensive and time consuming to conduct human-observer studies for this purpose at every stage of developmental research. This has led to the development of mathematical model observers as surrogates for humans in diagnostic imaging studies. Of particular interest are model observers which can predict human performance in clinically realistic tasks involving lesion search. Such tasks should probe how quantum and anatomical noise in the images affect observer performance. Anatomical noise is background structure that masquerades as a lesion or which obscures actual lesions. Together, quantum and anatomical noise comprise the image texture.

Popular model observers such as the channelized Hotelling (CH) observer¹ operate based on prior information in the form of image statistics. The statistics include covariance matrices that characterize the noise in an image set. The level of prior information afforded an observer is dictated by several well-known task paradigms. For signal-known-exactly detection tasks, the observer need only decide lesion presence at a fixed location. Information about the background can be either exact [background-known-exactly (BKE)] or statistical [background-known-statistically (BKS)]. The task of lesion detection and localization is signal-known-statistically (SKS) in nature (e.g., the shape of the lesion profile may be known although its location is variable), but may additionally be either BKE or BKS. These search tasks can be performed by scanning versions of the CH observer and other Hotelling-type models.²

With a BKE search task, a scanning model observer has knowledge of the quantum-mean background corresponding to the given test image. This knowledge greatly reduces the impact of anatomical noise on task performance. For BKS search tasks, the scanning CH observer computes image class means and covariances that account for anatomical variations. This computation can be extensive since the covariance matrices must be computed at all possible lesion locations. Moreover, it is unclear whether, given the covariance information, the scanning CH observer can adequately quantify how human observers respond to anatomical noise in individual images.

As an alternate approach, we have been investigating observer models which use less statistical knowledge.^3,4 These models perform distinct target search and analysis procedures, thereby simulating the visual-search (VS) process that trained radiologists engage in to interpret medical images.⁵ Our VS model observer applies a feature-guided search that identifies target-like structure for subsequent analysis. For this study, the observer identified hot “blobs” (regions of elevated tracer uptake) as candidates, which may be actual lesions or false positives. Statistical information about the image background was only supplied for the candidate analysis, at which point some nonconspicuous lesions may have been ignored. The SKS task paradigm is a key constraint for the model, as many of the complex processes of human vision and cognition are not accounted for. The effects of image texture are nonetheless intrinsic to search performance with this relatively simple model.

We tested the VS and scanning CH observers as competing ways of predicting human-observer performance in a realistic search task involving both quantum and anatomical noise. The task was detection and localization of pelvic lesions in simulated In-111 planar nuclear medicine images. A localization receiver operating characteristic (LROC) study examined how observer performance was affected by the resolution-versus-sensitivity trade-offs among a family of medium-energy imaging collimators. The human observers in our study read a subset of the images considered by the model observers. Comparison was also made with a channelized nonprewhitening (CNPW) observer that performed both BKE and BKS search tasks.

Human observers are often modeled as ideal observers with added inefficiencies.⁶ In our study, the CH and VS observers were applied with and without internal noise that was added to the observers’ decision variables. In addition, the VS model was tested with a perceptual search threshold⁷ that rejected relatively nonsuspicious search candidates. Various covariance-based internal-noise models have been proposed for the CH observer,⁸ but the motivations for choosing one model or another are often unclear. The inefficiencies that were tested for the VS observer may afford a better intuition.

2.

Model Observers

3.

Methods

3.2.

Model-Observer Inefficiencies

3.2.1.

Internal noise

Given the same image, human observers can make different decisions at different times. With the model-observer formulations presented above, the decision for a given image would never change. To address this issue, we experimented with adding internal noise to the CH and VS observers. The noise was implemented by adding a random Gaussian deviate to the perception measurements ${\lambda }_j$. The width of the Gaussian distribution was selected relative to the standard deviation ${\sigma }_I$ of the set $\{{\lambda }_j:j\in \mathrm{\Omega }\}$ for a given set of training images. With $N(\mathrm{0,1})$ representing a random deviate from a Gaussian distribution with zero mean and unit standard deviation, the noisy measurement was given by

Eq. (13)

$${\tilde{\lambda }}_j={\lambda }_j+N(0,\gamma {\sigma }_I).$$

Values of $\gamma$ between 0.5 and 2.0 were tested.

For the CH observer, internal noise can also be implemented through the covariance matrices instead of the analysis statistic. Several models of this type are discussed by Zhang et al.⁸ A corresponding process for the VS observer would be to add internal noise during the candidate search.⁷

3.2.2.

Perceptual search threshold

A second type of model inefficiency applied in our study was specific to the VS observer. Recall from Sec. 2.4 that the VS observer identifies blob focal points as suspicious candidates by means of image segmentation. The segmentation can produce many candidates that human observers would ignore. The result, particularly with high-count images, is that the BKE analysis with the CNPW discriminant will often locate the lesion. In such cases, the VS and CNPW-BKE observers may perform similarly. A solution is to limit the set of candidates by setting a threshold $\tau$ such that a focal point with greyscale intensity less than $\tau$ is disregarded. Parameter $\tau$ is thus comparable to the drowning threshold described in the watershed literature.¹⁴

This threshold was calculated with reference to the focal points from a set of training images. Each training image yielded a number of focal points, some of which were lesion-absent. The set of greyscale intensities from all of the lesion-absent focal points in the training images produced a distribution with mean and standard deviation denoted by ${\mu }_l$ and ${\sigma }_l$, respectively. We tested search thresholds of the form $\tau ={\mu }_l+\beta {\sigma }_l$ for four different scaling factors $\beta$ between 0.2 and 0.8.

Figure 1 shows the greyscale distributions of the normal and abnormal focal points obtained from a training set of images. The distributions are approximately Gaussian. Threshold values of particular interest might lie in the zone between the dashed vertical lines where the disease status is not obvious.

Fig. 1

Example greyscale intensity distributions for the normal and abnormal focal points identified by the VS observer in an image set.

3.4.

Collimators and System Resolution

Our LROC study compared observer performance with different collimators. This section reviews the properties of a gamma-camera collimator and how these properties affect image formation. The two main parameters for a parallel-hole collimator are the hole length $L$ and the hole width $d$, both of which affect the collimator blur and count sensitivity. The number of counts per unit time $\eta$ for regular hexagonal holes obeys the proportionality $\eta \sim 0.65d^2$.¹⁶

The system spatial resolution $R_{\mathrm{sys}}$ of a gamma camera has two components: intrinsic resolution and distance-dependent geometric resolution. Expressed in terms of the full-width at half-maximum of a Gaussian blur function, the resolution for photons emitted at a distance $c$ from the face of the collimator is

Eq. (14)

$$R_{\mathrm{sys}}(c)=\sqrt{R_{\mathrm{int}}^2+R_{\mathrm{geom}}^2(c)}.$$

The geometric component of the system resolution is completely determined by $L$ and $d$ according to the formula

Eq. (15)

$$R_{\mathrm{geom}}(c)=\frac{dc}{L},$$

while the intrinsic resolution depends on the gamma-ray energy and the crystal material. For the In-111 energies and a NaI detector, typical intrinsic resolutions are between 0.3 and 0.4 cm.¹⁷ Note that geometric resolution can also be defined in terms of distance from the face of the crystal.

For typical values of $c$ in patient imaging, the system resolution may be expressed with the linear model

Eq. (16)

$$R_{\mathrm{sys}}(c)=mc+R_0,$$

with slope $m=d/L$ and the intercept $R_0\approx R_{\mathrm{sys}}(c)-mc$ for a suitably large value of $c$.

Our study modeled variants of a typical medium-energy collimator with $L=4.06$ cm and $d=0.294$ cm. Eight collimators were defined by varying $d$ between 0.044 and 0.394 cm in increments of 0.05 cm while keeping $L$ constant. The intrinsic resolution was fixed at 0.38 cm. The relevant collimator parameters are listed in Table 1. The collimators are labeled using the notation C1, …, C8, listed in order of increasing $d$. The resolutions given in the table were computed for a distance $c=4L$. The sensitivities $\eta$ have units of millions of counts per unit time.

Table 1

Parameters for the eight collimator models. System resolution Rsys was measured at distance c=4L from the collimator face. The count sensitivity η has units of millions of counts per unit time.

	Collimator
	C1	C2	C3	C4	C5	C6	C7	C8
$d$ (cm)	0.044	0.094	0.144	0.194	0.244	0.294	0.344	0.394
$R_{\mathrm{sys}}$ (cm)	0.402	0.473	0.575	0.695	0.824	0.960	1.100	1.240
$\eta$	0.04	0.19	0.45	0.81	1.23	1.87	2.55	3.35

As $\eta$ and $R_{\mathrm{sys}}$ both increase monotonically with $d$, our study examined the trade-off between sensitivity and spatial resolution in terms of observer performance. Low sensitivity leads to relatively higher quantum noise in the images, whereas low resolution increases anatomical noise due to partial-volume and blurring effects. Sample images obtained with the collimator models for two lesion-present cases are shown in Fig. 2.

Fig. 2

Sample study images obtained for a pair of high-contrast lesion cases: (a) a prostate lesion; (b) a lymph-node lesion. For both cases the first row consists of images from collimators C1 to C4 and the bottom row C5 to C8. The lesion position for each case is indicated by the white arrow in the first image.

4.

Results

The human-observer results from our study are summarized in Table 2. The uncertainties in individual observer performance were on the order of $\pm 0.06$. The two-way ANOVA indicated a significant collimator effect ($p=8.6\mathrm{E}\text{-4}$) but the observer effect was not significant ($p=0.075$). A subsequent Tukey multiple comparisons test²¹ found significant differences between C6 and the group {C2, C3, C4}. Collimator C5 was positioned in-between, with the data unable to distinguish it from C4 or C6.

Table 2

Individual and average human-observer performances for the five tested collimator models C2–C6. The uncertainties in AL for the individual observers are ±0.06. The uncertainties in average performance represent one standard error in the mean.

Figure 3 shows how the model observers without internal noise or search threshold performed as a function of collimator and lesion contrast. The average human-observer results are included in Fig. 3(b). The uncertainties in the $A_L$ estimates for each of the model observers did not exceed $\pm 0.02$. With the high-contrast lesions, the model observers showed some qualitative similarities with the humans for the low-sensitivity collimators, but did not match the substantial drop in human performance that occurred with increased sensitivity.

Fig. 3

Model-observer and average human-observer performance as a function of collimator and lesion contrast. The model observers were applied without inefficiencies. (a) Model-observer performances with the low-contrast lesions; (b) human and model-observer performances with the high-contrast lesions. Uncertainties for the model observers did not exceed $\pm 0.02$.

The ANOVA (Table 3) and multiple-comparisons testing applied to the model-observer data indicated that collimator C1 performed significantly worse than the other collimators. The differences between each observer were also significant. Affected almost entirely by quantum noise, the CNPW-BKE observer consistently outperformed the other three model observers, each of which also contended to some degree with anatomical noise. In particular, the CNPW-BKE observer performed at a high level with collimators C2 to C8 regardless of lesion contrast. The differences in performance that occurred between the CNPW-BKE and VS observers are attributable solely to the initial candidate search performed by the latter. The greatest deviations from CNPW-BKE performance occurred with the CNPW-BKS observer for collimator C1. The two CNPW observers otherwise performed similarly for collimators C2 to C4, only beginning to diverge once more with increased anatomical noise. The performance trends for the CNPW-BKS and CH observers were also similar, although the covariance noise modeling for the CH observer substantially moderated the effects of the reference-image subtraction for some collimators. The CH observer consistently outperformed the VS observer.

Table 3

Results from the three-way ANOVA conducted with the model-observer scores. The analysis tested collimator, observer and lesion contrast as factors. All three effects and two-way interactions were significant at the α=0.05 level.

The performance trends in Fig. 3 for the scanning observers were largely independent of lesion contrast. This was not entirely so for the VS observer, for which Fig. 3(b) demonstrates an upward deflection in performance with the higher collimator sensitivities that does not show in Fig. 3(a). The deflection is small, with $A_L$ increasing by 0.02 between C4 and C8, while performance with the low-contrast lesions decreased over that span. The BKE analysis is at the root of this variable behavior, amplifying the image quality effects of the higher lesion contrast and increased sensitivity. As shown below, some of the effects of the BKE assumption can be mitigated by the addition of perceptual inefficiencies to the observer.

Of course, the scanning models also relied on some prior knowledge of the image backgrounds that the human observers did not have. Internal noise is routinely added to Hotelling-type observers to compensate for this knowledge. The results from adding Gaussian noise to the perception measurements for the high-contrast lesions with the CH observer are shown in Fig. 4. The standard errors in the estimates of $A_L$ for the model observers with inefficiencies were $\sim \pm 0.03$. Within the range of $\gamma$ values tested, the added noise steadily reduced performance at all collimator sensitivities, but tended to penalize performance with the lower sensitivities most. Thus, none of the model observer curves in Fig. 4 reproduced the performance trend displayed in the average human observer data. The left and right ends of the human performance curve were quantitatively fit with different values of $\gamma$.

Fig. 4

Effects of internal noise on CH observer performance with the high-contrast lesions. The noise parameter $\gamma$ varied from 0 to 2.0, with $\gamma =0$ indicating no noise. Also shown is average human performance. The uncertainties in model-observer performance were $\pm$0.03.

Figure 5 shows the individual effects of internal noise and search thresholding for the VS observer. Figure 5(a) pertains to the internal-noise results. As with the CH observer, changes in $\gamma$ generated performance changes that were fairly uniform with sensitivity. However, the VS observer was less affected by a given value of $\gamma$ compared to the CH observer. An interesting outcome for $\gamma \ge 1.5$ was the establishment of a nominal performance peak for C3 that mirrored the average human observer results.

Fig. 5

Individual effects of internal noise and search thresholding on VS observer performance with the high-contrast lesions. (a) Effects of internal noise; the noise parameter $\gamma$ varied from 0 to 2.0, with $\gamma =0$ indicating no noise. (b) Threshold effects; the threshold parameter $\beta$ varied from 0.2 to 0.8. The uncertainties in model-observer performance were $\pm 0.03$.

With the search threshold [Fig. 5(b)], the VS observer was able to duplicate the considerable drop in average human performance that occurred at the higher sensitivities. However, the threshold had a negligible effect on observer performance at the lower sensitivities. The relatively high system resolutions with those collimator models ensured that actual lesions would generally be associated with focal points having greyscale maxima that exceeded the threshold.

We also investigated how the VS observer responded with internal noise and the search threshold together. There were 16 inefficiency models in all, corresponding to the four values of $\beta$ and four nonzero values of $\gamma$ that were used for the plots in Fig. 5. Figure 6 compares the average human performance with the best-fitting results from the VS observer that used $\gamma =1.5$ and $\beta =0.6$.

Fig. 6

Effects of applying the VS observer with multiple inefficiencies. Average human performance is compared with the performance from the VS observer corresponding to $\gamma =1.5$ and $\beta =0.6$. The uncertainties in model-observer performance were $\pm 0.03$.

The computation times for the scanning and VS model observers in this study were disparate. Recall that the channel covariance matrices for the CH observer were computed (but not inverted) prior to the study. Despite this preprocessing, the CH computation times were relatively high: $\sim 90\min$ in IDL (Interactive Data Language, Exelis Visual Information Solutions, Boulder, Colorado) to read 450 test images compared to less than a minute for the VS and CNPW observers. These timings are for the model observers without internal noise or search thresholding. The lesion search region for the model observers contained 380 locations. From this, the VS observer averaged between 20 and 40 focal points per image, with the higher numbers associated with lower collimator sensitivities. However, the code implementation of the candidate analysis did not exploit this search reduction.

5.

Discussion

Our objective with this work was to compare the VS and scanning CH observers as surrogates for human observers in task-based assessments. A focus for the comparison was how these model observers respond to anatomical noise. The image sets in the planar imaging study presented varying proportions of quantum and anatomical noise as dictated by the collimator sensitivity and spatial resolution. Without inefficiencies, neither model observer matched the magnitudes or general trend in average human performance as anatomical noise increased in the images. The results with the observer inefficiencies show that the two models admit different sets of mechanisms for improving the agreement with the humans.

One must keep in mind the different task paradigms used with the CH and VS observers. With the CH observer, anatomical noise is equated with background variability as quantified by the image class statistics. Adding internal noise terms to the channel covariance matrices is a common means of reconciling differences with human observers. One may also inject greater anatomical noise into the model observer by modifying the reference image c_j [Eq. (6)], although comparison of the CNPW-BKE and CNPW-BKS results suggests this approach primarily affects observer performance with the higher-resolution collimators. Previous studies have also shown that excessive modification of the reference image can lead to artifacts that create persistent false positives for the observer.³

For the VS observer tested herein, anatomical noise only affects the initial candidate search since the analysis is a BKE process. Furthermore, the separate influences of quantum and anatomical noise are not discerned in the search. Instead, image texture as a whole impedes the identification of lesions as focal points. Accounting for noise in this manner effectively expanded the dynamic range in observer performance across the full set of collimators compared to the scanning models. Translating this expansion into greater statistical power will depend on developing methods of candidate analysis that are independent of the BKE and BKS paradigms. The use of greyscale intensity as the sole search feature for the VS observers in our study was only possible because of the BKE analysis. One promising alternative is based on the extraction of multiple morphological features.²²

Overall, the VS framework provides a relatively flexible environment for studying observer inefficiencies. As shown in Fig. 5, different inefficiencies can influence different parts of parameter space, offering an intuition that is not always available with covariance modeling of internal noise. The VS observer attained quantitative agreement with the human observers, but this was the result of ad hoc fitting of the internal noise and search threshold parameters ($\gamma$ and $\beta$, respectively). A version of the VS observer that can be reliably applied without the BKE or BKS background paradigms may require smaller amounts of internal noise and other inefficiencies to approximate human-observer performance, and discussion of how to set the inefficiency and noise parameters should await development of such a model.

Computational efficiency is another important point of comparison for the CH and VS observers. Runtimes for the CH observer were substantially longer than for the VS observer. Preinverting the covariance matrices would have improved efficiency, but the main factor in the longer times was the shift-variant CH observer template. With regard to the amount of preprocessing required for the CH observer, one must recognize that the background variations in this study were restricted to modeling different In-111 biodistributions in a single geometry of the XCAT phantom, so that the computation of ${\mathbf{K}}_j^{\mathrm{anat}}$ [Eq. (10)] and $\overline{\mathit{b}}$ was relatively simple. A more extensive study involving multiple phantom geometries could complicate the BKS computations by involving intensive Monte Carlo simulations as demonstrated for nonsearch tasks by Kupinski et al.²³ Observer studies with reconstructed images would substantially increase the computing time,²⁴ as would studies involving larger regions of interest, high-resolution imaging or image volumes.

Finally, there were a number of basic limitations with this study for purposes of collimator optimization, including the use of an analytic projector that disregarded the considerable scatter and penetration potential of In-111. Still, the results are relevant to the practice of task-based collimator optimization. The accepted approach to hardware optimization is to apply ideal observers in projection space; the scanning CH observer has been used previously as an approximate ideal observer (although with channels other than what was used herein²⁵). VS observer development has focused instead on modeling human observers as the gold standard for lesion-detection performance in much of diagnostic imaging. The planar imaging framework allows for an even comparison of the two approaches to optimization. Follow-up studies will analyze the prospects for optimization studies based on increasingly realistic detection tasks with the VS observer.

6.

Conclusions

When applied without inefficiencies, the VS and CH observers failed to match either the magnitudes or the general trend in average human performance as anatomical noise increased in the images. The result for the VS observer is partly attributed to reliance on the quasi-BKE task, which artificially inflated observer performance in images with high anatomical noise. Overall, the VS model observer offers a flexible and computationally efficient framework for studying the combined effects of quantum and anatomical noise in lesion search tasks. Future studies will examine versions of the model that are independent of the standard BKE/BKS task paradigms.