1.1.

Factors Affecting Diagnostic Accuracy

Several factors influence the occurrence of error in the decision process in medical diagnosis. These errors can affect subsequent treatment and have a significant negative impact on patients. In mammography, the characteristics of a case, such as breast density,^8,9 physician experience levels, and metrics describing the visual search process have been associated with the occurrence of diagnostic error during the interpretation of mammograms.^10–14 Studies have examined the perceptual and cognitive dimensions in visual diagnosis.

Perceptual errors, i.e., errors where the observer fails to recognize an abnormality in an image during the detection phase, are a primary cause of missed diagnoses in radiology.^15,16 These types of errors result from cognitive biases,^17–19 faulty visual search,^20,21 or a combination of both. Since there are interrelated cognitive contributors to visual errors, understanding the cognitive processes involved during the interpretation of images can shed light on inherent biases that occur and how these impact diagnostic decisions.

1.2.

Local Sequential Context Bias

Prior research has shown that medical decisions can be influenced by sequential context, particularly in cases where successive stimuli exhibit similar characteristics. Laming²² investigated systematic errors in diagnostic decisions in screen cervical smears. Laming reported that experienced pathologists made systematic errors resulting from psychological interactions in successive cases and offered recommendations on prevention. This type of systematic error, peculiar to sequential stimulus with similar characteristics (as is the case in mammography), indicates that judgments are influenced by features of and decisions about the preceding case in the sequence of examined cases, rather than being based solely on the peculiarities unique to the present case. This type of cognitive bias, known to psychophysicists as sequential context effect,^23–25 is studied in radiology literature and referred to in some cases as context bias.^4,5

1.3.

Global Sequential Context Bias

In contrast to local sequential context bias, global sequential context bias refers to the ability to detect patterns in sequences, such as prevalence in a specific type of case, and subconsciously adjusting a priori probabilities accordingly. In clinical practice, the sequential nature of diagnostic imaging, combined with the ability to detect patterns increases the possibility for biased decisions particularly in borderline cases where the practitioner may be less certain of the diagnostic decision. While previous studies have focused on the occurrence of bias resulting from case prevalence recognized within the case sequence (which may be a test-specific phenomena), fewer have studied the effect of prior diagnostic decisions on subsequent ones.

Sequential biases may occur in combination particularly in borderline cases where the contextual similarities with a previous case and the inference of case prevalence in case sequence combine to influence diagnostic decisions. In this work, we examine the diagnostic decisions of radiologists with varied levels of experience to determine if and to what extent contextual biases occur and if these occurrences can be predicted. To this end, we explore correlations between current and present perceptual behavior, diagnostic decisions, and case characteristics. We hypothesize that radiologist’s prior perceptual behavior and cognitive decisions are predictive of current decisions. To test our hypothesis, we tasked 10 image readers of varied experience to conduct blind reviews of 100 x-ray images. Eye-tracking data and diagnostic decisions were collected from each reader while reviewing four-view mammographic cases under conditions mimicking clinical practice.

2.1.

Data Collection Protocol

A description of methods for this study is provided in detail in a previous work.¹⁰ Briefly, we selected 100 screen-film mammograms from a corpus of mammographic cases available from the University of South Florida’s Digital Database for Screening Mammography²⁶ (DDSM). Each case included the craniocaudal (CC) and mediolateral oblique views (MLO) of both breasts with associated ground truth using BI-RADS™ lexicon.²⁷ Fifty percent of the selected cases were malignant, whereas the remaining 50% was evenly split between normal and benign pathologies. The parenchymal density ranged between 1 (fatty) to 4 (dense), according to the BI-RADS™ lexicon.

For this study, each of the 10 participants (three board certified radiologists and seven radiology residents) performed a review of all 100 cases and provided a report on location and corresponding BI-RADS™ rating of any suspicious mass. Reviews were conducted independently, and markings and BI-RADS™ ratings were reported by each reader through a custom-built graphical user interface, which allowed for image manipulation on dual displays as customary in clinical practice. Readers were outfitted with H6 head-mounted eye-tracking device, with a 60-Hz sampling rate, designed with eye-head integration from Applied Science Laboratories (ASL, Bedford, Massachusetts). Through the remote eye-tracking device, readers’ eye-position data were recorded to within 0.5-deg visual accuracy. Institutional review board approval was obtained prior to conducting this study. Human subject recruitment and data collection were performed according to a protocol approved by the Oak Ridge Site-Wide Internal Review Board. All readers reviewed and signed an informed consent form prior to participation.

Cases for each reader were presented in a randomized order. Although readers examined an identical set of cases, ordering resulting from the randomization process differed for each one. In addition, readers were permitted to complete the study in multiple sessions to accommodate personal preferences and scheduling conflicts. Some readers completed the experiment in two or more sessions on the same day, whereas others completed the experiment over two or more days.

2.2.

Data Processing

Details describing data preprocessing is provided in a previous work.¹⁰ In summary, eye-position data for each reader was aggregated from mammographic views across two displays and first preprocessed to extract a time ordered sequence of fixations along with other associated measures (including fixation duration, interfixation degree, and pupil size) using EyeNAL analysis software from ASL.²⁸ Data provided through EyeNAL software can be further processed to extract more informative features for a deeper analysis.

For our analysis, we used raw gaze data from eye movements to characterize overall visual behavior. Specifically, we characterize the trajectory of the scanpath using a metric known as fractal dimension (FD). The scanpath describes the trajectory of eye movements formed when time-ordered fixations or gaze points are connected. From the scanpath, we computed the $FD$) using the Minkowski–Bouligand box-counting method²⁹ as a measure of complexity in search behavior for each case. Information describing the computation and analysis of this metric can be found in detail in a previous work.¹⁰

The 10 participating readers were grouped into one of three experience levels according to years of training and practice: new trainee resident (N) for those residents with at most two mammo rotations, advanced trainee resident (A) for those residents with greater than two mammo rotations, and expert radiologist (E) for board-certified mammographers. Reported diagnostic decisions for each case were mapped into one of the three case pathologies (normal, benign, and malignant) based on BI-RADS™ rating. Cases without markings (i.e., no scores were given) were designated normal (N); BI-RADS™ ratings {2 and 3} as benign (B); and BI-RADS™ ratings {4A, 4B, 4C, and 5} as malignant (M). Three breast parenchyma density groupings were formed by combining heterogeneous and dense parenchymal cases in the same density grouping because of a small sample size.

Data for each reader were structured into overlapping pairs such that the $n$’th case ($C_n$) was paired with $k$ previous cases ($C_{n-i}$, where $i$ is an integer between 0 and 5) with each $n$’th case representing the current visual behavior and diagnostic decision, and the $i$ preceding cases representing previous visual behavior and diagnostic decisions. Each pair represents the reader’s diagnostic decision for the current case ($C_n$), and the previous behavior and diagnostic decisions (${\bigcup }_{i=0}^5{C}_{n-i}$), which may have potentially impacted it.

Our analysis was performed on each pair by extracting case descriptors. Namely, visual behavior, computed as the FD of scanpath, diagnostic decisions determined as BI-RADS™ ratings reported by each reader for the respective cases, and image characteristics/properties for which we utilize the breast parenchymal density (Fig. 1).

Fig. 1

Illustrates potential sequential dependence between diagnostic decisions ($D_n$) for a case ($C_n$) and the visual behavior (FD $F_{n-i}$), breast parenchymal density ($P_{n-i}$), and diagnostic decisions ($D_{n-i}$) during $i$ previous cases ($C_{n-i}$).

2.3.

Statistical Methods

To perform an initial analysis into the temporal relationship in behavior during mammographic screening, we used a statistical tool: autocorrelation function. Autocorrelation captures the Pearson correlation between values of a sequential process at different times, as a function of these times or the lag between them. It enables finding the correlation between previous measures and measures at subsequent time intervals.

In regression analysis using time series data, autocorrelation in a variable of interest is typically modeled with an autoregressive moving average model (ARMA). We represent the sequence of mammographic cases seen by each radiologist as a time series ($C_1,C_2,\dots ,C_n$). The autocorrelation function gives the correlation ($y$ axis) between the value of measurements at $C_n$ and the value of measurements at $C_{n-k}$ as a function of $k$, where $k$ is an integer representing the lag ($x$ axis).

To determine any combined or individual dependencies between the current diagnostic decisions and the radiologists’ current and previous visual behavior, previous diagnostic decisions, and characteristics of the current and previous images, we performed a multifactor fixed-effects ANOVA with six levels for diagnostic decision (based on the BI-RADS™ rating provided), three levels for breast parenchyma density, and the FD of scanpaths for eye movements.

3.1.

Autocorrelation Tests for Randomness

Autocorrelation is the correlation between a time series lagged one or more time periods and itself. The autocorrelation plot provides a visual representation of the degree of dependence (or independence) between a given time series or sequence and a lagged version of itself over successive intervals. In Fig. 2, we illustrate the autocorrelation function graphs of the FD of scanpaths and of the diagnostic decisions for a subset of readers.

Fig. 2

Illustrates autocorrelation function plots for some readers: (a, b) FD of scanpath and (c, d) diagnostic decision.

The autocorrelation coefficient summarizes the strength of the relationship between present and past values. The shaded region in the graph represents a 95% confidence interval for significance (i.e., those points within this shaded region do not meet this criteria for statistical significance). From Fig. 2, we note a number of nonsignificant correlation peaks, which tend to indicate a random average distribution. However, in a number of cases we identify several significant ($p<0.05$) peaks, which show both strong ($r_k\ge 0.5$) and weak ($0.5>r_k>0.2$) correlation between sequences at their respective time lags.

3.2.

Analysis of Variance

For our analysis, we restricted previous behavior and diagnostic decisions to five preceding cases. Therefore, each pair of observations consisted of the current diagnostic decision (dependent response variable) and a set of the current visual behavior and breast parenchymal density, along with the diagnostic decision, visual behavior, and breast parenchymal density of the five preceding cases (factors).

A multifactor fixed-effect ANOVA was performed to determine any individual or combined dependencies of the current diagnostic decision on current and previous behavior and diagnostic decisions. The ANOVA was performed individually for each reader. First, we mapped the diagnostic decision for each case to one of seven categories based on the provided BI-RADS™ rating: cases without markings (i.e., no scores or marks were given); BI-RADS™ ratings {2 and 3}; and BI-RADS™ ratings {4A, 4B, 4C, and 5}. We formed three breast parenchymal density groupings (fatty, fibroglandular, and heterogeneous/dense). This was achieved by combining heterogeneous and dense cases in a single group, which became necessary resulting from a small sample size.

Next, we formed pairs of observations with the current diagnostic decision at case $n$ ($D_n$) as the response variable, and the current and previous estimates for the FD of scanpath ($F_n,F_{n-1},F_{n-2},\dots ,F_{n-5}$) (continuous factor), the parenchymal density for the current and previous cases ($P_n,P_{n-1},P_{n-2},\dots ,P_{n-5}$), and previous diagnostic decisions ($D_{n-1},D_{n-2},\dots ,D_{n-5}$), as independent factors. In Table 1, we summarize the ANOVA results by presenting a subset of statistically significant individual and combined effects.

Table 1

Multifactor fixed-effect ANOVA test results with FD (F) of scanpath, prior diagnostic decision (D), and breast parenchymal density (P) for readers by experience level: new (N) and advanced (A) radiology resident, and expert (E) radiologists.

3.3.

Predictive-Model-Based Analysis

As described in Sec. 3.1, each data point consists of a pair: diagnostic decision ($D_n$) at case $C_n$ as target, and reader’s diagnostic decision and behavior for $C_{n-i}$ previous cases, where $0\le i\le 5$ (our analysis was limited to a maximum of five previous cases) as features. This process results in six distinct datasets representing ${\bigcup }_{i=0}^5(C_n,C_{n-i})$ pairings with a maximum of $m=100-i$ samples for each subject in each dataset. During the experimental protocol (described in Sec. 2.1), readers were permitted to complete the experiment over multiple sessions. The final number of pairs for each reader varied 85 to 98 (with a median value of 94) dependent upon the number of sessions, resulting in $m=100-i\times (s+1)$ pairs, where $s$ is the total number of sessions.

To assess the predictive potential of the observed dependencies between current diagnostic decisions and previous behavior, we developed a predictive model using a random forest classification model. The model was evaluated using a within-subject leave-one-case-out cross-validation model evaluation scheme. To achieve this, data were first partitioned into 10 independent datasets, representing response data from each of the 10 readers. Leave-one-case-out cross-validation scheme partitions each dataset into complementary subsets, where all but a single data sample is utilized in model-training (training set), which analysis is validated on the excluded sample (test case). This process is repeated until every sample in the dataset is utilized as the singular test case, resulting in $m$ models (where $m$ is the total number of samples).

The aggregated predictive values over all rounds serve as the final performance evaluation of the predictive model for each user. In this paper, we report the weighted-average $f$-score (the harmonic mean of precision and recall) performance metrics for predicting diagnostic decision on a current case using features derived from previous behavior and diagnostic decisions. In Table 2, we report the $f$-score for each user by $i$ (number of previous cases) and, for comparison purposes, Table 2 also includes performance metrics for a random chance classifier (a random classifier provides a simple baseline reflecting the predictive outcome of a random guess).

Table 2

Performance (f-score) of behavior-based prediction task for readers by experience: new (N) and advanced (A) radiology resident and expert (E) radiologists.