1.

Introduction

Observer performance studies are commonly used in radiology to determine the relative fitness of imaging techniques for a particular purpose. Typically, human observers view images that differ from one another in some way to gain insight into which might be better for a particular task. Two examples, taken from the articles considered for this paper, of research questions that may be addressed using an observer-performance study follow: (1) Does chest radiography with dual-energy subtraction images improve radiologists’ ability to find small lung cancers as compared with chest radiography without dual-energy subtraction images?¹ (2) Does computer-aided detection improve radiologists’ interpretations of computed tomographic (CT) colonography?² At other times, it may not be the images that change, but rather there is a change in their presentation or in the reading environment. Depending on the design of the experiment, observers may be asked to complete a specified task under two or more different conditions or to choose which of two or more images are best suited for a task. We have called the different image types or reading conditions that are being compared “tested conditions.”

Different experimental designs are associated with greater or lesser risk of memory bias. In alternative forced-choice designs, the observers are asked to choose the best suited among two or more tested conditions that are presented at the same time. For example, are flat-panel display monitors or high-resolution gray-scale display monitors better suited for display of anatomic features in chest radiographs? Balassy et al.³ addressed this question by asking observers to look at the same images projected side by side on different monitors and rank the visibility of anatomic features. In multipoint rank-order designs, an observer is asked to rank several tested conditions in order from best to worst. Sivaramakrishna et al.⁴ asked observers to compare unenhanced mammograms and the same images altered by four different enhancement algorithms, then rank them from best to worst for display of microcalcifications and masses, two findings that can indicate the presence of breast cancer. Such studies do not generally harbor a risk of memory bias. Indeed, the investigators would want the observers to remember image A when looking at image B as it is otherwise not possible to form an intelligent opinion of their relative merits.

In what we term sequential-viewing experiments, an image is viewed alone, and then immediately afterward it is seen along with an additional image or piece of information. Two examples from our reviewed articles were the study by van Rijn et al.⁵ in which observers interpreted spine MRI images alone and then with clinical information and the study by Li et al.⁶ in which observers rated the likelihood of malignancy of pulmonary nodules on CT first without and then with the results of evaluation by computer-assisted diagnosis. In sequential-viewing studies, memory bias is also not a concern as the images or other information displayed as part of the first tested condition are deliberately made available during the second interpretation. This is usually done to mimic the way the two tested conditions might be used together in actual image interpretation.

In study designs in which the tested conditions are displayed separately, memory for a previously encountered image does have the potential to create bias. In such designs, rather than looking at two CT images of the liver at the same time and choosing which one shows a tumor better, the task might be to look at them separately and try to find the liver tumor. The observers’ actual level of success at identifying liver tumors, rather than the observers’ opinion, would then indicate which type of CT image was more suitable for that task. In this type of study, if the observer remembers the first image while looking at the second image, that memory could affect the second reading and therefore bias the study.

Metz⁷ recognized the potential for bias due to recognition memory. He suggested that investigators arrange the order of readings so that any bias created by memory would cancel out. The simplest way to do this would be, in the case of an experiment comparing two different tested conditions, to have half the observers view images in tested condition A first and the other half of the observers view images in tested condition B first. This is termed counterbalanced methodology. Metz also suggested that, in addition to using such a counterbalanced approach, a time gap as long as possible should be used between viewings of the same image. Almost 20 years later, Sica⁸ made similar recommendations.

Since then, several studies have looked at the effect of time gaps between readings on memory for previously encountered images. They have found that very short time gaps of a few minutes to a few days are associated with slightly better-than-chance memory for images when observers were shown sets of single images in moderate numbers of about 20 to 40.^9–11 A short time gap of 2 days also created memory bias for fused positron emission tomography and CT (PET/CT).¹² A time gap of 50 days was sufficient to eliminate any conscious memory of previously viewed images.¹¹ Others, using nonmedical images, have found a sharp drop in recognition memory for images over the first month after they were originally viewed, though any images that had not been forgotten in a month might still be remembered a year later.^13,14

There has not been similar research regarding the contribution of other methods such as counterbalanced order of tested conditions as mentioned above to elimination of memory bias in observer-performance studies. In addition, it is not known what researchers are actually doing to cope with memory bias in observer-performance studies. Without this knowledge, it is difficult to consider whether any improvements in methodology may be warranted. We have studied 100 articles (110 experiments) from the online version of the journals American Journal of Roentgenology (AJR) and Radiology to determine what steps authors of published articles used to deal with potential memory bias.

2.

Materials and Methods

3.

Results

Data are presented with reference to the number of experiments analyzed. We reviewed 50 articles in AJR, in which 57 separate experiments were reported, and 50 articles in Radiology, in which 53 experiments were reported. Therefore, the total number of experiments was 110. Year of publication ranged from 1973 to 2016. Seven experiments (10.3%) were published between 1973 and 1979; 9 (13.2%) between 1980 and 1989; 9 (13.2%) between 1990 and 1999; 25 (36.8%) between 2000 and 2009; and 18 (26.5%) between 2010 and 2016.

The methods by which tested conditions were organized for presentation to observers are shown in Table 1. Separate viewing, in which the tested conditions are viewed apart from one another, was the most common choice and was used in 66 experiments (60.0%). Forty-one (37.3%) experiments used methods of ordering of tested conditions that are not sensitive to memory bias and therefore were not included in additional analysis. Three experiments (2.7%) did not indicate how tested conditions were ordered for viewing, and one of these experiments was excluded because comparison of very different-appearing modalities (MRI and ultrasound) made memory bias quite unlikely. This left 68 experiments for further analysis, 32 from AJR, and 36 from Radiology, respectively.

Within these 68 experiments, method of ordering of tested conditions can be evaluated in further detail as in Table 2. The most common subcategory was counterbalanced ordering, in which order of viewing of tested conditions is deliberately alternated. Twenty-two (32.4%) experiments used this method. Adding together those 22 experiments plus 6 that used random ordering and 2 that used pseudorandom ordering give a total of 30 (44.1%) experiments that used a method that will mitigate memory bias. Eleven (16.2%) experiments intermixed the tested conditions. This method may or may not mitigate memory bias depending on how the individual images were ordered for viewing within sets. Fifteen (22.1%) showed at least 1 tested condition in the same order to all viewers. Twelve (17.6%) experiments did not provide sufficient detail for further characterization.

Among these 68 experiments, ordering within a set of images to be viewed in one observation session was random in 40 (58.8%) experiments. Three (4.4%) experiments ordered them using a pseudorandom method. Reports for 25 (36.8%) experiments did not indicate how the images were ordered within sets.

Correlation between the decade in which the experiment was reported and either the methods used to order tested conditions for viewing or the methods used to order images within sets did not reveal any statistically significant difference over time.

Fifty-two of the 68 experiments (76.5%) indicated whether a time lapse was used between viewings. Six (8.8%) experiments did not use a time gap. Time gaps in experiments that utilized them ranged from 1 day for 6 (8.8%) experiments to 730 days for 1 (1.5%) experiment. Average time lapse was 36.6 days (40.5 days if those without a time lapse are excluded from the calculation). Median time lapse was 14 days whether calculated with or without the six experiments without a time lapse. The most commonly used time lapse was 7 days, used in 8 (11.8%) experiments, followed by 1 day, 14 days, and 28 days, each used in 6 (8.8%) experiments. Sixteen (23.5%) experiments did not indicate if a time lapse was used. (See Table 3)

Table 3

Time gap in days between viewings of the same or similar images.

Reports for 24 (35.3%) of the 68 experiments did not indicate how the authors dealt with 1 of the 3 design elements we studied—either use or nonuse of a method of ordering tested conditions that would mitigate memory bias, method of ordering of images within sets, or use of a time gap. Reports for 7 (10.3%) of the 68 experiments did not indicate how the authors dealt with 2 out of the 3 design elements. Reports for 5 (7.4%) of the 68 experiment did not indicate how the authors dealt with any of the 3 design elements. Therefore, reports for 36 (52.9%) experiments did not indicate how authors dealt with at least 1 of the 3 elements.

4.

Discussion

Metz⁷ and Sica,⁸ writing on design of imaging studies, both urged investigators to adopt methods that can mitigate memory bias. Our results showed that 22.1% of the 68 studies that were potentially affected by memory bias did not use methods of ordering the tested conditions that would decrease memory bias. The methods section of scientific papers should describe the research plan in sufficient detail that the experiment could be reproduced if desired. The STARD guidelines, both as originally published in 2003 and as updated in 2015, emphasize the need to describe methods clearly, with part of the intent being to allow readers to determine what sorts of bias may exist in the results.^17,18 Nonetheless, we found that over half of the reports of these experiments left unstated how the authors dealt with at least one design element significant to memory bias.

We suggest that improvements may be made with respect to memory bias both in designing experiments and reporting them. Whenever possible, investigators using an experimental technique that is subject to memory bias should choose design components that will mitigate such bias including, for example, counterbalanced presentation of tested conditions, random ordering of images within sets, and use of a time gap between viewings of the same or similar images. The methods used should then be reported. A study of detection of pulmonary nodules on chest radiography by Szucx-Farkas et al.¹⁹ is a good example of such reporting. The methods by which the tested conditions were ordered for viewing and the images were ordered within sets, and the use of a time gap were all set forth in one paragraph.

Like all studies, ours has limitations. One is that even small changes in the design of the study could have resulted in inclusion of a different assortment of articles and therefore have altered results. For example, we wished to include articles from two old, established journals that include reports on a wide variety of radiological topics. Although we believe that Radiology and AJR online were good choices, different journals could have been chosen. Another is that the variations brought by investigators to the design of their experiments were many and imaginative. We could not discuss 110 experiments individually and therefore had to group them together into categories depending on various aspects of experimental design. It is possible that not everyone would agree in every instance with our final decision as to which category might fit a particular experiment best. Nonetheless, we believe that we have gathered a representative sample of published articles and have garnered a good sense of what authors of observer-performance studies are doing to manage subjects’ memory of images. Furthermore, any investigators attempting this research would face the same challenges and therefore have the same limitations.

5.

Conclusions

We urge that reports of observer-performance studies that involve repeated viewing of individual images or closely related images should include details relevant to handling of observers’ memory. Specifically, they should indicate how they organized both the tested conditions and the images within sets for viewing. They should indicate if a time gap was utilized and, if so, how long it was. Including this information will allow other researchers to accurately replicate previous studies’ experimental procedures and will allow readers to judge whether memory bias was likely to have played a role in the results.

In designing observer-performance experiments, we would recommend that experimenters should minimize memory bias when observers will be viewing the same image (or closely related images) more than once (and not in a sequential-viewing methodology). This can be done by showing the images in different and random orders across viewings and observers. Entire sets of images can be alternated in a counterbalanced fashion to cancel out the memory effect that might occur if all observers saw the same tested condition before the other. Individual images can also be alternated in a more dynamic fashion so that observers encounter different assortments of images during different parts of the experiment. If experimental conditions allow for a time gap, seven weeks should be adequate between viewings, and if other memory-mitigating steps are taken, even without a gap, conscious memory for images should play a minor role.^9–11