2.1.

Automated Breast Ultrasound Imaging

ABUS images are acquired by a wide linear array ultrasound transducer sliding continuously over one breast, which is gently compressed by a dedicated membrane while the patient lies in a supine position. During the sliding motion of the transducer, the ultrasound scanner acquires more than 300 transversal images covering a large segment of the breast. These single slices are stacked to form a 3-D ultrasound image that can be examined in multiplanar reconstructions.¹⁵ Depending on the size of the breast, three to five views of each breast are acquired. The positioning and compression of the breast are standardized to some extent and include anterior–posterior (AP), lateral (LAT), medial (MED), superior (SUP), or inferior (INF) views, the breast being gently pushed in these directions, respectively. The latter one (INF) is acquired very rarely and was not contained in our datasets.

2.2.

Automated Breast Ultrasound Image Quality Aspects

The focus was put on the three most frequent quality aspects that could be avoided by a repeated scan. The first problem is an incorrect nipple position within the image. In some cases, the nipple is pushed very close to the edge of the breast in coronal view [see Fig. 1(a)]. This may cause severe posterior acoustic shadows, obscuring anatomical structures behind the nipple, which can usually be avoided by proper repositioning of the transducer. The second issue is the shadow of the nipple [see Fig. 1(b)]. In the area around the nipple, there is commonly no perfect contact between transducer and skin, resulting in an acoustic shadow behind the nipple on the ultrasound image. Air-filled ducts may contribute to this effect. In most cases, the image is nevertheless usable for diagnostics, but sometimes, the shadow covers noteworthy parts of the breast tissue. Applying more contact gel in a repeated scan often resolves this problem. The third aspect is also correlated to the positioning of the transducer and the breast. If the breast is not supported correctly by the provided cushions, there might be a lack of contact and the outer regions of the breast will not be not imaged [see Fig. 1(c)]. This results in large background areas in the image as well as irregular breast contour lines.

Fig. 1

Sample ABUS images with artifacts. (a) Nipple is too close to the contour of the breast, (b) nipple shadow is too prominent, (c) breast contour is too irregular. (d)–(f) Correctly acquired images of the same breasts as in (a)–(c), respectively. For each example, original transversal view (top) as well as reconstructed sagittal (left) and coronal views (main) are shown.

2.3.

Automated Image Quality Assessment

In this work, we propose an automated image quality assessment system checking the images during or directly after the acquisition. The current standard and the proposed additional workflow step are indicated in Fig. 2. The early automatic detection of image quality issues will initiate a repeated acquisition if indicated. This will only take a few minutes. If a problem that disturbs the diagnosis was only detected later by the radiologist, the woman would have to be recalled, which would take several days. In order to build a convenient application for clinical practice, we first gathered expert definitions of artifacts and had real image data annotated by clinicians. Approved image processing algorithms were employed to extract features characteristic of distinct quality aspects. Feature design was based on a training dataset (dataset A, introduced below) and aimed at translating the physical properties of the artifacts into computable values taking into account the radiologists’ descriptions. Classifiers were used to reproduce the manual annotations based on the most meaningful subset of available features. In order to be used in clinical environments and produce results before the patient leaves the facility, the algorithms had to have a low run time (a few minutes at most). Similarly, in order not to produce unnecessary inconveniences for patients, the false positive rate was sought to be very low (clearly below 10%).

Fig. 2

Workflow diagram for general ABUS screening application. Automated image quality assessment (gray box) could be performed directly after the data were sent to the PACS system.

3.1.

Relative Nipple Position

The position of the nipple relative to the rest of the breast in the image is important because it relates to acoustic shadows that hamper the clinical interpretation of the image. The absolute nipple position in the image was given by the technician during image acquisition and stored as a private DICOM tag as specified by the standard acquisition protocol. The ABUS images were prepared for feature extraction in several preprocessing steps. First, a two-dimensional (2-D) coronal breast mask was computed similarly to the approach proposed by Wang et al.¹⁶ Therefore, a coronal mean projection of a stack of 120 slices close to the skin was performed. However, the top 50 slices from the skin were excluded from the breast mask computation to avoid responses from skin tissue. The projection image was smoothed using a Gaussian filter with a sigma of 0.2 mm and binarized by applying Otsu’s thresholding method.¹⁷ In order to close holes within the breast mask or along its edges, the binary image was dilated and holes were filled before it was eroded again. Finally, the breast contour line was computed in 2-D based on the mask image, as shown in Fig. 3. Note that the nipple coordinates $x_{\mathrm{T}}$ and $y_{\mathrm{T}}$ are generally assumed to be the same for all coronal slices, and the $z$-coordinate of the nipple is always on top of the image, since there is direct contact between transducer and nipple. The breast contour line is the same in all slices due to the compression of the breast and the properties of ultrasound. Using this contour and the given nipple position, nine features were extracted.

Fig. 3

Steps toward nipple position classification: (a) smoothed coronal projection, (b) binary mask after Otsu threshold, (c) dilated mask, (d) closed mask, (e) holes are filled, (f) eroded and contoured mask with a marker at the nipple position (set by technician during image acquisition).

3.2.

Nipple Shadow

In order to estimate the size of a possible nipple shadow, it was assumed that the shape of the shadow could be approximated by a cylinder around the nipple with the axis going in the anteroposterior direction. As the nipple is (approximately) a disk in the coronal plane, once it has stopped the US wave, it produces a cylindrical acoustic shadow. The nipple position $(x_{\mathrm{T}},y_{\mathrm{T}})$ was obtained from the DICOM header as given by the technician during acquisition. The size of the dark cylindrical region around the nipple position was estimated by counting cylinder segments (rings) that had low pixel intensity. The radius of the different cylinder segments varied from 4.0 to 20.0 mm in steps of 4 mm (see Fig. 4). In the anteroposterior direction, the height of each cylinder segment was $\sim 2.0\mathrm{mm}$. The highest layer was positioned starting at 6 mm below the skin, avoiding potentially disturbing high-intensity signals due to skin fat or sound reflections within the coupling layers of the transducer. The deepest layer ended at 26.0 mm below the skin. It was empirically determined that these measures were useful to describe the extent of the nipple shadow. The following seven features were extracted:

Fig. 4

Arrangement of cylinder segments that were used to estimate the size of a nipple shadow. The symmetry axis was at the nipple position. Three out of ten used layers and three out of five used rings (radii) are shown.

3.3.

Breast Contour Shape

In order to extract the breast mask and its contour line, several preprocessing steps were performed. They were similar to those described in Sec. 3.1 but with a focus on the breast contour line. A 4.0-mm stack of coronal slices starting at a distance of 7.0 mm from the skin was used for breast mask generation. The top 7.0 mm of coronal slices were excluded since they often contain spurious signals caused by sound reflections within contact fluid on parts of the transducer that do not have skin contact. Coronal slices lying deeper than 11.0 mm were not included in order to avoid signals from the ribs that can already appear from this depth on, depending on the breast size and the transducer positioning. A total of 17 features were extracted:

3.4.

Learning Step

We first evaluated each of the three image quality aspects individually and afterward merged all above described features in order to detect images of generally insufficient quality, i.e., a fourth classifier was trained. This joint classification approach was motivated by the fact that a large portion of the positive images was affected by more than one artifact. The manual annotation of two experienced clinicians served as ground truth for classifier training.

Classification tasks were performed on dataset A (introduced below) using the random forests classifier,¹⁸ as provided by the OpenCV library (version 2.4.10).¹⁹ While the number of trees was set to 100, the number of considered random features for decision tree construction was determined internally by the classifier, as proposed in Ref. 18 as ${\log }_2(M)+1$, where $M$ is the number of given features. The maximum depth of each tree was set to 15, and the minimum sample count required at each node to be split was set to 10% of the total number of samples. Ten repetitions of 10-fold stratified cross-validation (CV) were conducted to evaluate the performance of the classification. For each repetition, the instances were randomly partitioned into 10 folds under the constraint that images of the same patient were within one fold to avoid bias.

The resulting receiver-operating characteristic (ROC) curves were fitted by a binormal function, as implemented in MATLAB and Statistics Toolbox (Release 2011a, The MathWorks, Inc., Natick, Massachusetts, United States). First, for each repetition of CV, the merged ROC curve of all 10 folds was computed by sorting all instances into one curve. These were used to determine the mean ROC curve and the 95% confidence interval (CI) of all 10 repetitions. The area under the ROC curve (AUC) was estimated from the fitted curves, whereas single values of sensitivity and specificity were retrieved from the original (unfitted) classifier outputs. To compare the performance of the joint approach to that of the single classifiers, the significance of the difference between the corresponding AUCs was computed as a $p$-value using the method described in Ref. 20, as well as the Bonferroni correction²¹ to account for multiple (three) comparisons. This means that the computed $p$-values were multiplied by 3 and then compared to a confidence level of $\alpha =0.05$. The number of actually positive and negative instances was used to compare two ROC curves.

3.5.

Additional Performance Tests

In order to investigate the robustness and potential overfitting of the trained classifiers, an independent test dataset (called B) was employed. These data were acquired in a different clinic and manually annotated by a different reader group than dataset A. After training the four classifiers on the complete dataset A, they were applied to dataset B and compared to the manual rating results. Classifier decision thresholds were chosen such that the specificity in the training step was 97%. To obtain statistics, the data were bootstrapped 100 times. The data were also used to examine the difference between the joint classification, which is based on all features at once, and the straightforward combination of the three single classifier outputs to a combined rating. Ground truth for this comparison was the combination of the manual annotations, i.e., if at least one artifact was detected concordantly by both readers, the case was considered positive. As an additional performance measure, the inter-rater agreement between the two readers (R2 versus R3) as well as between the automated image quality assessment and the manual rating (AQUA versus R2&R3) was computed as Cohen’s $\kappa$.²²

4.1.

Dataset

In total, 815 ABUS volumes acquired from 114 women were obtained in routine clinical care and split up into two datasets, A and B. The images were acquired using either the Somo-v automated 3-D breast ultrasound system (U-systems, Sunnyvale, California) or the ACUSON s2000 ABVS (Siemens, Erlangen, Germany). Details on the size and spatial resolution of the images are given in Table 1. According to the acquisition protocol, the nipple position (in coronal view) was indicated manually by the technicians after each measurement and stored in the DICOM header of the corresponding file so that it could easily be used for further image processing. In some images, the nipple is not visible at all. These cases were excluded from the analysis of the relative nipple position and the nipple shadow. Detailed description of the datasets is given in Table 2. The Institutional Review Board waived the need for informed consent and approved the use of anonymized images for this study.

Table 1

Size and resolution of the ABUS volumes used in this study.

Table 2

Training and test dataset properties.

All images were classified separately by two clinicians with several years of experience in ABUS imaging. Dataset A was annotated by Readers 1 and 2, whereas dataset B was annotated by Readers 2 and 3; i.e., one reader was the same and one was different for the two datasets. Among others, the above-mentioned quality aspects—nipple position, nipple shadow, and breast contour shape—were taken into account during manual classification.¹⁴ The detailed rating results for dataset A are shown in Fig. 5. The distribution of artifacts was similar in dataset B. Considerable inter-rater disagreement has already been observed in another study dealing with quality rating of ultrasound images.²³ It renders classifier training difficult, but excluding the unclear, i.e., differently rated, cases from the study would mean excluding the critical cases and might bias the results. As the focus of the proposed application was put on a high specificity, we decided to consider only those cases “positive” that were rated as such by both readers. For the joint rating, an image was considered positive if at least one artifact was detected concordantly by both readers. All other cases were considered “negative” and hence usable for diagnostics. Throughout this report, “positive” and “negative” only refer to the rating of the image quality and are not correlated to any diagnostic findings, i.e., tumors or lesions.

Fig. 5

(a) Incidence of the three considered artifacts in the training dataset. “Positive” and “negative” describe the cases with and without artifacts. (b) Distribution and overlap of artifacts in all positive cases.

4.2.

Relative Nipple Position

Repeated cross-validation yielded an AUC of 0.99 [see Fig. 6(a)]. Different operating points on the ROC curve can be chosen for the final application by varying the decision threshold for the classifier. Depending on the intended purpose, the user may give more weight to specificity or sensitivity. As summarized in Table 3, at a specificity of 0.99, the sensitivity was 0.36. The point closest to the upper left corner of the plot (“best operating point”) represents a specificity of $0.905\pm 0.009$ ($\text{mean}\pm 95\%$ confidence interval) and a sensitivity of $0.93\pm 0.01$. For comparison, the performance of each reader when compared to the other, respectively, is displayed in the plots. It can be seen that the automatic classification performed very similarly to the readers. This is a general trend that also accounts for the other considered quality aspects. In Fig. 7, extreme outlier cases are shown. A false positive case is shown in Fig. 7(a), where the breast is very large and not completely visible in the image. In this case, the breast mask fails to describe the true contour of the breast. The breast in Fig. 7(b) is small and skinny, which impedes proper ultrasound coupling. As a consequence, a bright rectangle caused by reflections is visible in the upper right corner of the image, and breast mask segmentation using the Otsu filter fails. Figure 7(c) shows a false negative case caused by the irregular breast contour shape of the breast, which in turn produces an erroneous breast mask. The average computing time for all nine features was $3\mathrm{s}\pm 2\mathrm{s}$ per volumetric image, whereas the computing time for the classification was in the order of milliseconds (also for the other quality aspects) and thus negligible.

Fig. 6

ROC curve plots (a) for the relative nipple position, (b) for the nipple shadow, (c) for the breast contour shape, and (d) for the joint approach.

Table 3

Summary of classification results. AUC is the mean of 10 repetitions of 10-fold stratified cross-validation. The confidence intervals (CI) are computed from the 10 repetitions as well. p-values describe the Bonferroni-corrected (multiplied by 3) significance of difference between the AUC of the single classifiers when compared to the joint approach.

Fig. 7

Examples for outliers of the nipple position classification. (a) A false positive case, where a significant part of the breast is not visible on the scan. (b) A false negative due to an erroneous breast mask caused by intense reflections within the coupling layers of the transducer. (c) A false negative case caused by the irregular breast contour shape.

4.3.

Nipple Shadow

Automatic classification yielded an AUC of 0.84 [see Fig. 6(b)]. At a specificity of over 0.99, sensitivity was 0.24. The best operating point was described by a specificity of $0.82\pm 0.02$ and a sensitivity of $0.73\pm 0.02$. Figure 8 shows three sample outlier cases. The false positive case in Fig. 8(a) is a small and skinny breast with a clearly visible nipple shadow close to the breast contour line. However, it was rated as negative by the readers since it is hardly possible to get better images of such a small breast in the present view and a repeated scan probably would not enhance the image. Figure 8(b) shows a false negative case, where the dark region is not directly below the nipple but rather in a half ring around it. In Fig. 8(c), the false negative classification was caused by a relatively bright and fuzzy shadow. However, the algorithm was designed to detect very prominent, low-intensity nipple shadows, as shown in Figs. 3(a) and 3(b). On average, it took $5\mathrm{s}\pm 2\mathrm{s}$ per ABUS image to compute all possibly relevant features.

Fig. 8

Incorrectly classified cases of the nipple shadow. (a) A false positive case caused by the nipple being very close to the breast contour line. (b) A false negative with a structured, ring-like nipple shadow. (c) A false negative case with fuzzy and bright nipple shadow.

4.4.

Breast Contour Shape

The classification of irregular breast contour shapes achieved an AUC of 0.89 [see Fig. 6(c)]. At a specificity of 0.99, the sensitivity was 0.15. At the best operating point, specificity was $0.82\pm 0.04$ and sensitivity was $0.79\pm 0.04$. Figure 9(a) shows a sample false positive case. The breast as such is imaged correctly, but parts of the axilla and the arm cause atypical contour lines, which are misinterpreted by the classifier. Figures 9(b) and 9(c) show false negative cases, where parts of the breast are not imaged correctly. Nevertheless, the breast mask has smooth contours, obscuring missing parts and misleading the classifier. The average computing time was $6\mathrm{s}\pm 4\mathrm{s}$.

Fig. 9

Examples for outliers of the breast contour classification. (a) A false positive case where parts of the axilla and the arm are visible on the image. The false negative cases in (b) and (c) show relatively smooth contours, obscuring the fact that parts of the breast are not imaged correctly.

4.5.

Joint Classification

In this approach, the manual ratings for the three distinct artifacts were combined to a joint quality measure. If, according to both readers, at least one artifact was present, an image was assigned the positive class. Random forest classification was based on all computed features at once and achieved an AUC of 0.94 [see Fig. 6(d)]. At a specificity of 0.99, the sensitivity was 0.55. The best operating point had a specificity of $0.91\pm 0.01$ and sensitivity of $0.81\pm 0.01$. According to the Bonferroni-corrected (multiplied by 3) $p$-values of Table 3, the AUC of the joint approach was significantly ($p<0.05$) smaller than that of the nipple position classification and significantly larger than that of the nipple shadow classification. The difference between the AUCs of the joint approach and the breast contour shape classification was not significant.

4.6.

Additional Performance Tests

Applying classifiers trained on dataset A to the independent test dataset B resulted in the values shown in Table 4. Here we also show the results of a simple combination of classifier outputs (named “combination” in the table) compared to using all features in one single classifier (named “joint”). The “combination” approach combined the outputs of the three distinct classifiers (nipple position, nipple shadow, breast contour) into one rating in the same way as the manual ground truth annotation was combined for the global quality rating: If at least one of the three artifacts was detected by the distinct classifier, the image was rated positive.

Table 4

Rating results retrieved from test dataset. “Combination” refers to the simple combination of the three single rating results, whereas “joint” describes the classifier that was trained on all features at once. R2 and R3 refer to Readers 2 and 3. κ refers to Cohen’s inter-rater agreement. Given values are mean (stdev) from 100 times bootstrapping.