2.1.

Common DWI Datasets

Three groups of DWI datasets were analyzed in the ADC-CP: two $b$-value in vivo breast scans (Br2b), four $b$-value in vivo breast scans (Br4b), and four $b$-value phantom scans (Ph4b). Analysis metrics and MRI diffusion protocol details for all data are summarized in Table 1. All in vivo datasets were from the IRB approved American College of Radiology Imaging Network (ACRIN) 6698 trial⁷ and were used with the permission of ACRIN. In vivo image files were deidentified as per the requirements of the Health Insurance Portability and Accountability Act [Digital Imaging and Communication in Medicine (DICOM) standard, supplement 142], while preserving private metadata attributes necessary for DWI processing. DICOM images were curated and shared via the Cancer Imaging Archive.⁸ Each protocol group included scans from three MRI scanner manufacturers: Siemens Medical (SM), Philips Medical (PM), and General Electric Healthcare (GEHC). In vivo scans were multislice axial acquisitions with full biaxial breast coverage using standard two-dimensional (2-D) single-shot echo-planar imaging sequences. Group Br2b consisted of three studies: ID101 (GEHC, Signa HDxt, 3.0 T), ID102 (PM, Intera, 3.0 T), and ID103 (SM, Avanto, 1.5 T). Group Br4b consisted of four studies: ID201 (GEHC, Signa HDxt, 3.0 T), ID203 (GEHC, Signa HDxt, 1.5 T), ID205 (PM, Achieva, 1.5 T), and ID207 (SM, Avanto, 1.5 T). For all in vivo scans, a single $b=0$ image was acquired and non-0 $b$-value images were acquired with three orthogonal diffusion encoding directions. For all cases except ID203, standard on-scanner processing was used, resulting in trace images for each non-0 $b$-value and online generated ADC maps, and only the trace images were available for analysis. For ID203, the full set of directional DWI images was preserved, and no trace images or online ADC map were calculated.

Table 1

Dataset information for the ADC Mapping CP.

The Ph4b datasets were of a diffusion phantom designed and constructed by the National Institute of Standards and Technology (NIST) and High Precision Devices (HPD Inc., Boulder, Colorado).^9,10 This phantom consisted of an array of 13 20-mL vials in a spherical vessel filled with an ice–water mixture to maintain a controlled temperature of 0°C. Three vials were filled with water and ten vials were filled with solutions of the polymer polyvinylpyrrolidone (PVP) in deionized water,¹¹ with two vials each at PVP mass fractions of 10%, 20%, 30%, 40%, and 50%. ADC values ranged from $\sim 1.1$ to $0.12\times {10}^{-3}{\mathrm{mm}}^2/\mathrm{s}$. Scans were multislice coronal acquisitions at 3.0 T, using standard 2-D single-shot echo-planar imaging sequences. Diffusion encoding was applied on three orthogonal axes, with reconstruction of standard trace images at each $b$-value. Only the trace images were provided for analysis. Three datasets were provided: ID401 (GEHC, Discovery MR750, Memorial Sloan-Kettering Cancer Center, New York, New York), ID402 (SM, Trio, University of Colorado, Boulder, Colorado), and ID403 (PM, Ingenia, University of Michigan, Ann Arbor, Michigan). All phantom images used in this study were obtained by the DWI task force of the Quantitative Imagining Biomarker Alliance (QIBA) of the Radiological Society of North America (RSNA).

2.2.

ADC-CP Participants

Participants in the ADC Mapping CP included 11 QIN sites and one non-QIN commercial group. A total of 15 DWI AIs were used (Table 2). Eight platforms were on-site developed private analysis packages programmed in MATLAB^® (The MathWorks Inc., Natick, Massachusetts; six implementations “AI-MAT1” to “AI-MAT6”), IDL^® (Exelis Visual Information Solutions, Boulder, Colorado; “AI-IDL”), or C++ (“AI-C++”). Five implementations utilized free, publicly available analysis packages: 3D Slicer DWModeling module of the SlicerProstate extension¹⁹ (Brigham and Womens Hospital; two implementations “AI-3DSl1” and “AI-3DSl2”), AFNI (University of California, San Diego, Analysis of Functional Neuro Images; “AI-AFNI”), OsiriX ADCMap plugin (Stanford; “AI-OsX1”), and QIBAPhan (RSNA/University of Michigan; “AI-QIBA”). Two implementations were commercially available analysis packages: Aegis™ (Hologic Inc., Sunnyvale, California; “AI-Aegis”) and OsiriX plugin IB Diffusion™ (Imaging Biometrics, Elm Grove, Wisconsin; “AI-OsX2”). Source websites for the publicly available software packages are included in the references in Table 2.

Table 2

DWI quantitative AIs included in the ADC Mapping CP.

2.3.

ADC-CP Parametric Maps

For the purpose of the ADC-CP, the basic monoexponential decay model for the MRI signal intensity from an isotropic tissue region was assumed

Site analysis consisted of generating a set of parametric maps from pixel-by-pixel analysis of each DWI dataset. Analyses performed for each data group are listed in Table 1. For all cases, a monoexponential ADC map utilizing all images was computed: ${\mathrm{ADC}}_2$ for Br2b, and ${\mathrm{ADC}}_4$ for Br4b and Ph4b groups. In addition, for the Br4b group, a perfusion minimized analysis was performed.²⁰ For this analysis, the three nonzero $b$-values were used to estimate the “slow” or tissue diffusion signal using Eq. (1) for $b\ge 100\mathrm{s}/{\mathrm{mm}}^2$, giving $S_{0\text{slow}}$ and ${\mathrm{ADC}}_{\text{slow}}$ as the fitted parameters characterizing the slow signal decay. The fraction of the signal attributable to a fast-decaying perfusion component was then calculated as

In addition to the parametric maps provided by the analysis sites, scanner manufacturers’ software (“online”) ADC maps were evaluated when they were provided with the original DWI data. This included the ${\mathrm{ADC}}_2$ for the Br2b group, ${\mathrm{ADC}}_4$ for the Br4b datasets with trace images (three of four studies), and ${\mathrm{ADC}}_4$ for the Ph4b group.

2.4.

Centralized ROI Analysis

All parametric maps were submitted to UCSF through a secure box system. No restrictions were placed on the choice of file format, and formats included DICOM ($N=7$), Neuroimaging Informatics Technology Initiative (NIfTI; $N=2$),²¹ Nearly Raw Raster Data (NRRD; $N=1$),²² Analyze (Mayo Clinic; $N=1$),²³ and MATLAB^® ($N=4$). Prior to concordance analysis, all maps were converted to a UCSF in-house modified multiframe DICOM format allowing integer or floating point data, along with storage of an analysis mask. Slice order was detected automatically for file formats that do not include orientation information and was reversed if necessary to match the slice order of the source images. ADC scaling was detected automatically by comparison with a reference UCSF ADC map, and scaling factors were set in the metadata (DICOM rescale slope attribute) to produce ADC maps in common units of ${10}^{-6}{\mathrm{mm}}^2/\mathrm{s}$. No manipulation of the actual map pixel data was done except for floating point formats (MATLAB^® implementations) in which pixels with a “not-a-number” value were reassigned to 0.0 and masked out for analysis.

ROI analysis was performed using standardized ROIs across all parametric maps (Fig. 1). For the in vivo breast cancer scans, a multislice, whole-tumor region defined for use in the primary study was used. For the phantom scans, ROIs were defined on the middle slice of each scan using 1-cm-diameter circular regions on each of the 13 sample vials. ROIs were applied to the parametric maps yielding mean values of the diffusion metrics for each analysis platform. All centralized analysis was done using software developed by the UCSF lab in IDL^®.

Fig. 1

Typical ROI placements for (a) breast studies and (b) phantom studies, shown on ${\mathrm{ADC}}_4$ maps. (a) A single representative slice from a multislice breast tumor ROI. The ROIs were drawn referencing the high $b$-value DWI and an accompanying DCE subtraction image. Calculated mean ADC was taken over the full multislice ROI. Phantom ROIs shown in (b) are single-slice, 1-cm-diameter circles labeled with the PVP concentration (0% to 50%) and a position subscript: $\mathrm{C}=\text{center}$, $\mathrm{I}=\text{inner}$, and $\mathrm{O}=\text{outer}$.

2.5.

Statistical Analysis

For each metric, pairwise within-subject coefficient of variation (wCV) was calculated between all implementation pairs to establish groups of implementations with similar results (intragroup $\mathrm{wCV}<0.1\%$ between all AI pairs). As no ground truth values could be established for the in vivo assessed DWI metrics, individual implementation concordance could only be evaluated from the percent difference of each ROI measurement from a consensus reference value for that measurement. This method was also used for the phantom scans even though reference ADC values were available, both for consistency of presentation and to avoid complications from scanner- and position-dependent ADC effects. A full analysis of the phantom ADC data relative to the ground truth reference values is presented by Malyarenko et al.²⁴ Reference value calculation for each of the metrics is described in Sec. 3. The two-tailed student’s $T$-test was used to test for significant differences among different implementations.

3.1.

Practicalities

From the 12 participating institutions, monoexponential ADC maps for the Br2b and Br4b groups and perfusion minimized ${\mathrm{ADC}}_{\text{slow}}$ values for Br4b were provided for 13 analysis platforms. Nine platforms from eight institutions also provided perfusion-fraction maps for the Br4b group. The Ph4b data group was analyzed on 11 platforms, 10 generating both ${\mathrm{ADC}}_4$ and ${\mathrm{ADC}}_{\text{hi-low}}$ parametric maps while one provided only ${\mathrm{ADC}}_4$. All sites were able to process DICOM image sets from all three vendors, but interpretation of the no trace, full directional data (Br4b, ID203) was challenging for several sites due to unfamiliarity with this format. After specification of the image storage order for this case, all sites were able to program their implementations to process this data, though in some cases we noted discrepancies in the results as shown in Sec. 3.2.

3.2.

Breast Scans

For the Br2b ${\mathrm{ADC}}_2$ metric, a majority of the AI (11 of 13) gave essentially identical results (maximum $\mathrm{wCV}<0.003\%$). For each dataset, the median ADC value from all offline results was used for the reference value for concordance. Figure 2 shows the percent difference from these reference values for each AI’s mean ROI ${\mathrm{ADC}}_2$ measure for each of the three Br2b scans. AI-MAT3 had a consistent 0.12% positive bias relative to the median, while AI-Aegis varied from $-0.04\%$ to $-0.06\%$. The GEHC and PM online maps were within 0.05% of the respective median values, but the SM map had a $-1.4\%$ bias.

Fig. 2

Concordance of two $b$-value in vivo ADC measurements across 13 offline AIs and online scanner-generated maps. Plotted is the percent difference for each ROI mean value from the median value for that measurement for all offline AI. Eleven offline AI had essentially identical results ($\mathrm{wCV}<0.003\%$) and thus show no offsets on the plot. The SM online ADC had a $-1.4\%$ bias relative to the consensus median value.

More variations were observed among platforms in the Br4b analyses. Figure 3(a) shows graphically the pattern of agreement among platforms given by the pairwise wCV measures. A majority of implementations (9 of 13) fell into two groups when using a threshold of $\mathrm{wCV}<0.1\%$ among all group members. Group A consisted of three AI (AI-IDL, AI-3DSl1, and AI-3DSl2) with $\mathrm{wCV}<0.01\%$, while group B consisted of six AI (AI-MAT2, AI-MAT3, AI-MAT5, AI-OsX1, AI-C++, and AI-Aegis) with $\mathrm{wCV}<0.1\%$. For each dataset, a reference value was calculated as the average of the mean value for group A and the mean value for group B. Figure 3(b) shows the percent difference from these reference values for ${\mathrm{ADC}}_4$ from each implementation for the Br4b datasets. ${\mathrm{ADC}}_4$ values differed significantly between groups A and B [$2.8\%\pm 0.2\%$ ($\text{mean}\pm \mathrm{SD}$), $p<0.003$], and up to 5% between nongrouped sites. Two of the four nongrouped implementations, AI-MAT4 and AI-MAT6, had only small variations ($\mathrm{wCV}<0.13\%$) from the group B values, while AI-MAT1 and AI-OsX2 showed more variability both between scans from different vendors and from the reference values. Two implementations, AI-MAT1 and AI-MAT4, had slightly anomalous results for ID203 (GEHC), believed to be due to different handling of the full directional diffusion data. Scanner-generated ${\mathrm{ADC}}_4$ maps were available for the three datasets with trace images. GEHC and SM maps gave mean ROI ADC values of $+3.6\%$ and $-3.3\%$, respectively, from the reference values, while the PM online map had a 28% offset. Further investigation revealed that this large deviation was due to loss of the DICOM rescale slope data employed by PM for parametric map intensity scaling. This loss appeared to have occurred during data transfer between the scanner and the imaging site’s PACS system.

Fig. 3

In vivo four $b$-value ${\mathrm{ADC}}_4$ ROI analysis results. (a) Pairwise wCV matrix for all implementations, shown graphically from $\mathrm{wCV}<0.1\%$ (white circles) to $\mathrm{wCV}>2\%$ (fully black circles), with groups A and B indicated. (b) Percent difference from the consensus ${\mathrm{ADC}}_4$ values for each of four datasets, for each implementation and online map. Mean difference in ADC between groups A and B was 2.8%. The 28% deviation on the PM online ADC was due to a DICOM header corruption problem.

Results for the perfusion minimized analysis tissue ADC (${\mathrm{ADC}}_{\text{slow}}$) were similar to the ${\mathrm{ADC}}_4$ results [Figs. 4(a) and 4(b)]. For $\mathrm{wCV}<0.1\%$ grouping, AI-IDL switched from group A to B, and AI-MAT6 was also now included in group B. Overall differences were generally smaller than for ${\mathrm{ADC}}_4$ but still statistically significant: $1.2\%\pm 0.2\%$ ($\text{mean}\pm \mathrm{SD}$, $p<0.003$) difference between groups A and B and maximum individual differences of any implementation $<\pm 1.3\%$ relative to the reference value. Perfusion fraction ($P_f$) was a nonstandard metric and was implemented on nine platforms. Two groups were again evident, though with different membership [Fig. 4(c)]: group A ($\mathrm{wCV}=0.04\%$) composed of AI-IDL and AI-MAT2 and group B ($\mathrm{wCV}<0.01\%$) with MATLAB^® implementations AI-MAT3, AI-MAT5, and AI-MAT6, with a small difference among the groups [$0.29\%\pm 0.10\%$ ($\text{mean}\pm \mathrm{SD}$), $p<0.03$]. Figure 4(d) shows the concordance for the $P_f$ metric results. $P_f$ results from AI-MAT1 showed large deviations ($-16\%$ to $-23\%$) from the consensus reference, indicating possible errors in the software implementation that was developed on-site for this CP. AI-C++ had a positive bias of 1.5% to 2.5%, which was found to be due to implementation of a biexponential decay model for this calculation. All other measures fell within $\pm 0.25\%$ of the reference values, except for the AI-MAT4 result for the GEHC directional diffusion dataset with a $-0.9\%$ deviation. No online parametric maps were available for the perfusion minimized analysis.

Fig. 4

wCV and ROI mean concordance results for the Br4b data group perfusion minimized analysis. For ${\mathrm{ADC}}_{\text{slow}}$, (a) shows the pairwise wCV matrix with groups with $\mathrm{wCV}<0.1\%$ indicated and (b) the corresponding data for differences in mean ROI ${\mathrm{ADC}}_{\text{slow}}$. Group results showed smaller variations than for ${\mathrm{ADC}}_4$. For $P_f$, (c, d) groups were less well defined, except for the three MATLAB^® AI indicated, which were nearly identical ($\mathrm{wCV}<0.01\%$). The small positive biases for AI-C++ were identified as due to use of a biexponential model.

3.3.

Phantom Scans

Analyses of the three Ph4b phantom datasets, ID401 (GEHC), ID402 (SM), and ID403 (PM), were submitted from 11 AI for the four $b$-value ${\mathrm{ADC}}_4$ metric and 10 AI for the two $b$-value ${\mathrm{ADC}}_{\text{hi-low}}$. For AI-C++, only the ID402 results were included, as a problem in the DICOM encoded ADC maps for ID401 and ID403 resulted in incorrect ROI ADC values in the centralized analysis. In a separate analysis completed after the encoding bug was fixed, these results were in concordance with the other implementations.²⁴ Online maps for ${\mathrm{ADC}}_4$ were available for all three phantom datasets, but only ID403 (PM) included an online map for ${\mathrm{ADC}}_{\text{hi-low}}$. Results for the two $b$-value ${\mathrm{ADC}}_{\text{hi-low}}$ were practically identical across all implementations. The maximum pairwise wCV among postprocessing implementations using all 39 ROI measurements from the three datasets was 0.04%. Looking at the percent difference of each ROI measure from the nine site median values, AI-QIBA showed a similar clinically insignificant bias (0.05%) to that seen in the Br2b datasets for AI-MAT2. The results from the online PM ${\mathrm{ADC}}_{\text{hi-low}}$ map were very close to the offline reference median values, yet showed a consistent but clinically insignificant bias of $0.31\pm {0.02\times 10}^{-6}{\mathrm{mm}}^2/\mathrm{s}$ across all ADC values. This gave a maximum percent difference of $-0.25\%$ for the lowest ADC sample.

For the ${\mathrm{ADC}}_4$ measures, paired wCV measurements over all phantom measurements gave similar groups to the Br4b results. Differences within and between the two postprocessing implementation groups were smaller than for the breast scans. The maximum wCV was 0.04% for group A (AI-IDL, AI-Sl1, AI-Sl2, and AI-MAT6) and 0.01% for group B (AI-MAT2, Al-QIBA, Al-OsX1, and Al-MAT5), and the between-group root mean square percent difference in ADC values for all 13 ROIs was 0.29%, 0.30%, and 0.62% for GEHC, SM, and PM scans, respectively. There was no significant bias among the ROI mean ADC values from the two groups ($p=0.15$, 0.07, and 0.19 for GEHC, SM, and PM scans, respectively). Figure 5 shows the differences from reference ${\mathrm{ADC}}_4$ (average of the mean group A and mean group B results) for the three Ph4b datasets. While differences are in general very small ($<0.5\%$), individual excursions were as high as 5.5%, with the highest differences on the lowest 2 ADC values ($\mathrm{ADC}<0.25\times {10}^{-3}{\mathrm{mm}}^2/\mathrm{s}$). Only the SM online map showed a statistically significant deviation from the reference values, with a small negative bias of $-0.31\%\pm 0.25$ ($\text{mean}\pm \mathrm{SD}$, $p=0.001$). Only the PM dataset analysis showed a trend with ADC value in the difference among the analysis groups, with group A tending to underestimate ADC relative to group B for higher ADC values and over estimate at lower. A linear regression of the percent difference between the groups versus the mean ROI ADC gave a slope of 1% per $1.0\times {10}^{-3}{\mathrm{mm}}^2/\mathrm{s}$ with $R^2=0.35$.

Fig. 5

Percent difference from reference ${\mathrm{ADC}}_4$ for Ph4b measurements for (a) GEHC, (b) SM, and (c) PM scans. The reference value for ${\mathrm{ADC}}_4$ for each individual ROI is the average of the groups A and B mean ${\mathrm{ADC}}_4$ values for that ROI. ROIs are ordered from highest ADC (0% PVP) to lowest ADC (50% PVP), left to right, for each AI or group. Concordance is excellent, except for a few measurements on the lowest ADC vials ($\mathrm{ADC}<0.25\times {10}^{-3}{\mathrm{mm}}^2/\mathrm{s}$).