2.1.

Phantom DWI DICOM

A quantitative DWI phantom developed by the National Institute of Standards and Technology (NIST)²⁸ provided a linear range of one-order-of-magnitude in ADC ($\pm 0.005$): $(0.125,0.24,0.40,0.60,0.83,1.10)\times {10}^{-3}{\mathrm{mm}}^2/\mathrm{s}$ (at 0°C, “ground-truth” values based on multisite studies).^29,30 This allowed simultaneous evaluation of fit accuracy over a broad range of DWI intensities in a single image. Briefly, the phantom contained two vials for each of five polyvinylpyrrolidone (PVP) water solutions (10% to 50%) and three vials with pure water (for a total of 13 vials) arranged hexagonally within the spherical phantom shell. For temperature control, the shell was filled with an ice-water bath. During scanning, the central water vial was positioned close to the magnet isocenter, and other vials were radially offset up to 55 mm from the central vial. The phantom DWI data were acquired at 0°C on Siemens Trio (S1), GEMS Discovery MR750 (S2), and Philips Ingenia (S3) 3T scanners for five coronal slices using $b\text{-}\text{values}=0$, 500, 900, and $2000\mathrm{s}/{\mathrm{mm}}^2$ and following a common scan protocol.¹⁸

Trace-DWI DICOM for these multivendor phantom scans was shared with the QIN MRI subgroup by Quantitative Imaging Biomarker Alliance (QIBA) DWI task force members. Full scanner-generated headers were provided in the single-frame format (DICOM MR Storage object) without deidentification. “S3” DWI DICOM was supplied in multiframe (DICOM enhanced MR storage object) and single-frame format. “S1” and “S2” scanners were not capable of generating multiframe DWI DICOM.²¹ Vendor-specific DICOM conformance documents were shared with the group through NCIP-HUB.³¹ Two vendors stored the diffusion $b$-value and associated directionality information in nonstandard attributes [S1: (0019, 100C/D) and S2: (0043, 1039/30)]. The information on vendor-specific image order and $b$-value storage attribute was requested by users of SW1 (S3), SW7 (S3), and SW8 (S1 and S3), while others were able to parse multivendor source DWI DICOM. Three sites (SW3, SW7, and SW10) analyzed multiframe DWI DICOM from the “S3” source.

2.2.

ADC Map DICOM

The data inventory for ADC maps submitted by sites is summarized in Table 1. Ten sites generated parametric ADC2 and eight sites provided ADC4 mono-exponential diffusion model fits. All sites performed ADC2 fit using a log-intensity ratio to high $b$-value. Different multi-$b$ fit algorithms reported by sites for ADC4 maps are listed in Table 1. SW1 and SW5 were fitting “unweighted” log-intensity DWI allowing extra fit parameter ($S_0$, DWI intensity at $b=0$). SW2 SW3 used a combination least-squares regression algorithm for log-intensities, which effectively “weights” signals by the average DWI intensity (that generally depends on the range of the applied $b$-values). SW2, SW4, SW6, SW7, and SW10 performed regression or least-squares fit for log-signal intensities that effectively “weights” DWI by signal-to-noise ratio (SNR). Five sites used third-party software (two on-scanners and three offline) to generate ADC maps, and five sites have utilized their in-house tools for offline analysis. According to the site reports, only one vendor SW8 was capable of cross-vendor DWI analysis off-scanner to generate ADC2 (ADC4 allowed, but not generated), and it required manual entry of $b$-value and DWI image order for S1 and S3 data. SW8 also produced interpolated ADC maps for S1 ($256\times 256$) and S3 ($512\times 512$) images. All other maps preserved source-image dimensionality. One vendor SW (SW10) provided both ADC2 and ADC4 maps for its DWI data source (S3).

Table 1

Details of ADC analysis software (SW) and algorithms used in the study.

For numerical consistency and header checks, ADC DICOM data provided by each of the sites were loaded into MATLAB^® using standard dicomread function. The image scaling recorded in the header [(0040, 9096) “RealWorldValueMapping” or (0028, 1053) “RescaleSlope”] was used to convert all maps to the common scale of ${10}^4$ integer intensity range (${10}^{-7}{\mathrm{mm}}^2/\mathrm{s}$) before cross-site analysis. SW2 provided incorrect scaling for S3 data, apparently carried over from the source DWI DICOM header, which was ignored. SW2, SW4, and SW7 implementations did not record scale information (tag missing or value set to 1), and apparent integer map scale was used instead (see Table 1). SW1 provided scale value with inverse interpretation, which was likewise corrected. No additional filtering or masking was performed for the site ADC maps.

2.3.

Numerical Consistency

The purpose of the numerical consistency analysis was (1) to determine a consensus ADC value range in relation to adequate scale and bit storage for the PM presentation and (2) to relate deviations between ADC4 and ADC2 fit algorithm results, with respect to the ground-truth values provided by the phantom, to the metadata required to describe observed discrepancies.

Apparent, slightly different placement of the phantoms in different scanners necessitated defining separate (source-specific) sets of regions-of-interest (ROIs) for statistical analysis of the ADC results. ADC ROI statistics were generated using the qiba_proc function of QIBAPhan software (Table 1). Briefly, a common set of 13 circular ROIs (1-cm diameter) was defined on PVP vials from a middle ($b=0$) slice image [Fig. 1(a)] for each data source (S1, S2, and S3) and automatically applied to the ADC maps [e.g., Figs. 1(b) and 1(c)] provided by participating sites to generate mean, median, standard deviation (SD), and ranges that were recorded in coma-separated-value tables along with the ROI coordinates. Location of the central tube ROIs varied among source data from 15 to 30 mm (from isocenter). The difference in locations of duplicate ${\mathrm{ADC}}_{\mathrm{PVP}}$ ROIs with respect to isocenter was from 25 to 50 mm. Consequently, up to 3% systematic variations were expected for mean ADC values between duplicate PVP ROIs of different DWI sources due to variability of source scanner gradient characteristics.³⁵

Fig. 1

(a) illustrates example screen-captures of a middle slice for T2-weighted image (with inscribed, numbered ROIs) for the PVP phantom (scanner 1 source, S1) and the corresponding (b) ADC2 and (c) ADC4 maps generated by “SW3” analysis. Two ROIs are defined for each $\%\mathrm{PVP}>0$ concentration and three for ice-water ($\%\mathrm{PVP}=0$). Common scale for the ADC2 and ADC4 maps is indicated by the color-bar between (b) and (c).

Numerical consistency of mean ADC2 versus ADC4 with respect to the ${\mathrm{ADC}}_{\mathrm{PVP}}$ reference was analyzed by scatter plots for 13 ROIs for all data sources within order of magnitude range provided by the PVP phantom. The coefficient of variance (CV) for ADC ROIs was calculated as ${\mathrm{SD}}_{\mathrm{ROI}}/{\mathrm{ADC}}_{\mathrm{PVP}}$. The Bland–Altman test was performed for the SW1 to SW7 data that provided results from both fit algorithms for all three DWI data sources. Deviation of mean ADC4 from ADC2 results supplied metrics for evaluation of the fidelity of the multi-$b$ fit algorithm. The source of deviations between ADC4 and ADC2 was tracked back to the specific fit algorithm and parameters (Table 1) to determine relevant descriptive metadata.

2.4.

DICOM Implementation Comparison to the PM Standard

Site ADC DICOM implementations were compared to the general PM²³ and DWI DICOM macro requirements.²¹ The required attributes and coded terms of the PM DICOM metadata were defined in the DICOM PM storage object.²³ These included image type, storage format, image scale and units, image geometry, and DWI source-image reference. Additional attributes important for numerical consistency analysis specific to the quantitative DWI PM, but not present in the original standard, included fit algorithm and parameters (i.e., the $b$-values used). The DICOM header evaluation was performed using the MATLAB^® structure query functions and a DICOM validator command-line application (dciodvfy).³⁶ To check compatibility with existing DICOM parsers/viewers, the ADC maps submitted by sites were loaded into a production-level viewer, iQ-View 2.8.0 (Image Information Systems, UK), for the clinical picture archiving and communicating system (PACS), as well as into five additional DICOM viewers widely utilized by the imaging research community [OsiriX 8.0.2 (Pixmeo, Switzerland), IDL 8.5 (Excelis, Boulder, Colorado), 3D Slicer 4.7.0 (NA-MIC, NIH), ImageJ 1.45s (NIH), and KPacs 1.6.0 (Image Information Systems, UK)]. Viewing of ADC DICOM for SW1, SW4, and SW6 (S2 and S3) required resetting window-level at full dynamic range.

The requirements regarding storage of the DWI model and fitting-related parameter deficiencies were used to develop a correction proposal³⁷ to improve the DICOM standard for quantitative analysis of DWI and to facilitate uniform implementation in the future by the QIN community through development of the open-source DICOM for Quantitative Imaging (dcmqi) library^25,38 of converter tools. This converter can be applied to the volumetric image data stored in any of the supported formats (e.g., MHD,³⁹ NifTi,⁴⁰ and Analyze⁴¹) and can construct a DICOM PM header by merging source DWI DICOM with optional metadata supplied by the user in an ASCII (JSON)⁴¹ file. The application details are discussed in the dcmqi user guide.⁴² The converter was tested by a central analysis lab (SW6) on Win64 platform. Briefly, an S2 ADC2 volumetric image dataset was converted to MHD (one of the allowed input formats) and an example JSON metadata file was edited to include units, $b$-value and model used. The converter was applied to volumetric MHD ADC, using the corresponding source DWI DICOM and metadata. The resulting DICOM PM object header was queried in MATLAB^® using dicominfo function and tested for the import to iQ-View, OsiriX, IDL, 3D Slicer (with and without the QuantitativeReporting extension), ImageJ, and KPacs.

3.1.

Numerical Consistency of ADC2 versus ADC4 across Site Implementations

Figure 2 shows high numerical consistency of mean ROI ADC values across sampled SWs. All measurements are tightly clustered around six values resolving discrete ADC contrast levels in Figs. 1(b) and 1(c), corresponding to different PVP concentrations. This confirms correct interpretation of the ADC map scaling information (Sec. 2.2). The clusters are slightly offset, depending on data source (color-coded), as ADC increases, consistent with the expected systematic differences between scanners gradient systems and ROI locations (see Sec. 2.3). A closer look at the lowest and highest mean ADC values (Fig. 2, panels a1 and a2) shows further clustering for each data source corresponding to two and three ROIs per PVP concentration, for low and high ADC, respectively. ROI cluster proximity to the true ${\mathrm{ADC}}_{\mathrm{PVP}}$ value (marked by magenta “+”) is evidently determined by ROI location and data source (within 2% for high and 8% for low ADC) and is nominally independent of the analysis SW. For specific ${\mathrm{ADC}}_{\mathrm{PVP}}$ value, the ADC ROI CV (data, not shown) depended mainly on the data source (independent of ROI location) and decreased with growing ${\mathrm{ADC}}_{\mathrm{PVP}}$ from 10% (a1) to 0.05% (a2), indicative of the contrast-to-noise ratio (CNR) limits for the low ${\mathrm{ADC}}_{\mathrm{PVP}}$.

Fig. 2

Panel (a) shows a scatter plot of ROI-mean ADC2 versus ADC4 values for all SWs and ROIs. Magenta “+” marks true ${\mathrm{ADC}}_{\mathrm{PVP}}$ values. Dotted diagonal line marks ADC2 = ADC4. Legends list symbol assignment for site software (SW) and color-code for data sources (S1, S2, and S3). The ROI clusters corresponding to the lowest ($0.125\times {10}^{-3}{\mathrm{mm}}^2/\mathrm{s}$) and the highest ($1.1\times {10}^{-3}{\mathrm{mm}}^2/\mathrm{s}$) ${\mathrm{ADC}}_{\mathrm{PVP}}$ values are enlarged in panels (a1) and (a2), respectively. Vertically aligned symbols correspond to ADC2 data submitted without ADC4 maps. Panel (b) shows Bland–Altman plot for ADC4 versus ADC2 generated by the seven sites that submitted all required DICOM for all data sources. Dotted horizontal lines mark 98th data percentile.

Panels a1 and a2 from Fig. 2 show enlargements of the highest and lowest ADC “clusters” illustrating that systematic mean ADC deviations among data sources (color-coded) and ROIs exceed the differences among analysis SW [symbol legend, Fig. 2(a)] along both the ADC2 (vertical) and ADC4 (horizontal) dimensions. For each individual ROI cluster, the spread is higher along ADC4 compared to ADC2, indicating better consistency of the ADC2 fit across sites. Note that for the on-scanner SW that provided only ADC2 maps (vertically aligned symbols), the deviation from the centers of the ROI clusters is usually higher than among the other offline tools (Table 1). This is likely related to additional signal interpolation and filtering applied by the scanner SW in contrast to offline processing.

The apparent increase in scatter along the ADC4 dimension for some ROI clusters (duplicate PVP concentrations in panels a1 and a2 from Fig. 2) is mainly due to the presence of three “outliers” (SW1, SW3, and SW5) that happen to use different multi-$b$ fit functions (signal “weighting”) compared to the other sites (Table 1, Sec. 2.2). For the sites that submitted both ADC2 and ADC4 maps, the differences between mean ADC4 and ADC2 for each SW are shown as a function of the average measured ADC [Fig. 2(b)]. When compared to the true phantom ${\mathrm{ADC}}_{\mathrm{PVP}}$, these differences can be used as a self-normalized measure of algorithm fidelity for the mono-exponential diffusion medium provided by the phantom. For the majority of data, the observed ADC4 versus ADC2 deviations are within $\pm 0.002\times {10}^{-3}{\mathrm{mm}}^2/\mathrm{s}$ and nearly independent of the ADC and ROI location, except for the above-mentioned three ADC4 fit algorithm “outliers.” The outlier deviations are higher (and exceed ROI CV) at higher ${\mathrm{ADC}}_{\mathrm{PVP}}$, suggesting that they are related to SNR bias for high $b$-values when all DWI signals (including $b=0$) are given equal weight in the fit (Table 1, Sec. 2.2). The largest relative deviations observed for SW1 and SW5 at high ${\mathrm{ADC}}_{\mathrm{PVP}}$ are likely related to the use of an extra (intercept) parameter by their corresponding ADC4 fit functions (Table 1, Sec. 2.2) compared to the slope-only calculation by ADC2 and $b$-value-dependent weighting used for ADC4 by SW2, SW4 to SW7. The detected variations among fit results are clearly significant enough to warrant inclusion of the corresponding descriptive metadata ($b$-values used and fit functions/parameters) in the generated ADC map DICOM header.

3.2.

Site DICOM Implementation versus Parametric Map Standard

Table 2 summarizes the results of ADC DICOM header evaluation across sites. All sites generated single-frame ADC maps and used in-house variations of the DICOM MR object for ADC map storage. Except for the SW2 that copied the source DWI DICOM header, coded terms for the site ADC DICOM implementations were supplied by OsiriX, MATLAB^®, IDL, and scanner DICOM dictionaries (Table 2, source applications). Source DWI geometry was preserved by most implementations. SW4, SW6, and SW7 reversed the slice order for S3, S2, and S2, respectively, while SW2 had wrong slice locations assigned to S3 ADC, which were manually corrected. Except for SW4 (32-bit maps), all ADC map DICOMs were successfully parsed and imported to iQ-View production PACS viewer, as well as, to five additional tested parsers/viewers (listed in Sec. 2.4), indicating general back-compatibility. All headers passed the DICOM validator test with “warnings” for nonstandard attributes and lookup tables, and “errors” related to missing conditional attributes (e.g., Appendix, Fig. 3).

Table 2

ADC DICOM attributes in site implementations.

Fig. 3

Screen-capture of dciodvfy check results for multivendor ADC DICOM generated by SW6.

None of the project participants stored the analysis results using the DICOM PM object. SW8 ADC DICOM contained the most common attributes with the DICOM PM (including standard attributes for ADC units and scale.) ADC fit parameters ($b$-values) and models (algorithms), source-image reference and ADC units/scale attributes were missing from most site implementations or stored in nonstandard (e.g., comment or private) fields (Table 2). Missing or nonstandard DICOM tags for units and scale attributes complicated meta-analysis and numerical interpretations (see Sec. 2.2).

The ADC map scaling consensus across all sites was in the range of (${10}^3$ to ${10}^4$) (Table 1), suggesting that 16-bit precision was found sufficient for ADC map storage. The actual bit-depth stored in site DICOMs exceeded the sufficient range and varied across the site SW tools [from uint12 to u/int16 for most, and uint32 (for SW4), Table 1]. SW1, SW2(S3), and SW8 have used “signed” integer data storage, which allowed unphysical (negative) ADC values. Only one SW4 has utilized 32-bit (integer precision) DICOM storage of comparable range to that offered by PM standard. However, this broad range apparently accommodated background noise values (outside of the observed ADC range).

In the course of the project, the need for coded terms to accommodate a DWI-specific fit algorithm and parameters resulted in preparation of a correction proposal, and the DICOM standard was extended to include the missing attributes. Based on the recently amended standard, multiplatform (PC, Mac, and Linux) open-source dcmqi command-line converter tools have been developed by QIICR to generate 32-bit floating-point multiframe ADC DICOM PM object for several commonly used input formats (NIfTi, Analyze, MHD, etc.). The tools automatically extract relevant source DWI DICOM header information and allow (manual) addition of optional (text) metadata input, including units, $b$-values, and diffusion model, to annotate volumetric ADC images, preconverted to the allowed input formats.

None of the currently supported converter formats included those used by sites in this project (Table 2). The test ADC2 map DICOM converted by this tool properly reflected both source DWI geometry and all optional metadata specified by the standard (Appendix, Fig. 4). This optional metadata was not validated against source DWI DICOM ($b$-value) or analysis environment (algorithm.) Critical metadata regarding fit algorithm functions and parameters was not found (likely still missing from the standard). The object storage precision was not automatically discovered by MATLAB^® (dicom-dictionary), but the 32-bit floating-point volumetric images were recoverable from the “dicominfo” structure “data dump” field. In addition to MATLAB^®, the test DICOM was successfully imported only by OsiriX and 3D Slicer (with QuantitativeReporting extension) and could not be parsed by iQ-View, IDL, ImageJ, 3D Slicer (without the extension), and KPacs viewers indicating limited back-compatibility for the new DICOM PM standard compared to legacy DICOM MR object.

Fig. 4

Example of critical ADC2 (S2) map metadata reflected in DICOM PM header generated by dcmqi.