2.1.

Cone-Beam CT and Image Artifacts

In CBCT-guided surgery, multiple scans may be acquired during a procedure, so minimizing the dose per scan is an important goal. One method to reduce dose is via sparse-view acquisition (using pulsed x-ray exposures) to acquire projection images at larger angular intervals (e.g., 2-deg intervals over 206 deg, as opposed to 1-deg intervals over 360 deg, amounting to a $\sim 2\times$ reduction in view sampling density and a $\sim 3.5\times$ reduction in dose), referred to below as a sparse short-scan. For standard filtered backprojection (FBP) reconstruction, such scan data result in images that are both noisy ($\sim 1.9\times$ increase in quantum noise) and subject to view sampling artifacts. For images acquired in the presence of metal, not only do standard beam-hardening metal artifacts arise [Fig. 1(a)], but strong view sampling streaks also occur [Fig. 1(c)] due to the high-frequency edges at the metal-tissue boundary.

The data associated with the sparse short-scan acquisition comprise 104 projection images, each $384\times 1024\text{pixels}$ [$0.776\times 0.388{\mathrm{mm}}^2\text{pixel}$ size, Fig. 2(b)]. As shown in Fig. 1, detector coordinates are denoted $(u,v)$, and the number of projections is denoted $p$. Log-normalization and transpose of this stack of images yields the 384 sinograms used to reconstruct the 3D image [Fig. 2(b)], with FBP reconstruction yielding a $512\times 512\times 192$ volume ($0.415\times 0.415\times 0.83{\mathrm{mm}}^3\text{voxel}$ size). While such sinograms can be reconstructed sequentially in classic axial CT, forming one axial slice at a time, cone-beam geometry necessitates non-axial divergent backprojection in image reconstruction. CBCT backprojection is therefore performed over the entire reconstructed volume, as in the Feldkamp-Davis-Kress algorithm for 3D FBP.⁴ Due to the cone-beam geometry and fully 3D volumetric nature of the backprojection reconstruction, many existing methods for CNN-based MAR in CT (summarized in Sec. 2.3) are not readily transferable to CBCT, as they incur major memory limitations in processing the full 3D sinogram and image volume (rather than a single 2D sinogram and 2D axial slice).

Fig. 2

Data for sparse-view image reconstruction. (a) Illustration of the O–arm™ imaging system showing cone-beam acquisition geometry. (b) Projection data acquired over a short-scan orbit are converted to log-normalized sinograms.

2.2.

Traditional MAR and Sparse Reconstruction Techniques

Traditional MAR methods commonly rely on segmentation of the metal in the reconstructed image volume (reconstructed without corrections), and the segmentation is forward-projected to achieve a sinogram metal trace that is in turn used to “inpaint” (i.e., replace with some computed value) the metal-containing regions of the original sinogram. A common approach uses linear interpolation for inpainting,⁵ and more recent methods use an anatomical prior to normalize the sinogram prior to inpainting.⁶

A limiting step in traditional MAR is segmentation of the metal object in the uncorrected reconstruction. Because the initial image is contaminated with strong metal artifacts, accurate segmentation is difficult and often ad hoc because of inconsistencies in reconstructed voxel values and biases in the reconstructed CBCT image (e.g., due to scatter and truncation). Even small segmentation errors can result in strong artifacts in the 3D reconstruction: when a segmentation underestimates the extent of a metal object, the uncorrected pixel values result in residual artifacts; on the other hand, segmentations that overestimate the extent of a metal object cause inpainting of pixels corresponding to adjacent anatomy, blurring the region surrounding the metal object in an unrealistic manner. Moreover, such approaches are severely challenged when the metal component lies outside of the reconstructed field-of-view (FOV) and image-domain segmentation of the component is not achievable. Iterating on the segmentation estimate through successive passes of the approach described above can improve performance, but residual artifacts persist at a level that challenges clear visualization of metal and adjacent structures.

Such limitations may be addressed using model-based iterative reconstruction (MBIR) to reduce metal artifacts by down-weighting low-fidelity data and utilizing polyenergetic models for the beam and material attenuation during reconstruction.⁷ Furthermore, known-component registration (KC-Reg)⁸ and reconstruction (KC-Recon)⁹ techniques have been developed to address such limitations, where a 3D model of the implant is registered to the projection data to obtain a high-fidelity metal trace that can be used for inpainting and/or MBIR. As an alternative to image-processing techniques, novel acquisition strategies have been developed that effectively avoid metal artifacts by means of tilted-axis¹⁰ or non-circular¹¹ orbits to avoid projection angles that contain regions of severe beam hardening.

Sparse-view image reconstruction typically relies on sinogram interpolation or MBIR techniques. The former synthesizes higher angular sampling density in the sinogram, originally investigated using linear interpolation,¹² and with modern implementations modeling view-to-view interpolation according to the scan geometry.¹³ On the other hand, iterative reconstruction techniques using total-variation regularization methods^14,15 may be employed to account for the sparsity of the data; however, the regularization parameters associated with such approaches must be carefully controlled (in a manner that is data-dependent) to avoid residual piecewise-constant artifacts in the image.

2.3.

Learning-Based MAR Techniques

Learning-based MAR techniques have been developed and can be separated according to the domain in which they operate, e.g., the sinogram, the reconstructed image, or both. Methods from Gjesteby et al.¹⁶ and Xu and Dang¹⁷ exclusively operate in the reconstructed image domain to directly remove metal artifacts from the image in a manner amenable to post-processing without access to the scan data.

Other methods, including Ghani and Karl,¹⁸ Park et al.,¹⁹ and Liao et al.²⁰ operate in a manner similar to traditional MAR in that they use a metal segmentation to inpaint within the projection/sinogram domain and avoid artifacts in the first place. Zhang and Yu²¹ utilized a CNN to achieve a reduced artifact prior image that is forward-projected and used to inpaint the original sinogram. Lin et al.²² developed a dual-domain technique, employing a network in both the sinogram domain for inpainting and in the image domain to enforce artifact suppression and sinogram consistency. By incorporating FBP reconstruction within the dual-domain network architecture, they were able to create an end-to-end model to backpropagate the image-domain loss information into the sinogram inpainting network, which reduced the introduction of new artifacts due to inconsistent sinogram modifications. The approach was shown to work well for CT imaging (i.e., slice-by-slice reconstruction); however, the full volume reconstruction in CBCT is challenged by the larger GPU memory requirements for CNN techniques.

While image domain methods can be effective in reducing metal artifacts, they can be limited in severely affected regions in which there is insufficient data fidelity (e.g., low signal and high noise) and/or insufficient sampling (e.g., sparse-view acquisition). CNN inpainting techniques tend to perform better than image-domain methods at eliminating severe artifacts; however, as with their traditional counterparts, they rely on an accurate metal segmentation prior to corrections, which is particularly challenging in intraoperative CBCT imaging where metal may be outside of the reconstructed FOV.

3.1.

Proposed Framework

While previously reported learning-based methods address MAR in fully sampled data (often CT, with the notable exception of Lin et al.²² addressing CBCT), the work reported below demonstrates a learning-based MAR technique for sparse-view CBCT imaging. We approach this challenge in a physically motivated, dual-network framework (referred to as CNNMAR-2)—the first network operating in the sinogram domain, and the second in the reconstructed image domain. Together, the approach addresses physical factors associated with both sparse-view sampling and beam-hardening artifacts. Unlike conventional MAR, the proposed approach does not rely on an explicit metal segmentation and it is non-iterative (i.e., free from repeated forward and backprojection as in conventional beam-hardening correction methods).

Figure 3 shows the general form of the CNNMAR-2 framework. Rather than processing the entire stack of sinograms or the entire image volume with a 3D CNN, the 3D images are reshaped into slabs of three neighboring slices (i.e., 2D images with three channels) and processed sequentially (or in sequential batches). The output of the 2D CNN in each scenario is a single-channel 2D image, relating to the middle channel of the three-channel input. Projection data (following standard correction of detector gain, offset, and pixel defects) are stacked in order of angular sampling and transposed to create the sinograms (Fig. 2). The sinograms are processed by the first CNN with slabs determined by detector row order, and the output sinograms are restacked for FBP reconstruction. Similarly, after each slab of axial slices is processed by the second CNN, the images are restacked into the image volume. The sinogram domain network addresses both sparsity and beam hardening, and the resulting FBP reconstruction of the corrected sinogram data is largely removed of biases, thereby leaving the image domain network to treat residual artifacts from imperfect or inconsistent corrections.

Fig. 3

Proposed framework (CNNMAR-2) for MAR in sparse-view CBCT. The first network operates in the sinogram domain and the second in the reconstructed image domain.

The approach is compared to an image-domain CNN (referred to as CNNMAR-1) that is tasked with removing both the sparse-view and beam-hardening artifacts in the reconstructed image. The approach is similar to prior work in MAR¹⁶ and sparse-view data.²³ However, to our knowledge, such factors have not been simultaneously treated by means of an image-domain CNN. Both approaches are shown in Fig. 4(a). It is worthy to note that the input for the image-domain network for CNNMAR-1 is the sparsely reconstructed FBP (denoted “Sparse FBP”), whereas the input for the image-domain network of CNNMAR–2 is the FBP reconstruction resulting from the sinogram-domain network (denoted CNNMAR-2 Sinogram Only).

Fig. 4

(a) Flowcharts for implementations of the two CNN frameworks. The CNNMAR-1 method addresses both beam-hardening and sampling in the image domain following FBP reconstruction of the sparsely sampled acquisition. The CNNMAR-2 method compensates for both sparsity and beam-hardening artifacts in sinogram domain, while also removing residual artifacts in the image domain. Detailed structure of the (b) dense and (c) residual blocks.

3.2.

Network Architecture

The networks were implemented in TensorFlow²⁴ using the Keras API and trained with the Adam optimizer²⁵ with a learning rate of ${10}^{-4}$. All networks used in this work used an identical architecture (Fig. 4), based generally on the architecture of Zhang,²³ which was designed for image-domain view-sampling artifact correction. The input slab is first processed by a 64-channel $7\times 7$ ConvReLU, followed by four dense encoder blocks.²⁶ As shown in Fig. 4(b), the dense blocks begin with a $2\times 2\max$-pooling, followed by four 64-channel $5\times 5$ ConvReLU blocks (each with “dense” concatenation of the previous outputs), and finally a 64-channel $1\times 1$ ConvReLU block. A series of four decoding residual blocks are then applied using a concatenation scheme similar to that of a U-Net.²⁷ The residual block [Fig. 4(c)] begins with $2\times 2$ bilinear upsampling, and the output is concatenated with the associated encoder block output. The result is passed to a 128-channel, $5\times 5$ ConvReLU block, and then a 64-channel, $1\times 1$ ConvReLU block before being added to the output of the bilinear upsampling. The output of the decoder network is passed to a three-channel $3\times 3$ Conv block, which is then added to the network input and passed to a final one-channel $7\times 7$ Conv.

3.3.

Training Data and Metal Simulation

Training data were generated from CBCT scans of multiple cadaver specimens imaged with the O-arm™ Imaging system (Medtronic, Minneapolis, Minnesota). The dataset comprised 25 thoracic and lumbar scans acquired over a 360-deg orbit—24 scans at 0.5-deg increments, and 1 scan at 1 deg increments. Six of the scans (three of which contained implanted spine pedicle screws) were reserved for testing. From the raw projection image data, sparse short-scans were generated by subsampling these scan data at 2-deg intervals over 206 deg.

Metal screw simulation was performed by augmenting the metal-free training data to include surgical screws. Computer-aided design models of spinal pedicle screws (each comprising a screw shaft and a polyaxial tulip head) of various lengths and diameters (31 shaft and two tulip head models) were randomly sampled and placed within the coordinate frame of the cadaver image. These screw models were then digitally forward-projected according to the projective geometry of the imaging system, yielding projection images that define the path lengths (in mm) for rays traveling through the metal component. As the shaft ($s$) and tulip ($t$) differ in material composition, they are forward-projected independently (yielding line integral distances $d_s(u,v)$ and $d_t(u,v)$ in mm for each projective angle) so that they may be treated separately for beam hardening simulations (Fig. 5, bottom left).

Fig. 5

Flowchart of the metal simulation process. Original projections were first optionally injected with noise to simulate low tube current projections. Then screws were defined in 3D anatomy and forward projected under the imaging system geometry of the projection images to obtain line-integral distances of the shafts ($d_s$) and tulips heads ($d_t$). These simulated forward projections were used to augment the metal-free projection data to create metal-injected projections $P_{\text{poly}}$ and $P_{\text{mono}}$. The images were then reconstructed with FBP to yield the polyenergetic and monoenergetic reconstructions. The lack of metal artifacts in the monoenergetic model is noteworthy.

Prior to metal injection, a reduced tube current (mA) simulation²⁸ of the projection data $P(u,v)$ was performed to achieve physically accurate noise augmentation. Simulated screw projections were added to the existing raw projection images of anatomy (Fig. 5) using the following simulation model based on the Beer–Lambert law:

3.4.

Evaluation

4.1.

Results on Sparse Data Containing Metal

Figure 6 depicts the output for each of the methods on three cadaver images, each containing surgically implanted metal screws. For each cadaver, the first image shows the Full Dose FBP condition with no MAR method applied (206-deg scan at 0.5-deg increments for cadavers 1 to 2, 1-deg increments for cadaver 3). The second image shows the Sparse FBP reconstruction (206 deg scan at 2 deg increments). The third column shows the output of the sinogram-only step of CNNMAR-2 followed by the final outputs of CNNMAR-1 and CNNMAR-2.

Fig. 6

Reconstructed images for three cadaver scans. From left-to-right: Full-Dose FBP, Sparse FBP, CNNMAR-2 Sinogram-Only, CNNMAR-1, and CNNMAR-2.

The output of CNNMAR-1 (which takes the Sparse FBP as input) indicates that the fully image-domain approach provides significant image quality improvement, removing nearly all view-sampling artifacts and much of the beam-hardening artifact. Remaining artifacts appear localized to the tulip region (i.e., screw head), particularly in regions between two tulips. In such regions, it appears that the beam-hardening artifact almost completely obscured the anatomy, and the network learned to primarily blur out this region (as seen in Cadaver 2, CNNMAR-1). The CNNMAR-2 method addresses both view-sampling and beam-hardening artifacts prior to FBP reconstruction which is observed in Fig. 6 to remove nearly all the shadowing and streaking artifact. Not only is the artifact reduced, but the original anatomy (particularly in locations between two screws) is preserved. Much of this improvement can be attributed to the sinogram-domain step of CNNMAR–2, where, compared to the sparse FBP, the sinogram-only output significantly reduces both sparse-view and beam-hardening artifact, yielding a more reliable starting point for the image-domain network to correct the residual artifacts (compared the Sparse FBP starting point of CNNMAR-1).

Compared to standard inpainting methods, CNNMAR-2 does not rely on an explicit segmentation of the metal objects, typically generated by performing an uncorrected FBP reconstruction and applying a threshold segmentation. While such inpainting methods generally perform well at reducing metal artifacts, the performance is diminished when metal is present outside of the reconstructed FOV and a segmentation of the metal cannot be obtained, as observed in a different axial slice of Cadaver 1 [Fig. 7(b) which contains a pair of forceps outside the reconstructed FOV with artifacts emanating from the top-left corner of the image. The output of CNNMAR-2 [Fig. 7(c)], however, shows a robustness to the truncation of metal objects, which is a common scenario in the limited FOV setting of intraoperative CBCT.

Fig. 7

Illustration of performance when metal is present outside the reconstructed FOV. In this case, a pair of forceps cause metal artifacts emanating from the top-left corner of the image. From left-to-right: (a) Sparse FBP; (b) Full Dose Reconstruction using a standard MAR inpainting method; and (c) CNNMAR-2.

Qualitative analysis in Fig. 6, indicated that while both CNNMAR methods significantly reduced metal artifacts, CNNMAR-1 tended to blur anatomical content in regions of strong artifact. The effect was quantified by computing the standard deviation of the gradient image in regions between the screws, particularly emphasizing the spinal canal. An exemplary region is shown in red boxes in Figs. 8(b) and 8(c), noting that regions were specifically selected to highlight pertinent anatomy while avoiding high-frequency artifacts. The box plots in Fig. 8(d) show a significant ($p<0.005$, paired $t$-test) improvement in the standard deviation of the gradient within these regions, indicating that CNNMAR-2 better preserves anatomical content in regions of intense artifact.

Fig. 8

Quantification of anatomical structure preservation. (a) Sparse FBP for reference. (b) CNNMAR-1 and (c) CNNMAR-2 outputs. (d) Standard deviation of the gradient magnitude image is evaluated region between two metal tulip heads that contain pertinent anatomical content (spinal canal) and avoid high-frequency artifacts [exemplary ROI shown in red boxes (b) and (c) with gradient magnitude image to the right]. Higher values indicate that more anatomy is preserved.

Quantification of artifact removal was examined by computing the ROI standard deviation in the region between the tulip heads in regions that should otherwise be homogenous. Exemplary ROIs in Fig. 9 show such regions of consistent strong artifact due to the aligned metal edges of the tulip heads. While both CNNMAR methods remove the majority of the metal artifacts in the image (as observed in the qualitative analysis of Fig. 6), the boxplots in Fig. 9 show significant ($p<0.005$, paired $t$-test) reduction of the artifacts in these particularly severe regions ($5\times {10}^{-3}\pm 2\times {10}^{-3}{\mathrm{mm}}^{-1}$ [mean ± std] for CNNMAR–1 versus $3.5\times {10}^{-3}\pm 1.3\times {10}^{-3}{\mathrm{mm}}^{-1}$ for CNNMAR-2).

Fig. 9

Quantification of artifact removal. (a) Sparse FBP, CNNMAR-1, and CNNMAR-2 reference images with exemplary ROIs shown in red boxes. (b) Standard deviation within the ROIs was computed for both CNNMAR methods. Higher values indicate an increased presence of artifact.

4.2.

Results on Sparse Data with Simulated Metal

Figure 10 shows the performance of the CNNMAR methods on metal data simulated using the polyenergetic simulation method shown in Fig. 5. Both methods remove nearly all metal artifacts from the image, yielding a highly accurate reconstruction of the screws; however, due to the sparse view acquisition strategy, reduced contrast in soft-tissue and bony structures is observed for both methods. The boxplots of Fig. 10(b) show the RMSE between the CNNMAR outputs and the ground truth images in ROIs near the screw locations (with metal and air regions removed from the RMSE computation). A small but significant ($p<0.005$, paired $t$-test) reduction in RMSE is observed using the CNNMAR-2 method, again indicating that it provides greater preservation of anatomical structure ($3.7\times {10}^{-3}\pm 0.4\times {10}^{-3}{\mathrm{mm}}^{-1}$ (mean ± std) for CNNMAR–1 vs. $3.4\times {10}^{-3}\pm 0.4\times {10}^{-3}{\mathrm{mm}}^{-1}$ for CNNMAR-2).

Fig. 10

Performance of CNNMAR methods in data with simulated metal screws. (a) From left-to-right, exemplary images of the monoenergetic fully sampled ground truth, polyenergetic Sparse FBP, CNNMAR-1 output, CNNMAR-2 output. (b) Boxplots depicting the RMSE error of the CNNMAR methods compared to ground truth.

Figure 11 illustrates the presence of metal blooming for both CNNMAR-1 and CNNMAR-2 in the simulated metal dataset. Both methods markedly reduce the metal blooming, clearly revealing details in the screw threads. The boxplots in Fig. 11(b) shows a small but significant ($p<0.005$, paired $t$-test) reduction in the blooming ratio using CNNMAR-2 (1.15 mean blooming ratio) versus CNNMAR-1 (1.19 mean blooming ratio).

Fig. 11

Blooming ratio evaluation of CNNMAR methods applied on sparse scans with simulated polyenergetic screws. (a) Exemplary images with Sparse FBP reference and CNNMAR-1 and CNNMAR-2 outputs. Red lines depict an exemplary cross-section for evaluation the screw diameter, with the inset plots showing line profiles with the diameter computed using the full width at 20% of the maximum. (b) Boxplots depict the distribution of the blooming ration among the dataset for both CNNMAR methods.

4.3.

Results on Sparse Data at Reduced Tube Current

Reducing the tube current in sparse-view image acquisition allows further reduction of the total dose of the scan while maintaining the 2-deg angular sampling of the sparse-view acquisition. Figure 12(a) shows the performance of the CNNMAR-2 method as the mA is lowered (using simulated noise injection) from 25 to 1.56 mA, where the top and bottom row show nearby axial slices (with the top row containing pedicle screws). Compared to the 1.56 mA Sparse FBP, the 1.56 mA CNNMAR-2 output greatly reduces the image noise (from $6.1\times {10}^{-3}$ to $1.6\times {10}^{-3}{\mathrm{mm}}^{-1}$ noise standard deviation) with only a slight decrease in the measured edge-spread-function width (${\sigma }_{esf}$) from 0.5 to 0.6 mm, where a comparable reduction in noise using a Gaussian filter (Sparse FBP: Blur) would increase ${\sigma }_{esf}$ to 0.9 mm. Therefore, the noise reduction associated with CNNMAR comes with only a minimal expense of spatial resolution. Furthermore, qualitative inspection of the 1.56-mA CNNMAR-2 output shows similar performance in artifact reduction compared to the 25-mA CNNMAR-2 reconstruction.

Fig. 12

Evaluation of CNNMAR-2 at reduced tube current. (a) Visualization of CNNMAR-2 performance at 1/16th the nominal tube current. Top and bottom rows depict two nearby axial slices. Columns from left to right show CNNMAR-2 at 25 mA, Sparse FBP at 1.56 mA, Sparse FBP at 1.56 mA with Gaussian blur, and CNNMAR-2 and 1.56 mA. Below each column show the measured ESF width noise standard deviation. (b) Plot of the tube current vs. noise standard deviation for CNNMAR-2 and Sparse FBP.

Figure 12(b) plots the noise (standard deviation in voxel values in an otherwise uniform region) as the tube current is lowered down to 1/32nd of the nominal mA for the scan. The noise for CNNMAR-2 remains under $2\times {10}^{-3}{\mathrm{mm}}^{-1}$ across all tube current levels, outperforming Sparse FBP at its highest dose level. Therefore, tube current reduction presents a promising means to further reduce dose in a manner compatible with CNNMAR-2 workflow (with benefits in metal as well as non-metal regions).