We propose a learning-based method to automatically segment lung tumor from positron emission tomography (PET)/computed tomography (CT) images. A dual pyramid mask R-CNN is introduced to enable end-to-end segmentation. To avoid the effect of useless region, mask R-CNN is used to get rid of non-tumor regions via fist locate the tumor region-of-interest (ROI) and then segment tumor within that ROI. Dual pyramid networks are used as backbone in mask R-CNN to extract comprehensive features from both CT and PET images. The binary mask of tumor of an arrival patient’s CT and PET image is generated by the welltrained network. To evaluate the proposed method, we retrospectively investigate 42 lung PET/CT datasets. On each dataset, lung tumor was delineated by physicians and was served as ground truth and training target. The proposed method was trained and evaluated by a six-fold cross validation strategy. The average centroid distance, volume difference and DSC value among all 42 datasets are 0.496±0.933mm, 0.303±0.458cc and 0.894±0.080, which indicates that the proposed method is able to generate target contour within 0.5mm error in displacement, 0.3cc error in volume size and around 90% overlapping compared with ground truth. The proposed method has great potential in improving the efficiency and mitigating the observer-dependence in tumor detection and delineation for diagnosis and therapy.
Image-guided radiation therapy (IGRT) is an important technological advancement that has significantly contributed to the accuracy of radiation oncology treatment plan delivery in the last decade. However, the current standard IGRT technique of linac-mounted kilovoltage (kV) cone-beam Computed Tomography (CBCT) has limited soft tissue contrast and is prone to image artifacts, which detract from its clinical utility. It is even worse in chest CBCT compared to other anatomic sites due to respiratory motion, which could lead to mistreatment. Therefore, it is highly desirable to improve CBCT image quality to the level of a planning CT scan. In this study, we propose a novel deep learning-based method, which integrates histogram matching (HM) into a cycle-consistent adversarial network (CycleGAN) framework called HM-CycleGAN, to learn a mapping between chest CBCT images and paired planning CT images obtained at simulation. Histogram matching is performed via an informative maximizing (MaxInfo) loss calculated between planning CT and the synthetic CT (sCT) derived by feeding CBCT into the HM-CycleGAN. The proposed algorithm was evaluated using 15 sets of patient chest CBCT data, each of which has 3-5 daily CBCTs. The planning/simulation CT was used as ground truth for sCTs derived from CBCTs. The mean absolute error (MAE), peak signal-to-noise ratio (PSNR), and normalized cross-correlation (NCC) indices were used to quantify the correction accuracy of the proposed algorithm. The mean MAE, PSNR and NCC were 63.2 HU, 30.2 dB, 0.96 over all CBCT fractions. The proposed method showed superior image quality, reduced noise, and artifact severity compared to the scatter correction method. Upon further improvement and clinical assessment, this method could further enhance accuracy of the current IGRT technique. The CBCT-based synthetic CT could be the critical component to achieve online adaptive radiation therapy.
Due to the inter-fraction and intra-fraction variation of respiratory motion, it is highly desired to provide real-time volumetric images during the treatment delivery of lung stereotactic body radiation therapy (SBRT) for accurate and active motion management. Motivated by this need, in this study we propose a novel generative adversarial network integrated with perceptual supervision to derive instantaneous 3D image from a single 2D kV projection. Our proposed network, named TransNet, consists of three modules, i.e., encoding, transformation, and decoding modules. Rather than only using image distance loss between the generated 3D image and the ground truth 3D CT image to supervise the network, perceptual loss in feature space is integrated into loss function to force the TransNet to yield accurate lung boundary. Adversarial loss is also used to improve the realism of the generated 3D image. We conducted a simulation study on 20 patient cases, who had undergone 4D-CT scan and received lung SBRT treatments in our institution, and evaluated the efficacy and consistency of our method at four different projection angles, i.e., 0°, 30°, 60° and 90°. For each 3D CT image of a breathing phase in the 4D CT image set, we simulated its 2D projections at these two angles. Then for each projection angle, a patient’s 3D CT images of 9 phases and the corresponding 2D projection data were used for training, with the remaining phase used for testing. The mean absolute error (MAE), peak signal-to-noise ratio (PSNR) and structural similarity index metric (SSIM) achieved by our method are 99.5±13.7HU, 23.4±2.3dB and 0.949±0.010, respectively. These results demonstrate the feasibility and efficacy of our method on generating a 3D image from single 2D projection, which provides a potential solution for in-treatment real-time on-board volumetric imaging to guide treatment delivery and ensure the effectiveness of lung SBRT treatment.
Motion management is of utmost importance to ensure the effectiveness of radiation treatment for lung cancer patients. The emerging 4D radiotherapy technique aims to actively manage lung tumor motion by providing a time-dependent treatment. This requires the delineation of tumors on all the 3D phase CT images in 4D CT, which is labor intensive and time consuming. In this study, we propose a novel deep learning-based method to automatically localize and segment lung tumor on 4D CT to facilitate the clinical workflow of 4D radiotherapy. The proposed network, named motion Regionbased Convolutional Neural Networks (R-CNN), consists of four stages, i.e., feature extraction, rough tumor location, fine tumor location, and segmentation within tumor region-of-interest (ROI). To aid the network to get rid of unreasonable detected ROIs, different from traditional mask R-CNN, our proposed method first fed 4D CT with consecutive phases into the backbone to extract tumor motion information and then utilized these motion information to estimate global and local deformation vector fields (DVFs), which is useful for measuring the movement of the tumor ROI between each two phases. Our method was tested on 20 patients’ lung 4D CT images in this study. Five metrics were used for quantitative evaluation, i.e., center-of-mass distance (COMD), Dice similarity coefficient (DSC), 95th percentile Hausdorff distance (HD95), mean surface distance (MSD), and volume difference (VD) calculated between the manual tumor contour and the contour obtained by our method. Averaged over the 20 cases, our method yielded 1.14±0.95 mm on COMD, 0.76±0.25 on DSC, 2.29±1.31 mm on HD95, 0.75±0.44 mm on MSD, and 0.51±0.51 cc on VD. These results have demonstrated the feasibility and efficacy of our motion R-CNN method for automatic localization and segmentation of lung tumors on 4D CT images. Our method is also expected to be applicable to in-treatment real-time volumetric imaging to provide 3D markerless tumor tracking during treatment delivery.
Decreasing administered activity directly reduces radiation exposure to patients and medical staff, but meanwhile has adverse impacts on image quality and PET quantification accuracy. In this work, we propose to integrate multi-modality images and self-attention strategy into a cycle-consistent adversarial network (CycleGAN) framework to generate the full count PET image from low count PET and CT images. During the training stage, deep features are extracted by 3D patch fashion from low count PET and CT images, and are automatically highlighted with the most informative features by self-attention strategy. Then, the deep features are mapped to the full count PET image by using 3D CycleGAN. During the testing stage, the paired patches are extracted from a new arrival patient’s low count PET and CT images, and are fed into the trained networks to obtain the synthetic full count PET image. This proposed algorithm was evaluated using 16 patients’ data. Four-fold cross-validation was used to test the performance of the proposed method. The proposed method suppressed image noise significantly, and obtained images close to the diagnostic PET images. The organ boundaries can be better visualized on the PET images generated with the proposed method. We have investigated a method to estimate diagnostic PET image from low dose data. Experimental validation has been performed to demonstrate its clinical feasibility and accuracy. This technique could be a useful tool for low dose PET imaging.
Deriving accurate attenuation maps for PET/MRI remains a challenging problem because MRI voxel intensities are not related to properties of photon attenuation and bone/air interfaces have similarly low signal. This work presents a learning-based method to derive patient-specific pseudo computed tomography (PCT) maps from routine T1-weighted MRI in their native space for attenuation correction of brain PET. We developed a machine-learning-based method using a sequence of alternating random forests under the framework of an iterative refinement model. Anatomical feature selection is included in both training and predication stages to achieve excellent performance. To evaluate its accuracy, we retrospectively investigated 17 patients, each of which has been scanned by PET/CT and MR for brain. The PET images were corrected for attenuation on CT images as ground truth, as well as on PCT images generated from MR images. The side-by-side image comparisons and joint histograms demonstrated very good agreement of PET images after correction by PCT and CT. The mean differences of voxel values in selected VOIs were less than 4%, the mean absolute difference of all active area is around 2.5%. This work demonstrates a novel learning-based approach to automatically generate CT images from routine T1-weighted MR images based on a random forest regression with patch-based anatomical signatures to effectively capture the relationship between the CT and MR images. Reconstructed PET images using the PCT exhibit errors well below accepted test/retest reliability of PET/CT indicating high quantitative equivalence.
The accuracy of attenuation correction on whole-body PET images is subject to inter-scan motion, image artifacts such as truncation and distortion, and erroneous transformation of structural voxel-intensities to PET mu-map values. We proposed a deep-learning-based method to derive synthetic CT (sCT) images from non-attenuation corrected PET (NAC PET) images for AC on whole-body PET imaging. We utilized a 3D cycle-consistent generative adversarial networks (CycleGAN) to synthesize CT images from NAC PET. The model learns a transformation that minimizes the difference between sCT, generated from NAC PET, and true CT. It also learns an inverse transformation such that cycle NAC PET image generated from the sCT is close to true NAC PET image. Both generators are implemented by a fully convolutional attention network (FCAN), and followed by a discriminator which is structured as a fully convolutional network. A retrospective study was performed with a total of 60 sets of whole-body PET/CT, 40 sets for training and 20 sets for testing. The sCT images generated with proposed method show great contrast on lung, soft tissue and bony structures. The mean absolute error of sCT over true CT is less than 110 HU. Using sCT for whole-body PET AC, the mean error of PET quantification is less than 1% and normalized mean square error is less than 1.4%. We proposed a deep learning-based approach to generate synthetic CT from whole-body NAC PET for PET AC, which demonstrates excellent synthetic CT estimation accuracy and PET quantification accuracy.
Traditional deformable image registration (DIR) algorithms such as optical flow and demons are iterative and slow especially for large 4D-CT datasets. In order to quickly register the 4D-CT lung images for treatment planning and target definition, the computational speed of the current DIRs needs to be improved. Deep learning-based DIR methods that enable direct transformation prediction are promising alternatives for 4D-CT DIR. In this study, we propose to integrate dilated inception module (DIM) model and self-attention gate (Self-AG) into deep learning framework for 4DCT lung DIR. To overcome the shortage of manually aligned ‘ground truth’ training datasets, the network was designed to train in an unsupervised manner. Instead of using only the fixed and moving images as input, we also included the gradient images in x, y, z directions of the fixed and moving images as input to provide the network additional information to help the transformation prediction. The DIM was able to extract multi-scale structural features for robust feature learning. Self-AG were applied at different scales throughout the encoding and decoding pathways to highlight the structure representing feature differences between moving image and fixed image. The network was trained using pairs of 3D image patches that were extracted from any two random phases of one 4D-CT images. The loss function of the proposed network contains three parts which are image similarity loss, adversarial loss and a regularization loss. The network was trained and tested on 25 patients’ 4D-CT datasets using five-fold out cross validation. The proposed method was evaluated using Mean absolute error (MAE), peak signal to noise ratio (PSNR) and normalized cross correlation (NCC) between the deformed image and the fixed image. MAE, PSNR and NCC were 19.2±6.5, 35.4±3.0 and 0.995±0.002 respectively. Target registration errors (TREs) were calculated using manually selected landmark pairs. The average TRE was 3.38 ± 2.36 mm, which was comparable to traditional DIR algorithms. To summarize, the proposed method was able to achieve comparable performance to that of the traditional DIRs while being orders of magnitude faster (less than a minute).
Oncology PET protocols utilize administered activities ranging between 10-15 mCi and 2-3 min/bed positions to obtain diagnostic quality PET images. There is a continued desire to reduce administered activity to reduce radiation exposure and to decrease scanning time to improve patient comfort. However, reducing those parameters lowers photon counts and degrades quantification accuracy. In this work, we proposed a deep-learning-based method to estimate the diagnostic PET image from low count data. A cycle-consistent generative adversarial network (Cycle GAN) was introduced to capture the relationship from low count to full count PET images while simultaneously supervising an inverse full count to low count (full-to-low) transformation model. The network simultaneously makes itself better at both creating synthetic full count PET images and learns how to identify full count PET images. Residual blocks were integrated into the models to catch the differences between low count and full count PET in the training dataset and better handle noise. The proposed model was implemented and evaluated on whole-body FDG PET images with only 1/8th counts. The proposed method obtained the average mean error and normalized mean square error in the whole body of 0.14%±1.43% and 0.52%±0.19%. Normalized cross-correlation was increased to 0.996, and the peak signal-to-noise ratio is increased to 46.0 dB with the proposed method. We developed a deep learning-based approach to accurately estimate diagnostic quality PET datasets from one-eighth of photons, with great potential to substantially reduce the administered dose or scan duration while maintaining high diagnostic quality.
We propose to integrate multi-modality images and self-attention strategy into cycle-consistent adversarial networks (CycleGAN) to predict attenuation correction (AC) positron emission tomography (PET) image from non-AC (NAC) PET and MRI. During the training stage, deep features are extracted by 3D patch fashion from NAC PET and MRI images, and are automatically highlighted with the most informative features by self-attention strategy. Then, the deep features are mapped to the AC PET image by 3D CycleGAN. During the correction stage, the paired patches are extracted from a new arrival patient’s NAC PET and MRI images, and are fed into the trained networks to obtain the AC PET image. This proposed algorithm was evaluated using 18 patients’ datasets. Six-fold cross-validation was used to test the performance of the proposed method. The AC PET images generated with the proposed method show great resemblance with the reference AC PET images. The profile comparison also indicates the excellent matching between the reference and the proposed. The proposed method obtains the mean error ranging from -1.61% to 3.67% for all contoured volumes of interest. The whole-brain ME is less than 0.10%. These experimental studies demonstrate the clinical feasibility and accuracy of our proposed method.
Deriving accurate attenuation correction factors for whole-body positron emission tomography (PET) images is challenging due to issues such as truncation, inter-scan motion, and erroneous transformation of structural voxelintensities to PET μ-map values. In this work, we proposed a deep-learning-based attenuation correction (DL-AC) method to derive the nonlinear mapping between attenuation corrected PET (AC PET) and non-attenuation corrected PET (NAC PET) images for whole-body PET imaging. A 3D cycle-consistent generative adversarial networks (cycle GAN) framework was employed to synthesize AC PET from NAC PET. The method learns a transformation that minimizes the difference between DL-AC PET, generated from NAC PET, and AC PET images. It also learns an inverse transformation such that cycle NAC PET image generated from the DL-AC PET is close to real NAC PET image. Both transformation network architectures are implemented by a residual network and outputs are judged by a fully convolutional network. A retrospective study was performed on 23 sets of whole-body PET/CT with leave-one-out cross validation. The proposed DL-AC method obtained the average mean error and normalized mean square error of the whole-body of -0.01%±2.91% and 1.21%±1.73%. We proposed a deep-learning-based approach to perform wholebody PET attenuation correction from NAC PET. The method demonstrates excellent quantification accuracy and reliability.
We propose a learning method to generate synthetic CT (sCT) image for MRI-only radiation treatment planning. The proposed method integrated a dense-block concept into a cycle-generative adversarial network (cycle-GAN) framework, which is named as dense-cycle-GAN in this study. Compared with GAN, the cycle-GAN includes an inverse transformation between CT (ground truth) and sCT, which could further constrain the learning model. A 2.5D fully convolution neural network (FCN) with dense-block was introduced in generator to enable end-to-end transformation. A FCN is used in discriminator to urge the generator’s sCT to be similar with the ground-truth CT images. The well-trained model was used to generate the sCT of a new MRI. This proposed algorithm was evaluated using 14 patients’ data with both MRI and CT images. The mean absolute error (MAE), peak signal-to-noise ratio (PSNR) and normalized cross correlation (NCC) indexes were used to quantify the correction accuracy of the prediction algorithm. Overall, the MAE, PSNR and NCC were 60.9−11.7 HU, 24.6±0.9 dB, and 0.96±0.01. We have developed a novel deep learning-based method to generate sCT with a high accuracy. The proposed method makes the sCT comparable to that of the planning CT. With further evaluation and clinical implementation, this method could be a useful tool for MRI-based radiation treatment planning and attenuation correction in a PET/MRI scanner.
We propose a method to automatically segment multiple organs at risk (OARs) from routinely-acquired thorax CT images using generative adversarial network (GAN). Multi-label U-Net was introduced in generator to enable end-to-end segmentation. Esophagus and spinal cord location information were used to train the GAN in specific regions of interest (ROI). The probability maps of new CT thorax multi-organ were generated by the well-trained network and fused to reconstruct the final contour. This proposed algorithm was evaluated using 20 patients' data with thorax CT images and manual contours. The mean Dice similarity coefficient (DSC) for esophagus, heart, left lung, right lung and spinal cord was 0.73±0.04, 0.85±0.02, 0.96±0.01, 0.97±0.02 and 0.88±0.03. This novel deep-learning-based approach with the GAN strategy can automatically and accurately segment multiple OARs in thorax CT images, which could be a useful tool to improve the efficiency of the lung radiotherapy treatment planning.
We propose a method to generate patient-specific pseudo CT (pCT) from routinely-acquired MRI based on semantic information-based random forest and auto-context refinement. Auto-context model with patch-based anatomical features are integrated into classification forest to generate and improve semantic information. The concatenate of semantic information with anatomical features are then used to train a series of regression forests based on auto-context model. The pCT of new arrival MRI is generated by extracting anatomical features and feeding them into the well-trained classification and regression forests for pCT prediction. This proposed algorithm was evaluated using 11 patients’ data with brain MR and CT images. The mean absolute error (MAE), peak signal-to-noise ratio (PSNR), and normalized cross correlation (NCC) are 57.45±8.45 HU, 28.33±1.68 dB, and 0.97±0.01. The Dice similarity coefficient (DSC) for air, soft-tissue and bone are 97.79±0.76%, 93.32±2.35% and 84.49±5.50%, respectively. We have developed a novel machine-learning-based method to generate patient-specific pCT from routine anatomical MRI for MRI-only radiotherapy treatment planning. This pseudo CT generation technique could be a useful tool for MRI-based radiation treatment planning and MRI-based PET attenuation correction of PET/MRI scanner.
We propose a learning method to generate corrected CBCT (CCBCT) images with the goal of improving the image quality and clinical utility of on-board CBCT. The proposed method integrated a residual block concept into a cyclegenerative adversarial network (cycle-GAN) framework, which is named as Res-cycle GAN in this study. Compared with a GAN, a cycle-GAN includes an inverse transformation from CBCT to CT images, which could further constrain the learning model. A fully convolution neural network (FCN) with residual block is used in generator to enable end-toend transformation. A FCN is used in discriminator to discriminate from planning CT (ground truth) and correction CBCT (CCBCT) generated by the generator. This proposed algorithm was evaluated using 12 sets of patient data with CBCT and CT images. The mean absolute error (MAE), peak signal-to-noise ratio (PSNR), normalized cross correlation (NCC) indexes and spatial non-uniformity (SNU) in the selected regions of interests (ROIs) were used to quantify the correction accuracy of the proposed algorithm. Overall, the MAE, PSNR, NCC and SNU were 20.8±3.4 HU, 32. 8±1.5 dB, 0.986±0.004 and 1.7±3.6%. We have developed a novel deep learning-based method to generate CCBCT with a high image quality. The proposed method increases on-board CBCT image quality, making it comparable to that of the planning CT. With further evaluation and clinical implementation, this method could lead to quantitative adaptive radiotherapy.
We propose a high-resolution CT image retrieval method based on sparse convolutional neural network. The proposed framework is used to train the end-to-end mapping from low-resolution to high-resolution images. The patch-wise feature of low-resolution CT is extracted and sparsely represented by a convolutional layer and a learned iterative shrinkage threshold framework, respectively. Restricted linear unit is utilized to non-linearly map the low-resolution sparse coefficients to the high-resolution ones. An adaptive high-resolution dictionary is applied to construct the informative signature which is highly connected to a high-resolution patch. Finally, we feed the signature to a convolutional layer to reconstruct the predicted high-resolution patches and average these overlapping patches to generate high-resolution CT. The loss function between reconstructed images and the corresponding ground truth highresolution images is applied to optimize the parameters of end-to-end neural network. The well-trained map is used to generate the high-resolution CT from a new low-resolution input. This technique was tested with brain and lung CT images and the image quality was assessed using the corresponding CT images. Peak signal-to-noise ratio (PSNR), structural similarity index (SSIM) and mean absolute error (MAE) indexes were used to quantify the differences between the generated high-resolution and corresponding ground truth CT images. The experimental results showed the proposed method could enhance images resolution from low-resolution images. The proposed method has great potential in improving radiation dose calculation and delivery accuracy and decreasing CT radiation exposure of patients.
We propose a CBCT image quality improvement method based on anatomic signature and auto-context alternating regression forest. Patient-specific anatomical features are extracted from the aligned training images and served as signatures for each voxel. The most relevant and informative features are identified to train regression forest. The welltrained regression forest is used to correct the CBCT of a new patient. This proposed algorithm was evaluated using 10 patients’ data with CBCT and CT images. The mean absolute error (MAE), peak signal-to-noise ratio (PSNR) and normalized cross correlation (NCC) indexes were used to quantify the correction accuracy of the proposed algorithm. The mean MAE, PSNR and NCC between corrected CBCT and ground truth CT were 16.66HU, 37.28dB and 0.98, which demonstrated the CBCT correction accuracy of the proposed learning-based method. We have developed a learning-based method and demonstrated that this method could significantly improve CBCT image quality. The proposed method has great potential in improving CBCT image quality to a level close to planning CT, therefore, allowing its quantitative use in CBCT-guided adaptive radiotherapy.