2.1.

Kidney Area Is Segmented from the Surrounding Abdominal Structures by U-net

Water images of the Dixon method used in this study have a clear renal region. In addition, this section describes a rough extraction of the renal region. Therefore, we have decided to use U-net, which is known to provide good results for the area extraction of medical images.¹³ Coronal sections of Dixon water images were used. There are three to six sliced images per subject, and the image size is $320\times 320$. We used 1201 images of 174 cases. All images were created by a specialist as a renal segmentation image. Figure 2 shows the U-net used for renal region extraction. The network discriminates between three classes: kidney, kidney boundaries, and other tissue. However, there is an imbalanced distribution of samples in the kidney boundaries class as compared with the other tissue class. We used a weighted cross-entropy loss in order to compensate for this imbalance and achieve more accurate learning when training the network. We used softmax with weighted cross-entropy loss for network output and true label comparison. Cost minimization on 50 epochs was performed using the adaptive moment estimation optimizer with a learning rate of 0.0001. The training time for this network was $\sim 1\mathrm{h}$ on a workstation with an NVIDIA TITAN RTX GPU × 2. A total of 1201 images were divided into 600 and 601 images, and classification was performed to extract the renal region. The renal function estimation experiment was then performed for all subjects. Figure 3 shows the extraction results obtained by U-net. Figure 3(a) is an input image, and Fig. 3(b) is a teacher image. Figure 3(c) shows the extraction result. It can be confirmed that there are few false positives and that the region close to the renal region can be extracted.

Fig. 2

The U-net architecture used for renal region extraction in this study.

Fig. 3

Results of renal region extraction by U-net: (a) original, (b) mask, and (c) result.

2.2.

False-Detection Removal by 3D Labeling

Although the renal region detected by U-net was highly accurate, false detection occurred. False positives are extracted for organs other than kidneys, as shown in Fig. 4(b). This may be because when U-net is trained with $64\times 64$ patches, spatial information larger than the patch size is lost. To improve this problem, methods including detecting the location of the kidney by object detection, such as Faster R-CNN¹⁴ or YOLOv3,¹⁵ and applying semantic segmentation in a narrow region are considered. However, in this study, the boundary of the kidney by U-net is well discriminated, and there are few false positives for other organs, so there is little need to complicate the process. Therefore, we decided to perform 3D labeling for each subject and exclude areas other than those with a large area. The processed results are shown in Fig. 4. Figures 4(a)–4(c) show the extraction results by U-net. It can be confirmed that false positives occur in areas other than the renal region. Figures 4(d)–4(f) show the results of 3D labeling and exclusion of areas with a 3D area of 2500 pixels or less. This procedure was applied to all 1201 images, and it was confirmed that there were no cases of the renal region being accidentally excluded.

Fig. 4

Kidney area obtained by U-net: (a) slice 1, (b) slice 2, and (c) slice 3. Result of the elimination of noise area by 3D labeling: (d) slice 1, (e) slice 2, and (f) slice 3.

2.3.

Nonrigid Registration of Kidney Area and Target Mask

An effective method for estimating renal function is to analyze the structure of the kidney while considering the kidney anatomically.^5,16 However, time and effort are required for a specialist to manually extract the renal region. In addition, manual measurements will result in different results depending on the specialist. Therefore, we propose a method by which to automatically identify the positions of the cortex and medulla of all images by registration of all of the renal regions extracted by U-net to one target mask. The registration method is performed in two steps using a MATLAB function. The first step is a brightness-based affine transformation. The brightness-based registration method calculates the similarity between two images, repeats the affine transformation so that the similarity is high, and estimates the geometric transformation (translation/rotation/scaling/shear) with the highest similarity. The processing results are shown in Fig. 5. Figure 5(a) is the original image. Figure 5(b) shows the results of extracting the renal region by U-net. Figure 5(c) is the target mask. Figure 5(c) is the renal region of a normal renal image extracted by a specialist. A typical patient has two kidneys. In this study, we analyze the kidney with a larger area. In Fig. 5(b), the left kidney is larger. As such, the left kidney is extracted and registered with the target mask. Figure 5(d) shows a pseudocolor image of the target mask and the initial position of the extracted kidney. The pseudocolor image shows the target mask in green, the extracted kidney in magenta, and the overlapping pixels of both in white. The result of the affine transformation is shown in Fig. 5(e). Next, the shape is finely modified by registration using a brightness-based displacement field.^9,10 In this paper, the registration is performed using the displacement field based on Thirion’s demons algorithm.^1,2 The result of modification of the image shown in Fig. 5(e) using the displacement field is shown in Fig. 5(f). The fine shape has been modified to approach the target mask. Figure 5(g) shows the final registration of the image.

Fig. 5

Result of rigid transform: (a) original image, (b) result by U-net, (c) target mask, (d) initial position, (e) affine transform, (f) displacement field, and (g) result of registration.

2.4.

Calculation of TLCO

In this study, the TLCO method is used for the analysis of renal function. The TLCO method specifies the inner (medullary side) and outer (cortical side) boundaries of the renal parenchyma and divides the renal region into 12 layers from the outside to the inside. The method for layering is not explained in the TLCO paper.⁸ Therefore, in this study, we calculated the TLCO using geodesic distance conversion.¹⁷ When geodesic distance conversion inputs a binary image of the renal region and a seed image, a distance conversion image is generated based on the seed image. Figure 6(a) shows a binary image of the renal region extracted by a specialist. Figure 6(b) shows a seed image made by a specialist. Figure 6(c) shows a pseudocolor image of the geodesic distance conversion. However, the image shown in Fig. 6(c) was not obtained by dividing the renal region into 12 layers. Therefore, the result of geodesic distance conversion was normalized into 12 layers. Specifically, the number of layers was divided by the maximum value and then multiplied by 12. The normalized geodesic distance conversion pseudocolor image is shown in Fig. 6(d). The target mask is very important because it directly affects the calculation of the TLCO method. In this study, we decided to use two types of images, a normal kidney image selected by a specialist and the kidney image with the largest region. Since the TLCO method divides the kidney into 12 layers, calculation with a small atrophied kidney is difficult. Therefore, a kidney image with a small area cannot be used as a target mask. Figure 7 shows the results of applying the TLCO method to MRI images of patients. Figure 7(a) is a normal image selected by a specialist. Figure 7(b) shows the results of the specialist extraction of the renal region from Fig. 7(a). Figure 7(c) is a pseudocolor image divided into 12 layers by the TLCO method. Figure 7(d) is an image of the kidney with the largest area. Figure 7(e) shows the result of the specialist extracting the renal region from Fig. 7(d). Figure 7(d) is a pseudocolor image divided into 12 layers by the TLCO method.

Fig. 6

Calculation of the TLCO: (a) mask image, (b) seed image, (c) geodesic distance conversion, and (d) TLCO.

Fig. 7

TLCO of the target mask: (a) normal kidney, (b) mask, (c) TLCO, (d) large kidney, (e) mask, and (f) TLCO.

2.5.

eGFR Used to Estimate Renal Function by RSVM

In the TLCO method, we used 12 measurements, one for each layer, to estimate the corticomedullary gradient of MR signal intensity based on the following definitions: cortical (mean of the three superficial layers), medullary [mean of deep layers (layers 8 to 10)], and the gradient [linear fit to the data points (layers four to seven) when plotting MR signal intensity versus % of depth]. It has been reported that the gradient of the MR signal intensity correlates with the eGFR.⁸ As chronic damage progresses in the kidney, the difference between the outer and inner layers in the internal structure disappears. The nephrologist evaluates these changes from the images. Therefore, computing the gradient of MR signal intensity from the outer to inner layers, the fourth to seventh layers provides an index that replaces the human evaluation index. The process has been explained in previous reports, and this index is known to be useful in the evaluation of kidney images.^5,16,18 In this study, we decided to use machine learning to estimate renal function. At this time, eGFR is a continuous value. Therefore, eGFR is estimated using a regression support vector machine (RSVM). In addition, the TLCO method uses the average value of each layer. However, in this study, the histogram of each layer is used as a feature. The number of bins in the histogram is 50. The finally calculated features are the mean, kurtosis, skewness, median, a median of each TLCO (12 feats) and histogram (50 feats) of all renal regions, and the histogram of each TLCO ($12\times 50=600\text{feats}$) was calculated. In addition, a noise-removed image was created using a median filter. The noise-removed image was created by applying the median filter once and twice. The feature was calculated from the noise-removed image by the same method. Therefore, the number of features calculated as a feature candidate is 1998.

3.1.

Patient Data and MRI Acquisition Protocol

CKD patients with a sufficient number of visits for the treatment and evaluation of CKD (more than three times during a period of at least 1 year) were enrolled ($n=165$, 105 men and 55 women, aged from 19 to 87 years, and with a mean age of $60\pm 15\text{years}$). The proposed framework has been tested on two-dimensional water images of the Dixon method data sets collected from 165 subjects. Magnetic resonance examinations were performed on a 3.0 T (Skyra, Siemens Healthcare, Erlangen, Germany) scanner using a spine coil and an 18-channel phased-array body coil. Coronal 3D T1-weighted volume-interpolated breath-hold examination Dixon images were generated from in-phase, out-of-phase, fat-only, and water-only images.¹² The MRI parameters were as follows: echo time (TE); delta TE (= 2.46 ms); delta (= 1.23); repetition time (= 5.35 ms); flip angle (= 10 deg); field of view ($=360\times 360\times 144\mathrm{mm}$); recon matrix (= 320); and respiratory compensation (= breath-hold). During the observation period, blood and urine data were sampled every 3 to 6 months. The eGFR was calculated using the formula for Japanese patients,¹⁹ i.e., eGFR ($\mathrm{mL}/\min /1.73{\mathrm{m}}^2$) = 194 × Cr − 1.094 × age − 0.287 (× 0.739 for female cases). Since eGFR always decreases, we performed a linear approximation using blood test results from multiple time points and calculated the eGFR at the date of imaging. We also estimated the rate of decline in eGFR using the same linear approximation method and subsequently defined the “annual rate of decline in eGFR = eGFR slope” as an index of CKD prognosis. Urinary protein (UPro) levels were noted and corrected for urinary creatinine (urine protein: creatinine ratio, g:g). The Institutional Review Board of Saitama Medical University approved this research and its publication in accordance with the Declaration of Helsinki and the Japanese guidelines for clinical research (Approval Nos. 12065, 18107, 19048.01, 20107.01).

3.2.

Selected Features of the Proposed Method

In this study, 1998 features were calculated, and feature selection was performed using the forward step-wise method. The eGFR estimation required an average of 14 features. The least-cases scenario involved seven features, and the most-cases scenario involved 31 features. Table 5 indicates the number of features required for convergence. Table 7 shows the features selected for estimating eGFR (RSVM + Large kidney + Affine & Int). The histograms calculated from the entire mask were selected from 7 out of 31 features. However, most of the histograms were created for each A-TLCO layer and 24 of the 31 features. Figure 8 shows which layer was selected in the TLCO layer-by-layer histogram. At this time, layer 1 is the outermost layer, and layer 12 is the innermost layer. Layers 4 to 6 are the boundaries between the cortex and the medulla. Although the features are selected from the histogram of each layer, relatively many bins of the layers corresponding to the inside and outside are selected. This may indicate that changes in the cortex and medulla are effective. In addition, many features of the image from which noise was removed by applying a median filter were selected.

Table 7

Features selected for eGFR estimation.

Fig. 8

A-TLCO histogram features selected by eGFR estimation.

Table 7 shows the number of features required for convergence of the eGFR. Table 8 shows the features selected for estimating the eGFR slope (RSVM + Large kidney + Affine & Int). The mean, kurtosis, skewness, median, and histogram calculated from the entire mask were selected from 2 of 14 features. However, most of the histograms were created for each A-TLCO layer and 12 of the 14 features. Figure 9 shows which layer was selected in the TLCO layer-by-layer histogram. As few as 12 features are selected, and relatively many bins of the layers corresponding to the inside and outside are selected. In addition, many features of the image from which noise was removed by applying a median filter were selected.

Table 8

Features selected for eGFR slope estimation.

Fig. 9

A-TLCO histogram features selected by slope estimation.

4.1.

Performance of the Proposed Method

In this study, renal function was estimated using water images of the Dixon method. The eGFR accuracy was 0.38 when estimated by multiple regression analysis using the average value for each layer. On the other hand, when a normal kidney with a large area was used as the renal region and affine transformation and displacement field registration were used for registration, the RMSE was 11.99 and the correlation coefficient was 0.83, which is a large improvement. Large kidneys tended to have better RMSE and correlation coefficients than normal kidneys. For comparison, features were calculated and evaluated using the TLCO manually extracted by a specialist. The TLCO method has an RMSE of 15.07 and a correlation coefficient of 0.69. Better results were obtained with the A-TLCO method than with the TLCO method. It is thought that the reason for this is that the value provided by the TLCO method varies because the renal region is manually divided into layers and dividing small kidneys, such as atrophic kidneys, into 12 layers is difficult. The results obtained by GoogLeNet had an RMSE of 21 and a correlation coefficient of 0.23. Using GoogLeNet + SVM did not provide good results, giving an RMSE of 21 and a correlation coefficient of $-0.01$. It became clear that estimation by simple deep learning was difficult. This probably indicates the small number of data used for learning as well as the difficult estimation of information that specialists cannot recognize from images, using deep learning. However, using the A-TLCO method, we found that eGFR could be estimated with high accuracy from water images of the Dixon method.

A similar experiment was performed on the eGFR slope. The eGFR slope was calculated using a normal kidney with a large area as a renal region, and affine transformation and displacement field registration were used for registration. The RMSE was 4.8, and the correlation coefficient was 0.5. In the estimation of the eGFR slope, a correlation coefficient of 0.5 was also obtained using R-RF. However, the estimated eGFR slope tended to have a significantly lower correlation coefficient than the eGFR estimation. The TLCO manually extracted by a specialist was also evaluated using the same method. The RMSE was 4.82, and the correlation coefficient was 0.45. Better results were obtained with the A-TLCO method. The results obtained by GoogLeNet had an RMSE of 5.35 and a correlation coefficient of $-0.09$. Using GoogLeNet+SVM did not provide good results, giving an RMSE of 5.25 and a correlation coefficient of 0.11. It was again clear that estimation by simple deep learning was difficult. However, it was found that the prognosis of renal function can be estimated to some extent using the A-TLCO method, even using water images of the Dixon method. In order to improve the accuracy of eGFR slope estimation, it is necessary to consider different MRI images, such as the T2* map generated by BOLD MRI.

4.2.

Adequacy for Clinical Application of Selected Features of the Proposed Method

Nephrologists have empirically predicted renal function from renal morphology. In diagnosing a case of renal dysfunction of unknown course, acute kidney injury and chronic kidney disease are diagnosed from blood tests and an imaging study including MRI, and an examination/treatment policy is determined. For example, the discrepancy between the predicted eGFR from the images and the actual eGFR is known to be large in the case of acute kidney injury, while in the case of chronic kidney injury, the discrepancy is small. In addition, MRI is used as a reference for predicting renal function recovery by treating patients with acute kidney injury. However, nephrologists empirically predict renal function based on the size of the kidney, surface irregularities, and the clarity of the boundary between the cortex and medulla, and quantitative evaluation is difficult. On the other hand, this study is expected to allow quantitative assessment and reproducibility to diagnostic imaging by MRI. The proposed method is a framework for quantifying the state of the kidney and can be applied to various types of MRI, such as T2* mapping. Therefore, the proposed method is expected to become a new imaging method for evaluating various physiological and pathological aspects of the kidney.