1.

Introduction

Preservation of cavernous nerves during radical prostatectomy for prostate cancer is critical for preserving sexual function after surgery. These nerves are at risk of injury during dissection and removal of a cancerous prostate gland because of the close proximity of the nerves to the prostate surface (Fig. 1 ). Their microscopic nature also makes it difficult to predict the true course and location of these nerves from one patient to another. These observations may explain in part the wide variability in reported potency rates (9 to 86%) following prostate cancer surgery.¹ Therefore, any technology capable of providing improved identification, imaging, and visualization of the cavernous nerves during prostate cancer surgery would aid the preservation of the nerves and improve postoperative sexual potency.

Fig. 1

(a) Cross sectional diagram of the human prostate showing the location of the neurovascular bundles and their close proximity to the prostate surface. The dotted line indicates the route of dissection between the prostatic capsule and the neurovascular bundle. (b) Image of human prostate during surgery. Arrows indicate the surgical dissection plane, and the dashed line indicates the position of the periprostatic neurovascular bundle under a superficial layer of fascia.

OCT is a noninvasive optical imaging technique that can be used to perform high-resolution, cross sectional in vivo and in situ imaging of microstructures in biological tissues.² OCT imaging of cavernous nerves in rat and human prostate has recently been demonstrated.^{3, 4, 5, 6} However, improvements in the quality of the OCT images for identification of the cavernous nerves are necessary before clinical use.

For the present work, OCT images were acquired in vivo using a clinical endoscopic OCT system (Imalux, Cleveland, Ohio) based on an all single-mode fiber common-path interferometer-based scanning system (Optiphase, Van Nuys, California). An 8-Fr ( $2.6\text{-}\mathrm{mm}$ -OD) probe was used with the OCT system. The system is capable of acquiring real-time images at $200\times 200\text{pixels}$ with $11\text{-}\mu \mathrm{m}$ axial and $25\text{-}\mu \mathrm{m}$ lateral resolutions in tissue.

The following study describes a step-by-step approach that employs three complementary image processing algorithms (Fig. 2 ) for improving identification and imaging of the cavernous nerves during OCT of the prostate gland. In previous work, a segmentation approach was successfully used to identify the cavernous nerves.⁷ However, it has proven challenging to image deeper prostate tissues with OCT. Therefore, the segmentation system in the left branch of Fig. 2 is augmented by the denoising and edge detection systems in the right branch of Fig. 2. This edge detection system is later shown to improve OCT imaging of deeper prostate tissue structures.

Fig. 2

Flow chart describing a step-by-step application of complementary image processing algorithms for OCT of the prostate nerves.

In the left branch of Fig. 2, 2-D OCT images of rat prostate are segmented to differentiate the cavernous nerves from the prostate gland. It should be noted that ultrasound image segmentation of the prostate, which allows clinicians to design an accurate brachytherapy treatment plan for prostate cancer, has been previously reported.⁸ Various alternative segmentation approaches have also recently been applied in retinal OCT imaging.^{9, 10, 11, 12, 13, 14, 15, 16} However, large irregular voids in prostate OCT images require a segmentation approach different than that used for segmentation of the more regular structure of retinal layers. Therefore, to detect cavernous nerves, three image features are employed: a Gabor filter, Daubechies wavelet, and Laws filter. The Gabor feature is applied with different standard deviations in the $x$ and $y$ directions. In the Daubechies wavelet feature, an eight-tap Daubechies orthonormal wavelet is implemented, and the low-pass subband is chosen as the filtered image. Finally, Laws feature extraction is applied to the images. The features are segmented using a nearest-neighbor classifier. N-ary morphological postprocessing is used to remove small voids.

As a next step to improve OCT imaging of the prostate gland, wavelet denoising is applied. Recently, wavelet techniques have been employed successfully in speckle noise reduction for MRI, ultrasound, and OCT images.^{17, 18, 19} A locally adaptive denoising algorithm is applied before edge detection to reduce speckle noise in OCT images of the prostate.²⁰ The denoising algorithm is illustrated using the dual-tree complex wavelet transform. After wavelet denoising, an edge detection algorithm based on thresholding and spatial first-order differentiation is implemented to provide deeper imaging of the prostate gland. This algorithm addresses one of the main limitations in OCT imaging of the prostate tissue, which is the inability to image deep into the prostate. Currently, OCT is limited to an image depth of approximately $1\mathrm{mm}$ in most opaque soft tissues. In the following sections, a segmentation approach is first described, followed by details of denoising and edge detection approaches.

2.

Segmentation System

The input image is first processed to form three feature images. The prostate image is then segmented into nerve, prostate, and background classes using a $k$ -nearest neighbors classifier and the three feature images. Finally, N-ary morphology is used for postprocessing. The generation of the feature images are first described, followed by descriptions of the classifier and postprocessing.

3.

Wavelet Shrinkage Denoising

Wavelet shrinkage is denoising by shrinking (nonlinear soft thresholding) in the wavelet transform domain. The observed image $X$ is modeled as an uncorrupted image $S$ and multiplicative speckle noise $N$ . On a logarithmic scale, speckle is converted to additive noise $X=S+N$ . The wavelet shrinkage denoising algorithm requires the following four-step procedure,²⁰

4.

Edge Detection System

A block diagram of the edge detection system is shown in Fig. 3 . After luminance thresholding on the input image $f(x,y)$ , a first-order spatial differentiator of orthogonal gradient is performed to produce the differential image $g(x,y)$ with accentuated spatial amplitude changes. Morphological postprocessing is then used to accentuate edges.

Fig. 3

Edge detection system block diagram.

5.

Combined Algorithms

Figure 2 shows the order of the combined algorithms. The segmentation algorithm was applied to differentiate the cavernous nerves from the prostate gland. This algorithm is independent of the denoising process. However, the edge detection algorithm to provide deeper imaging of the prostate gland based on thresholding and spatial first-order differentiation is dependent on the denoising process. In other words, edges are sensitive to the noise. First, the input image was denoised, then the edge detection was implemented. With a noisy image, threshold selection becomes a tradeoff between missing valid edges and creating noise-induced false edges.

The algorithms were executed on a Core 2 Duo, $1.86\text{-}\mathrm{GHz}$ desktop personal computer. There were two parallel processes of Fig. 2, $8\text{-}\mathrm{s}$ denoising and edge detection and $10\text{-}\mathrm{s}$ segmentation. The total time for the combined processing algorithms was $10\mathrm{s}$ .

6.

Results

The unprocessed time-domain (TD)-OCT images of the cavernous nerves at different orientations (longitudinal, oblique, and cross sectional) along the surface of the rat prostate are shown in Figs. 4, 4, 4 . Histologic sections of the cavernous nerves were previously processed for comparison.²⁰

Fig. 4

OCT images of the rat cavernous nerve: (a) and (b) longitudinal section; (c) and (d) cross section; (e) and (f) oblique section. (a), (c), and (e) show before; and (b), (d), and (f) show after denoising.

Figures 4, 4, 4 show the images after denoising using CDWT. The global signal-to-noise ratio (SNR) is calculated as

Figures 5, 5, 5 show the same OCT images of Figs. 4, 4, 4 after segmentation. The cavernous nerves could be differentiated from the prostate gland using the segmentation algorithm. The error rate was calculated by: $\text{error}=\left(\text{number}\text{of}\text{error}\text{pixels}\right)∕\left(\text{number}\text{of}\text{total}\text{pixels}\right)$ , where $\left(\text{number}\text{of}\text{error}\text{pixels}\right)=(\text{number}\text{of}\text{false}\text{positives}+\text{number}\text{of}\text{false}\text{negatives})$ . The overall error rate for the segmentation of the nerves was 0.058 with a standard deviation of 0.019, indicating the robustness of our technique. The error rate was measured as a mean of error measurements for three different sample images at different orientations (longitudinal, cross sectional, and oblique). A different image was used for training. The error rate was determined by comparing manually segmented images to the automatically segmented images of the nerves. These manually segmented images of the cavernous nerves were previously created according to histologic correlation with OCT images.²⁰ Figures 5, 5, 5 combine edge detection of the denoised images and the segmentation results. Manual segmentation of the prostate gland was implemented to calculate performance of the edge detection algorithm. The overall error rate for the segmentation of the prostate gland was 0.076 with a standard deviation of 0.022.

Fig. 5

OCT images of the rat cavernous nerve: (a) and (b) longitudinal section; (c) and (d) cross section; (e) and (f) oblique section. (a), (c), and (e) show segmented and (b), (d), and (f) show edge detected images.

7.

Discussion

The proposed edge detection approach was successful in accentuating prostate structures deeper in the tissue, and the cavernous nerves could be differentiated from the prostate gland using the segmentation algorithm. The glandular structure of the prostate could be observed to a depth of approximately $1.6\mathrm{mm}$ in Figs. 5, 5, 5 in comparison with an approximately $1\text{-}\mathrm{mm}$ depth in the unprocessed OCT images in Figs. 4, 4, 4. Overall, the edge detection technique enhanced structures deeper in the prostate gland, and the proposed image segmentation algorithm performed well for identification of the cavernous nerves in the prostate.

It should also be noted that the rat model used in this study represents an idealized version of the prostate anatomy, because the cavernous nerve lies on the surface of the prostate, and is therefore directly visible. However, in human anatomy, there can be an intervening layer of fascia (Fig. 1) between the OCT probe and the nerves, making identification more difficult. Since one major limitation of OCT is its superficial imaging depth in opaque tissues, an important advantage of these image processing algorithms is that the final OCT image should be able to provide deeper imaging in the tissue and locate the cavernous nerve when it lies at various depths beneath periprostatic tissues.

8.

Conclusion

The segmentation technique is applied to differentiate cavernous nerves from the prostate gland in rat prostate. The wavelet shrinkage denoising technique using a dual-tree complex wavelet transform is used for speckle noise reduction, and by using edge detection, deeper imaging of the prostate gland is accomplished. These algorithms for image segmentation, denoising, and edge detection of the prostate may be of direct benefit for implementation in clinical endoscopic OCT systems currently being studied for use in laparoscopic and robotic nerve-sparing prostate cancer surgery.