User-guided segmentation for volumetric retinal optical coherence tomography images

. Despite the existence of automatic segmentation techniques, trained graders still rely on manual segmentation to provide retinal layers and features from clinical optical coherence tomography (OCT) images for accurate measurements. To bridge the gap between this time-consuming need of manual segmentation and currently available automatic segmentation techniques, this paper proposes a user-guided segmentation method to perform the segmentation of retinal layers and features in OCT images. With this method, by inter-actively navigating three-dimensional (3-D) OCT images, the user first manually defines user-defined (or sketched) lines at regions where the retinal layers appear very irregular for which the automatic segmentation method often fails to provide satisfactory results. The algorithm is then guided by these sketched lines to trace the entire 3-D retinal layer and anatomical features by the use of novel layer and edge detectors that are based on robust likelihood estimation. The layer and edge boundaries are finally obtained to achieve segmentation. Segmentation of retinal layers in mouse and human OCT images demonstrates the reliability and efficiency of the proposed user-guided segmentation method. © The Authors


Introduction
Optical coherence tomography (OCT) is a noninvasive imaging technology that is capable of capturing micrometer-resolution three-dimensional (3-D) images of optical scattering media, including biological tissue. 1,2 It is now widely used as a research and clinical tool for the investigation of retinal diseases in both humans and animal models. [3][4][5][6][7] The measurement of geometrical dimensions (i.e., biometry) of physiological and pathological retinal layers and features is one of the primary functions of OCT imaging. Images of changes in layered retinal structures aid in the diagnosis, treatment, and management of many blinding diseases. [8][9][10] In this regard, the algorithms for correctly segmenting the retinal layers or features play an important role. However, OCT images, particularly those acquired from patients and animal models with eye disease, are sometimes of low contrast (i.e., noisy) and have irregular features. For these simple reasons, it remains a challenging task to achieve satisfactory segmentation of the retinal layers in OCT images using automated segmentation techniques.
The retina is composed of distinct physiological layers, including the nerve fiber layer (NFL), ganglion cell layer (GCL), inner plexiform layer (IPL), outer plexiform layer (OPL), external limiting membrane (ELM), and inner/outer segments (IS/OS). Because the different cell compositions of each layer have distinct optical scattering properties, these layered tissues usually appear as varied intensities in the OCT images with a signal transition from one layer to another which appears as boundary line (interface) (see Fig. 1). It should be pointed out that the boundary between the GCL and IPL in human eye appears weak. The GCL in mouse eye is thin, which is sometimes difficult for OCT to resolve. Therefore, most segmentation studies treat the GCL and IPL as one layer, giving the NFL, GCL+IPL, OPL, ELM, and IS/OS layers in the segmentation. [11][12][13][14][15][16][17][18][19][20][21][22][23][24][25][26][27] Over the last decade, various segmentation techniques were developed to segment the images from one-dimensional to 3-D based on algorithms from simple intensity threshold to complex machine learning algorithms. 11,12 The majority of these segmentation algorithms are based on analyzing the variations of OCT signal intensities along the A-scans (depth scan) or in two-dimensional (2-D) cross-sectional images (B-scan). [13][14][15] It is possible to achieve fast segmentation based on this gradient information; 16 however, these algorithms are prone to errors when detecting abnormal edge information or images that appear noisy. To reduce the erroneous edge and layer detection when the image appears noisy (either globally or locally), two approaches have been proposed. One is to improve the robustness of the algorithms to detect the edges or layers. 17,18 Another approach is to connect the individual edge detectors together using a 2-D curve or 3-D mesh. 19,20 By analyzing the information presented in the neighborhood, such as curvature, or applying a global optimization algorithm, the resulting boundaries of the retinal layers become sufficiently smooth and the chance of erroneous edge detection is reduced. 21 Usually, the parameters of the edge detection algorithms are assigned empirically or are trained through the use of other manually segmented subsets. 17,18,22 To achieve high accuracy in the quantification of geometrical dimensions (e.g., thickness) of retinal layers, two-step strategies, in which different algorithms are utilized in each step, are often utilized to perform segmentation. 23 The key to the success of segmenting retinal layers is to correctly detect the layers or their boundary edges in the OCT images. Most previously proposed segmentation techniques can meet this goal when it is treated as a local optimization problem. It is still a challenging task to achieve global optimization.
Despite the fact that a number of prior robust algorithms that are capable of achieving high-level automatic segmentation have been reported, [22][23][24][25][26][27] many clinical practices still continue to use manual segmentation as the technique of choice for OCT image segmentation. Reluctance to accept the fully automatic approach may be due to the concerns about its insufficient reliability in cases where the target layers and features may differ from the norm and because of insufficient accuracy in delineating the desired physiological and pathological features in a noisy background.
A similar situation occurs in the arena of medical imaging, including magnetic resonance imaging (MRI). In order to increase the reliability and confidence of feature segmentation in MRI, there has been increased interest in developing userguided segmentation instead of fully automated algorithms. 28,29 The advantage of user-guided segmentation algorithms is that they have the potential to efficiently combine the features of human and computer to achieve the segmentation of medical images. Humans are remarkably sensitive and accurate in recognizing tissue patterns to achieve global optimization, while the computer can achieve local optimization with high speed. The user-guided segmentation system can, therefore, be reliable and efficient. In the system, a user typically assigns some key features such as points or curves, and then the computer finishes the rest of the segmentation task in delineating or searching for the tissue of interest guided by the user-defined key features. Determining how to assign and interpolate the key features and how to trace entire layers based on these key features are the main tasks in designing user-guided segmentation algorithms.
In this paper, a user-guided algorithm is proposed and designed for the segmentation of retinal layers in OCT images that were acquired using a custom-built OCT system reported in Refs. 30 to 35. The user-guided algorithm, the double line model (DLM) (described in detail in Sec. 2.2), is based on a 2-D robust layer and edge detectors, and was used to extract the shapes of an embryo heart for analyzing heart development. [36][37][38] DLM detects local layer patterns and edges in places where the images appear noisy or distorted due to pathological reasons. However, DLM does not work well in terms of global optimization problems; for example, it often fails to trace the layered structures at regions with irregularly layered structures and/or with low contrast and holes within the retinal layers. To mitigate this problem, we propose that before the algorithm begins to work on the 3-D dataset, a user first delineates a number of user-defined sketch lines on irregular or otherwise difficult features to correctly guide the DLM in tracing the retinal layers. The proposed user-guided segmentation method is reliable and can segment noisy and irregular OCT images of the retina.

Method
In this section, a 3-D OCT image of the posterior segment of the mouse eye is used as an example to illustrate the application of the user-guided segmentation algorithm. Because it is difficult for DLM to deal with a layer whose thickness is thinner than 3 pixels, as a preprocessing step we select an appropriate scale to adjust the OCT image size in the depth direction so that the thinnest target layer is thicker than 3 pixels.
The steps of the user-guided segmentation of retinal layers are shown in In the following subsections, segmentation of the retinal layers is illustrated in detail.

Assigning the User-Defined Lines
The first step of segmentation is to assign a number of userdefined lines in the 3-D OCT images [see Figs. 2(b) and 3]. To assign the user-defined lines, an interactive interface is developed for users to navigate the 3-D OCT images frame by frame. The XYZ coordination is defined as shown in Fig. 3: the X axis is the abscissa of the B-scan image, the Y axis is the sequence number of the image, and the Z axis is the ordinate of the B-scan image (depth). The user selects a 2-D B-frame (cross-sectional  image) that may be at any location within the 3-D image and then rotates the image around its local Z axis by using a computer mouse in order to observe anatomical features appearing within it. On the computer screen, the user-defined lines are then assigned in the cross-sectional image.
The user-defined lines are classified into static lines and adaptive lines. A static line is a line that does not change its attributes, such as the shape, orientation, and position, when using DLM to detect and trace the retinal layers during segmentation, i.e., it is a fixed line [see Figs. 3(b) and 3(c)]. An adaptive line is a line that is assigned to a retinal layer that would change its attributes in order to arrive at a state that optimizes its position according to the local features, i.e., it has an adaptive nature during segmentation [ Fig. 3

Double Line Model
The DLMs assigned along the adaptive line are used to detect the retinal layer and edge in the 2-D sectional image [see Fig. 3 (a)]. The DLM is a robust layer and edge detector that can detect local layers and edges within the 2-D image. 36 It is developed from the adaptive model based on the theory of robust signal detection and estimation. 39,40 It has two parallel lines that divide the surrounding region into internal and external regions (Fig. 4). By computing the likelihood between the DLM and the internal and external regions of the local image, the DLM can efficiently identify the target tissue from the surrounding tissue.
When the DLM is used to detect local layers or edges within the tissue of interest [see Fig. 4 When the DLM is used to detect retinal layers, it is designated as the layer-DLM. In OCT images, the signal strength at the interface between two layers is usually higher than that of its neighboring regions. The intensities within one retinal layer also sometimes appear different from those of other layers. Accordingly, the features of the retinal layer-DLM are divided into a high-intensity foreground region R mh , a low-intensity foreground region R ml , a right background region R br , and a left background region R bl [see Fig. 4(c)]. The foreground regions, I fh and I fl , are estimated using the median intensity within R fh and R fl , and the background regions, I br and I bl , are similarly estimated within R br and R bl . When the layer-DLM matches the target layer, the intensity of the DLM would be similar to that of the foreground of the target tissue but different from the background. Thus, the likelihood L l between the layer-DLM at the state M and the target tissue layer in a cross-sectional image is computed by where IðxÞ is the intensity of the pixel located at position x in the B-scan image and AðÞ is the area of the region that represents the sum of the pixels within the region. For example, AðR mi Þ is the number of pixels in the region R mi . The first summed item computes the difference between internal region intensity and background, and the second and third summed items compute the similarity between the internal region intensity and the foreground. The local retinal layer is found by shifting, scaling, and rotating the layer-DLM to maximize L l .  When the DLM is used to detect the edges, it is designated as the edge-DLM [see Fig. 4(d)]. The regions of the edge-DLM are internal region R mi , the left background region R bl , and the right background region R br . The likelihood L e between the edge-DLM and the real edge in an image is computed by where I f is the foreground that is the median intensity within the internal region R mi . Other variables are the same as described in Eq. (1). The edge likelihood computes the difference between the internal region intensity and the background as well as the similarity between internal region intensities and foregrounds. Because the edge should be narrow, the width of the edge-DLM is fixed at 4 pixels. The local edge in the B-scan image is found by shifting and rotating the edge-DLM to maximize L e .

Tracing the Layer Using Layer-DLM
Once the user assigns a number of static lines and adaptive lines in the 3-D images [see Fig. 2 Fig. 5(a), where each physiological interface is enclosed by the two lines of all DLMs.
Upon scrutinizing Fig. 5(a), it is evident that segmentation using the layer-DLM needs some improvement, particularly in the interface boundaries that appear noisy, e.g., NFL. It is, therefore, necessary to perform an additional adjustment to the boundary lines of the segmented retinal layers. This task is performed by the edge-DLM as described below.

Adjusting Retinal Layer Boundaries Using
Edge-DLM Certain retinal boundary lines, such as the interface between the vitreous humor and the NFL, can appear quite noisy in the original image [see Fig. 5(b)]. In order to adjust the boundary lines of the NFL, the original image is preprocessed to enhance the retinal layers. First, the intensity is smoothed using a 3 × 3 pixel mean filter to reduce noise in the original image. Then, an adaptive threshold is applied to process the image. If the intensity of a pixel is stronger than the median intensity in a small local window, it is considered as the foreground and a new intensity value of 128 is assigned. Otherwise, it is considered background and assigned a value of 0. The small window size is approximately twice that of the target tissue layer as defined by the layer-DLM. Finally, the intensity is smoothed using a 3 × 3 pixel mean filter to further reduce noise. This process significantly enhances the retinal layer boundary information [see

Thickness of the Segmented Layers
After performing segmentation of the retinal layers, it is a straightforward process to provide quantification of the thickness of the retinal layers. Most prior studies compute the thickness by using the distance between two cross-points (point A and B in Fig. 6) of the A-scan line and layer boundaries. However, when the A-scan line is not perpendicular to the boundary of the layer, the thickness computed using this traditional method is not accurate, because it would be thicker than the real thickness. In an effort to improve the accuracy of layer thickness quantification, we computed the thickness by considering the local normal to the surface (see Fig. 6). To compute the correct local 3-D normal to surface, the scale of the segmented layers is adjusted to the same scale in X, Y, Z directions. Then, the local normal N a on the top surface and the local normal N b on the bottom surface are computed. Next, we compute a normal N c , that is, a unit vector with its direction the same as N a þ N b . The thickness T is the projection of AB on N c and is computed by T ¼ jABj • N c , which is the length of BC. Finally, the thickness map is obtained by computing the thickness at all positions on the XY plane.

Results
The segmentation algorithm was applied to 3-D datasets resulting from optical micro-angiography (OMAG) imaging of the posterior segments of eyes in both mouse models and humans. OMAG (Refs. 41 to 45) is a technological extension to OCT, capable of providing volumetric microstructural image identical to that from OCT, and the blood flow image in parallel. The OMAG imaging system is based on a Fourier domain OCT system, the details of which, including the system parameters, can be found in the previous literature. 33,35 Each 3-D image used in this study consisted of 400 2-D image sequences (B-scans or B-frames) and 512 A-scans in each B-scan. An example B-scan image acquired from a mouse eye is shown in Fig. 7, where the microstructure and the corresponding blood flow images were obtained in parallel with the OMAG imaging technique. The segmentation algorithm was carried out on the structural image [ Fig. 7(a)] to obtain retinal layer boundary surfaces [ Fig. 7(c)], from which the quantitative biometry, e.g., thickness map, was obtained. The resulting segmentation results were equally applicable to the corresponding OMAG 3-D blood flow image [ Fig. 7(b)], so that the depth-resolved microvascular network within the retinal layers can be revealed [ Fig. 7(d) Fig. 8(b)]. Although the system used to capture the mouse retinal images was of low spatial resolution (∼7 μm), the appearance of the en face view of the IS/OS layer [ Fig. 8(e)] most likely corresponds to the distribution of murine photoreceptors. The user-guided segmentation is robust and works especially well at locations where the boundaries of the retinal layers are of low contrast. In particular, the interface between GCL and IPL is of low contrast, and most prior segmentation methods have had difficulty obtaining satisfactory segmentation of the IPL layer. However, using the layer-DLM to trace the IPL, the IPL was successfully obtained in our study. Finally, we can also observe the microvasculature network from within the   Because OMAG provides the microstructure and microcirculation images in parallel, we can apply the segmentation results obtained from the microstructure images (Fig. 8) to the corresponding blood flow images in order to achieve depth-resolved vascular mapping. The results are shown in Fig. 9, where the vascular network of each layer, NFL, IPL/INL, and OPL, is clearly delineated. From the segmentation results shown in Fig. 8, it is a straightforward process to provide quantitative thickness maps of NFL and NFL+GCL+IPL. Figure 10 provides these quantitative results. Note that in the quantification of mouse retina, the values have accounted for the scaling factor that was used to generate images with an appropriate size to facilitate the segmentation procedures as described above. In the mouse NFL, the vessels and nerve fiber bundles coexist. Using the blood vessel information provided by the OMAG blood flow images, we are able to separate the vessels [red color in Fig. 10(a)] from nerve fiber bundles [blue color in Fig. 10(a)]. To radially quantify the retinal layer thickness variation from the ONH, the mean thickness of each region demarcated by concentric circles (100 pixels or ∼390 μm in width) from the center of ONH to the peripheral retina was computed and provided in Figs. 10(b) and 10(c) for the NFL and NFL+GCL+IPL layers, respectively. The thicknesses of the NFL and NFL+GCL+IPL layers gradually become reduced proceeding from the ONH to the periphery, which is consistent with previously published observations. 22,23

Segmentation of the Retinal Layers in 3-D OCT Image from Normal Human Subject
The OMAG images of a healthy human retina were successfully segmented into the NFL, GCL+IPL, IPL/INL, OPL, and IS/OS layers [see Fig. 11(a)]. The corresponding projection maps of the NFL, GCL+IPL, IPL/INL, OPL, and IS/OS layers are presented in Figs. 11(b) to 11(f), respectively. The GCL layer was obtained from the region between the bottom boundary of the NFL and the top boundary of the IPL. The NFL, GCL, and IPL layers merge into one common layer near the fovea [see insert of Fig. 11(a)]. Similar to the information obtained from the mouse retina, distinct physiological features within the human retina can be readily appreciated. The orientation and alignment of the nerve fiber bundles within the NFL [ Fig. 11(b)] can be visualized, as well as the photoreceptor distribution within the IS/OS layer [ Fig. 11(f)], which is of high density within the fovea region and gradually reduces radially outward. Such information can be critically important in understanding the diagnosis, treatment, and prognosis of patients affected by retinal diseases, such as age-related macular degeneration (AMD) and diabetic retinopathy. With the OMAG blood flow dataset, we also successfully segmented the microvascular networks located in three different layers within the retina, the GCL+IPL [ Fig. 12(a)], IPL/INL [ Fig. 12(b)], and OPL [ Fig. 12(c)]. In contrast to the mouse retina, there are relatively large supplying arterioles and collecting venules in the human retina within the GCL+IPL layer. In particular, the blood vessel network within the OPL [ Fig. 12(c)] shows simple but distinct and well-organized capillary beds in the healthy human eye. The visualization of such detailed blood vessel organization in distinct retinal layers at the true capillary level in 3-D is currently possible only with OMAG imaging technology and is also relevant to understanding the  pathophysiology, treatment, and prognosis of patients affected by glaucoma, AMD, and diabetic retinopathy. 46,47 The thickness maps of each of the human retinal layers are readily available from the 3-D dataset after segmentation using the user-guided algorithm. Figure 13 illustrates the thickness maps from the NFL, NFL+GCL+IPL, and the whole retina (from NFL to IS/OS). At the location of the fovea, the NFL cannot be identified as one single layer in the OCT image and it merges with the GCL and IPL. Thus, the NFL near the fovea is artificially thicker than the actual thickness of the NFL [see gray part in Fig. 13(a)]. The mean thickness in each of the circumferential regions is 100 pixels (∼333 μm) in width from the center of the fovea to the periphery as shown. The retinal thickness decreases radially from the fovea to the periphery for the computed layers.

Segmentation of the Retinal Layers in 3-D OCT
Images from Pathological Eye As mentioned above, segmenting the retinal layers in patient eyes in clinical practice still heavily depends on manual segmentation largely due to the challenge faced by currently available automatic segmentation algorithms. Our user-guided segmentation method was tested on the images acquired from various patient eyes scanned by both the OCT/OMAG system and a After quantifying the segmented IS/OS layer, it is easy to calculate that the size of the exudate is 1.65 × 10 6 μm 2 and its volume is 3.9 × 10 7 μm 3 . It is interesting that there are some bright spots seen in the fluid space [arrow in Fig. 14(f)], which most likely correspond to the lipid deposits. In addition, the bright   The thicknesses of the NFL and NFL+GCL+IPL in the AMD patient eye were also computed (see Fig. 15). Compared to the thickness of the retinal layer in the healthy human eye (Fig. 13), the thicknesses of the NFL and NFL+GCL+IPL in the pathological eye become thinner on the average. All the information provided by the successful segmentation of the proposed userguided algorithm would be very useful in the analyses of the microstructural changes that would aid clinical diagnosis, treatment, and management of disease.

Discussion
To evaluate the accuracy of the proposed segmentation method, we compared the results of manual segmentation with those from our user-guided segmentation protocol. Five graders, who were experts in OCT imaging in ophthalmology, conducted manual segmentation on 100 images of 2-D B-scans, randomly selected from six 3-D scans of mouse and four 3-D scans of human eyes. The mean position of the boundary lines of the five manual segmentations was computed and considered as the correct segmentation (gold standard). The mean of the absolute errors of the user-guided segmentation and its standard deviation were computed for the mouse and human retinas, in pixels and in microns, using the software program SPSS20 (Table 1). In all layers, the absolute error is smaller than 1 pixel (∼4 μm), except for the NFL in the murine retina and the IPL/INL layer in the human retina, where it is slightly more than 1 pixel. This error would be acceptable for an OCT imaging system that typically has a spatial resolution of < 10 microns, demonstrating the robustness of the proposed user-guided segmentation. Finally, the difference between the user-guided segmentation value and the mean of the five manual segmentation values for each of the 60 mouse retina and 40 human retina images is noted to be within 2 standard deviations (95% confidence interval) for each retinal layer. In the vast majority of images, the differences between the user-guided versus manual segmentation results were <2 pixels. Together, these results confirm the accuracy of the user-guided segmentation protocol in comparison to consistent manual segmentation values in both mouse and human OCT retina images.
The current implementation of the algorithm was carried out using a computer with a 3 GHz CPU and 4G memory. The  program was coded with C++ based on the image processing library OpenCV. A drawback of the current method is that the processing time is relatively long. This is because the user is required to navigate the B-scan image through a 3-D dataset onto which the user would be able to instruct the program to correct the areas of likely erroneous segmentation. We tested the user-guided segmentation program using various 3-D eye images with 400 B-scans in each 3-D dataset captured by both the commercial OCT and OMAG/OCT. The segmentation time for an entire 3-D dataset ranged from ∼3 min (normal eyes) to ∼30 min (eyes with severe pathological signs) depending on the quality of the B-scan images. For high-quality images from normal subjects, the layer tracking is almost automatic. However, when the quality of the images from patients is low and the boundary between the retinal layers is difficult to identify, the user needs to intervene to place user-defined lines and interactively correct the erroneous segmentation, which increases the time used. Compared to pure manual segmentation (typically 3 to 4 h), the user-guided segmentation is, however, much more efficient.
By providing a number of user-defined lines around layers or features within the 3-D OCT images, the user-guided method shows reliable and satisfactory segmentation of the retinal layers in images with complex shapes, holes, and low contrast layer boundaries. Furthermore, the user can also correct an incorrect segmentation by modifying the adaptive line using the userguided segmentation method. As a drawback of assigning the user-defined lines, an artificial layer structure may be generated at the region where there is no layer, e.g., at the ONH and fovea regions. This artifact, however, can be easily corrected by interactively assigning the region where there is no known retinal layer on a 2-D en face image (see the gray ellipse region on thickness images in Figs. 10, 13, and 15).
In the future, two approaches will be considered to improve the performance of the current user-guided segmentation system. One approach is to develop an interpolating algorithm between user-defined lines so that it is possible to quickly construct the 3-D surface, saving user operation time. Another approach is to improve the automated layer tissue and edge detectors. Current robust DLMs can be migrated into an advanced computing platform, e.g., GPU, for further improvement of the overall performance of the proposed algorithm. When developed and clinically validated, we hope that the proposed semiautomatic strategy will have an impact on the current clinical practice of functional OCT imaging of the retina.

Conclusion
We have proposed a user-guided segmentation method in which the user can provide a number of defined lines around the layers and features within 3-D OCT images to guide the program to perform robust detection of the boundaries and layers of retinal layers. We have demonstrated, through implementation of the algorithms on retinal OCT images of both mouse and human eyes, that the system is reliable and efficient. The proposed approach will be useful with the current usage of OCT imaging systems because it can serve in filling the gap between current automated segmentation techniques and obtaining accurate measurements for use in clinical practice. OMAG, optical micro-angiography.