2.1.

Image Preprocessing

Like most camera-acquired images, retinal fundus images suffer from nonuniform illumination and variable visual contrast. Accordingly, preprocessing operations aim to prepare the image with better properties using shade correction and contrast enhancement. The RGB and HSI color spaces were compared, and the RGB space was found to be more suitable for HE detection. A color fundus image is provided as the input; it includes color information for each pixel in RGB color space. In this work, the image is initially resized to 640×480 pixels so that the proposed method could be applied to a variety of databases. The green channel component (G) is used for the detection of retinal structures and HEs because their contrast is greatest in this channel.

To correct for the nonuniform illumination of the image, a morphological top hat operation was used. This is based on producing a reasonable estimate of the background across G using morphological opening and closing operations with a large-enough structuring element to avoid entirely fitting within small candidate regions. To avoid unexpected bright areas at the borders and around the optic disk, alternating sequential opening and closing is used to calculate the approximation of the background E(G) as follows:⁴

To enhance the contrast of image features, we employed a fuzzy enhancement algorithm, called minimum of fuzziness.¹⁰ The selection of this algorithm is based on computational efficiency expense. For clarity and immediate relevance, some aspects are summarized below. The gray levels of G ^′ are “fuzzified” by the membership function as follows:

Fig. 2

Preprocessing and retinal structure results: (a) Green channel image, (b) result of preprocessing, (c) initial blood vessel image, (d) blood vessel image, (e) initial boundary of the optic disk, (f) localization result of the optic disk, (g) region of interest for fovea localization, and (h) localization result of the fovea.

2.2.

Blood Vessel Detection

Information provided by blood vessels can serve to localize the optic disk and fovea as well as provide vital diagnosis and follow-up information. Retinal blood vessels appear as dark line structures with different ranges of diameter, length, and orientation. Consequently, the multiscale approach is suitable to isolate features of blood vessels from the background.

Detection of linear structures using multiscale techniques have been attempted by some researchers, such as Gelman ¹¹ Osareh and Shadgar,¹² and Mendonca and Campilho.¹³ These systems work well on predefined image specifications, but they are not able to provide accurate outcomes when image quality is variable. Consequently, we propose a computationally fast method by applying a morphological closing operation to the preprocessed image, twice, with two different scales of disk-shaped structuring elements ς₂ and ς₃. On the basis of many experiments on different sizes and shapes of structural elements, we found (for the image resolution adopted) that the most suitable radii for the larger and smaller disk-shaped structure elements are 8 and 2 pixels, respectively. Then, the closed image with the smaller structuring element is subtracted from the larger one followed by thresholding to obtain an initial blood vessel image (B _v), shown in Fig. 2c, as follows:

Because of the variability of the condition and quality of images, the initial blood vessel image may include other types of dark regions. Discrimination of the true blood vessels from these other dark regions was achieved using a classification technique based on regional features, such as major axis length, minor axis length, aspect ratio (ratio of major axis length to minor axis length), area, perimeter, circularity, and eccentricity. A series of experiments on feature selection and vessel classification were carried out using a rule-based system, based on a number of quantities and logical rules. A rule-based system consists of if-then rules, which are used to formulate the conditional statements that comprise the complete knowledge base. In our work, these rules are empirically derived from the training data by a series of comparisons between many pairs of features from the feature vector and looking for functions of every two features. A number of such rules based on empirical constraint criteria were incorporated into the image-analysis and quantification program. These operations lead to an effective classifier. The final binary blood vessel image is shown in Fig. 2d, clearly showing a reduction in the number of false-positive blood vessel regions.

2.3.

Optic Disk Localization

The location and size of the optic disk are important because they can aid in the location of the fovea, which itself is important for grading the severity of retinopathy. Moreover, the bright optic disk normally exhibits similar visual features to HEs, often resulting in misclassification. Thus, optic-disk localization is important to prevent it being detected as an exudate during grading.

Several researchers have attempted the localization of the optic disk, such as can be found in Li and Chutatate,¹⁴ who used a technique of principal component analysis, which has the advantage of the top-down strategy to extract the common characteristics among training images. Niemeijer ¹⁵ used a KNN regressor to predict the distance in pixels in the image to the object of interest at any given location based on a set of features measured at that location. Because it is easy to implement and computationally fast, the method described by Sekhar ¹⁶ was followed in this work. A circular region of interest is used to find the contour of the optic disk. The region of interest is computed using the magnitude gradient of the image by morphological closing and opening operations. The image is reconstructed using morphological reconstruction, and then the gradient is calculated by subtracting the eroded image from the dilated one. The boundary of the optic disk and its center are determined by applying the circular Hough transform to the gradient image as shown in Fig. 2e. The result of optic disk localization is illustrated in Fig. 2f.

2.4.

Fovea Localization

Because the fovea contains the largest concentration of photoreceptor cells, it is the most specialized part of the retina. Determining the location of the fovea is therefore essential to grade the severity of lesions. In other words, the detection and diagnosis of lesions can provide a more precise and meaningful evaluation of risk when their spatial locations are described with reference to the location of the fovea. Several techniques have been proposed to localize the fovea using variable strategies, such as can be found in Sinthanayothin,¹⁷ who proposed a method to identify the fovea location using matching correlation together with characteristics typical of a fovea. In many retinal images, the fovea may be partially obscured by lesions or artifacts, or subject to nonuniform illumination or other distortions. Li and Chutatape¹⁸ and Welfer ¹⁹ attempt to exploit the known anatomy of the relative locations of the other retinal structures to localize the fovea center.

In this paper, the fovea is located by defining a candidate region of interest with reference to the established retinal landmarks, followed by a shape and intensity search. The two main retinal blood vessels (known as the arcade), together, can be approximated as a parabola, and in most retinal images, the fovea is located within this arcade. The position and shape of this parabola is estimated using a combination of the Hough transform and linear least-squares fitting.¹⁸

On the basis of the parabola information, the candidate region for the fovea is defined as a circle with a diameter of twice that of the optic disk (DD) along the main axis of fitted parabola and centered at a distance of 2DD from the vertex. Although the fovea is approximately equal in size to the optic disk,²⁰ we select the region of interest four times as large to ensure that all fovea pixels are within this region. The threshold value is calculated within this region in such a way that the segmented area has the same area as the optic disk. In the case of obscured foveal features due to any of the aforementioned reasons, the method may fail in finding a suitable threshold value, in which case the fovea is approximated as a circle of diameter DD at the center of the candidate region, as illustrated in Fig. 2g. The result of fovea localization is shown in Fig. 2h.

2.5.

Detection of Hard Exudates

Variations in contrast and brightness inside most retinal images make it difficult to distinguish HEs from other bright features in the image. Fortunately, most bright regions due to HEs are characterized by having distinct borders in different degrees, depending on the degree of retinopathy, while bright regions due to light reflection do not.⁴ Based on this, a novel two-step algorithm for exudate detection is proposed as follows:

2.5.1.

Coarse hard exudate segmentation

To delimit HE candidates as regions of interest, the distinctness of their borders is exploited. To achieve this, the standard deviation around each pixel is calculated to get the local variation image. To avoid detecting the high contrast blood vessels and the optic disk as HEs, they must be eliminated before applying the local variation operator. To eliminate the optic disk, its location is masked with a color equal to the average background intensity, as shown in Fig. 3a.

Fig. 3

Steps of HE detection: (a) Preprocessed image after optic disk removal, (b) result of blood vessel removal, (c) result after applying local variation, thresholding and morphological operations, (d) result after classification, (e) combination of images from (a) and (d), and (f) final fine HE segmentation.

The blood vessels are eliminated by applying a morphological closing operator with a structuring element larger than the maximal width of the blood vessels so that all vessels get closed. Experimentation with different sizes and shapes of the structural element has shown that a disk (ς₄) of radius 8 pixels is most suitable for images of the resolution adopted. The resulting image (G ₁) is shown in Fig. 3b,

Eq. 8

[TeX:] \documentclass[12pt]{minimal}\begin{document}\begin{equation} G_{\rm 1} = \psi ^{\varsigma _4 } (G_{\rm p})\, .\end{equation}\end{document} $$G_1={\psi }^{{\varsigma }_4}\left(G_{\mathrm{p}}\right).$$

The local variation operator was applied on G ₁ and the resulting image is denoted by G ₂ as follows:

Eq. 9

[TeX:] \documentclass[12pt]{minimal}\begin{document}\begin{equation} G_{\rm 2} (x) = \frac{1}{{N - 1}}\sum\limits_{i \in w(x)} {\left[ {G_1 (i) - \mu (x)} \right]^2 } \, ,\end{equation}\end{document} $$G_2\left(x\right)=\frac{1}{N-1}\sum _{i\in w\left(x\right)}{\left[G_1\left(i\right)-\mu \left(x\right)\right]}^2,$$

where x is a set of all pixels in a subwindow w(x) of N pixels and μ(x) is the mean value of G ₁(i). The selection of window size is based on the necessary balance between the most informative performance measures, namely, the sensitivity and PPV, as discussed in Sec. 4. A window size of 9×9 was found most appropriate for the images studied here. In order to refine G ₁ from artifacts and objects that have low local variation (soft exudates), an automatic Otsu thresholding was applied. Then, a morphological dilation operator with a disk-shaped structuring element (ς₅) of radius 3 pixels was applied to the thresholded image to ensure that the majority of neighboring pixels are included in the candidate regions,

Eq. 10

[TeX:] \documentclass[12pt]{minimal}\begin{document}\begin{equation} G_{\rm 3} = D^{\varsigma _5 } [{\rm TH}_{\alpha _2 } (G_{\rm 2})],\end{equation}\end{document} $$G_3=D^{{\varsigma }_5}\left[{\mathrm{TH}}_{{\alpha }_2}\left(G_2\right)\right],$$

where TH is a thresholding operator with automatic level α₂ and D is a dilation operator.

Figure 3c illustrates the result of the first phase of coarse segmentation, and although it has been successful at isolating the HEs, some artifacts remain due to residual contrasted vessels and bright regions around the retina and masked optic disk. Coarse HEs can be distinguished by many features, such as color, shape, size, and texture. We have tried to keep a reasonable and adequate number of features because misclassification probability and classifier complexity tend to increase with the number of features. A set of eight binary and gray-scale representation features are extracted for every candidate to be used as input to a rule-based classifier. The binary features are area (A), perimeter (P), circularity (C) (measure of roundness from perimeter and area), length (L), width (W), and aspect ration (R) (ratio of length to width). The gray-scale features are mean intensity (I _m) and standard deviation (S _td). Empirically, the rules listed in Table 1 were found most suitable for coarse HE discrimination. The result of coarse HE detection after applying the rule-based classifier is shown in Fig. 3d.

Table 1

Rules for the rule-based classifier to discriminate true coarse HEs. “rows” refers to the horizontal size of the image (in pixels), I avg is the average intensity of the whole image and σG is the standard deviation of the whole image.

Features	A	C = 4πA/p2	R = L/W	L	Im	Std
Rules	>25	≥0.5	<2	<(rows/3)	>1.2Iavg	<0.8σG

2.6.

Grading of Hard Exudates

In addition to size and number, a description of spatial locations of detected HEs can provide a more precise evaluation of clinical risk. Ophthalmologists usually use a polar coordinate system, centered either at the optic disk or the fovea, to estimate the grade of severity of HEs with a laborious and time-consuming process. To safely reduce the burden of manual grading, we adopt a foveal coordinate system (FCS) centered at the fovea, the center of vision, to assess the severity of HEs on the patient's vision. According to the brief description presented by Li and Chutatape¹⁸ and information provided by a medical information source,²⁰ the retinal image is divided into ten fields. as illustrated in Fig. 4. The center of the fovea is used as the center of three circles of radii 0.33DD, 1DD, and 2DD. Four coordinates are used to divide each of the two bigger circles into four fields.

Fig. 4

Fovea coordinate system for a right eye.

After detection of HEs, their spatial locations are calculated throughout the fields of FCS to enable clinicians to assess the degree of harm to the vision, where this increases with HE proximity to the inner fields and center. The result of this stage is represented by a table of the ten fields with numbers of HE pixels in each field to be then evaluated by the clinician.

3.1.

Materials

A database of color retinal images, known as DIARETDB0, has been made publicly available by Kauppi ²⁴ This consists of 130 color images of which 110 contain different signs of DR according to the evaluation of clinical experts. These were captured at a 50-deg field of view and a resolution of 1500×1152 pixels. A set of ground truths for the different types of lesions, such as the microaneurysms, hemorrhages, and HEs that are reported by experts, are provided as a description of visual appearance of the DR findings with this database. Another database by Kauppi,²⁵ known as the DIARETDB1 database, is also publicly available. This consists of 89 further images of which 38 contain HEs. These images have the same specifications of those in DIARETDB0. Images of HEs manually annotated by human experts are provided with this database, which can be used in testing the proposed method.

A set of 30 images, of which 17 images contain HEs, from the Messidor database,⁴ were also used in this paper. These images are captured at a 45-deg field of view with a size of 640×480 pixels. We have selected these 30 images among all images of the Messidor database because they are available with their ground truth. Thus, it is possible to use them for validation purposes.

3.2.

Performance Calculation

The proposed methods for blood vessel detection, fovea localization, and HE detection were trained using a set of 130 images from the DIARETDB0. Satisfactory evaluation can be achieved when the testing images are from another source than the training source. Thus, a set of 119 images (89 from the DIARETDB1 database and 30 from the Messidor database) with their ground truth images were used to test the performance of the proposed method using two criteria: pixel-based calculation and image-based classification. In the pixel-based calculation, the performance is assessed based on pixel number of exudates correctly detected.²⁶ While in the image-based classification, the algorithm is assessed based on its ability to classify an image without HEs as normal image or with HEs as a pathological image.²⁶ In pixel-based calculation, four types of pixels are considered as follows: true positives (TP), false positives (FP), false negatives (FN), and true negatives (TN). These quantities were computed for each individual image to measure the following performance measures:

3.3.

Influence of Parameters

In practical application, some error in the full automatic thresholding is inevitable. Hence, the disparity between the actual thresholding results and the ideal results (ground truth) can be used to enhance the performance of the algorithms. To do so, an intervention on some parameters used in our work has been tried to study their influence on the performance measures.

The number of true negatives that are correctly identified as non-HEs by both the grader and proposed method is the major number of image pixels; thus, the specificity and accuracy in pixel-based calculation are always near 99%. Hence, they are not very meaningful for the purposes of evaluation or comparison. Because the sensitivity and PPV are the most informative measures in the performance evaluation in the pixel-based calculation, these were used as the basis of assessment of parameter influence on performance. Several experiments have been carried out on parameters such as the window size w and the coarse and fine thresholds α₂ and α₃. Experimental results show that, as the window size is increased, PPV is increased but at the expense of the sensitivity, because the smaller HEs are more likely to be missed.

From these experimental results, we conclude that α₂ and α₃ have noticeable influence on the performance measures. Figure 5a shows the receiver operating characteristics (ROC) curves for the influence of each α₂ and α₃ separately on the overall measures. From these curves, it can be seen that, for the same rate of change, the threshold of the fine segmentation α₃ has more positive and less negative influence on the sensitivity and PPV than those of the coarse segmentation α₂. The reason for this is that a change in the coarse segmentation may significantly increase undesired false positives or decrease wanted true positives more than that in fine segmentation, and because variation of fine segmentation is limited within the delimited regions of interest (candidate regions). In other words, as the coarse segmentation delimits regions of interest, a lack in these regions can dramatically affect the number of true positives or an extra increase in these regions may cause an extra increase in false positives.

Fig. 5

ROC curves about influence of threshold parameters on performance: (a) Influence of the threshold α₂ and α₃, separately, and (b) influence of the threshold α₂ and α₃ together.

The influence of change in α₂ and α₃ together is also investigated, and the ROC performance curves are shown in Fig. 5b. The optimal percentage changes in the threshold α₂ and/or α₃ depend on the requested balance between the sensitivity and PPV, and the decision for that is up to clinicians and decided based on the diagnostic requirements.

3.4.

Importance of Hard Exudate Grading

Grading of HEs is of great importance to help doctors in making their treatment decisions. To demonstrate the importance of the grading operation, in Fig. 6 we present two examples of binary HE detection results of our proposed method and the ground truth for two retinal images superimposed on the green channel of their corresponding images, where the FCS are overlaid.

Fig. 6

Comparison between two examples in terms of HE grading: (a,b) HE distribution for example 1 by the proposed method and a clinician, respectively, and (c,d) same as for (a) and (b) but with the images of example 2.

The severity of lesions in a retina depends not only on their size and number, but also on their spatial distribution throughout the fields of the FCS. Hence, the visual inspection and the pixel-based calculation, shown in Table 2 and illustrated in Figs. 6a, 6b for example 1 and Table 3, and Figs. 6c, 6d for example 2, indicate that, although the number and size of HEs in example 1 exceed those in example 2, those in example 2 are more harmful to vision and clinically in more urgent need of treatment because of their spatial distribution and proximity to the fovea.^{2, 20}

Table 2

Distribution of HEs in example 1 on the fields of the FCS.

Table 3

Distribution of HEs in example 2 on the fields of the FCS.

3.5.

Performance Evaluation

A comparison between meaningful performance measures, namely, the sensitivity and PPV (in pixel-based calculation) and sensitivity and specificity (in image-based classification) for the proposed and some related recent works is summarized in Table 4, which shows that the proposed method detects HEs with equal sensitivity in pixel-level calculation, and competitive sensitivity and specificity in image-based classification. The main drawback of the proposed method is that the PPV falls short of that reported by some researchers. The reason for this is that our method detects more false positives than the other methods used in comparison. In spite of this shortcoming in terms of PPV, a more meaningful comparison should include the full specifications of the proposed method, including the computational efficiency and the ability to deal with images of variable quality. Although detection of faint HEs is not urgent for the purpose of treatment, it is important for early tracking of patients; however, the proposed method can detect faint HEs with better performance than the other works cited as demonstrated by sensitivity performance numbers presented in Table 4. For more information about the proposed method's performance, five samples of green channel images, their proposed method results, and their ground truths are illustrated in Fig. 7.

Fig. 7

Visual comparison of proposed method results and their ground truths for five retinal images. (a–e) Five samples for the green channel component of the original images, their results produced by the proposed method, and the clinician hand-labeled images.

Table 4

Comparison of performance measures to the proposed method and previous related works for HE detection (SE = sensitivity, SP = specificity).