We developed machine learning methods to identify fibrolipidic and fibrocalcific A-lines in intravascular optical coherence tomography (IVOCT) images using a comprehensive set of handcrafted features. We incorporated features developed in previous studies (e.g., optical attenuation and A-line peaks). In addition, we included vascular lumen morphology and three-dimensional (3-D) digital edge and texture features. Classification methods were developed using expansive datasets (∼7000 images), consisting of both clinical in-vivo images and an ex-vivo dataset, which was validated using 3-D cryo-imaging/histology. Conditional random field was used to perform 3-D classification noise cleaning of classification results. We tested various multiclass approaches, classifiers, and feature selection schemes and found that a three-class support vector machine with minimal-redundancy-maximal-relevance feature selection gave the best performance. We found that inclusion of our morphological and 3-D features improved overall classification accuracy. On a held-out test set consisting of >1700 images, we obtained an overall accuracy of 81.58%, with the following (sensitivity/specificity) for each class: other (81.43/89.59), fibrolipidic (94.48/87.32), and fibrocalcific (74.82/95.28). The en-face views of classification results showed that automated classification easily captured the preponderance of a disease segment (e.g., a calcified segment had large regions of fibrocalcific classifications). Finally, we demonstrated proof-of-concept for streamlining A-line classification output with existing fibrolipidic and fibrocalcific boundary segmentation methods, to enable fully automated plaque quantification. The results suggest that our classification approach is a viable step toward fully automated IVOCT plaque classification and segmentation for live-time treatment planning and for offline assessment of drug and biologic therapeutics.
Intravascular optical coherence tomography (IVOCT) has significant advantages for characterizing vascular lesions compared with other imaging modalities used by interventional cardiologists. This is significant as percutaneous coronary interventions (PCIs) are performed in the United States every year.1 Although clinicians routinely use angiography for treatment planning to describe the vessel lumen, angiography does not provide specific information regarding vascular wall composition, except in the case of severely calcified lesions.2 Intravascular ultrasound (IVUS) can identify the location of coronary calcium but cannot assess calcium thickness because the signal reflects from the calcium/soft tissue interface, which results in an acoustic shadow following the calcium front surface.3 IVOCT, however, provides the location and often the thickness of a calcification.4 In addition, IVOCT has a superior resolution (axial: 15 to ; lateral: 20 to ) compared with IVUS (axial: 150 to ; lateral: 200 to ).5,6 Thus, IVOCT is the only intravascular imaging modality with the resolution and contrast sufficient for identifying thin-cap fibroatheromas,7–9 which are significant prognosticators of plaque rupture and subsequent myocardial infarction.10–13 Previous studies have suggested that IVOCT can successfully identify coronary fibrous, lipid, calcium, macrophage deposition, and plaque erosion.14–17
We focus on the application of stent treatment planning with IVOCT and determine the “fibrocalcific” and “fibrolipidic” portions of a lesion. Hard, fibrocalcific coronary plaques can restrict stent expansion, necessitating the use of prestent lesion preparation (atherectomy, cutting balloon, ultrahigh balloon pressures, etc.).18 Softer, fibrolipidic lesions may not preclude stent expansion. However, when confronting such cases, appropriate determination of stent sizing and its geographic landing site is necessary to ensure that the edge of a stent does not land in a fibrolipidic region prone to vascular dissection or stent rupture. A challenge for live-time stent treatment planning, however, is that IVOCT generates image frames in a single pullback scan. Visual determination of fibrolipidic and fibrocalcific portions of a lesion requires careful consideration of image features by an expert, making detailed manual analysis of every image frame time-consuming, labor-intensive, and certainly inappropriate for live-time use in the clinic.5 This creates a need for automated analysis of IVOCT images.
Previous studies have achieved promising results for automated IVOCT plaque classification. Several groups, including our own, have demonstrated that the “optical attenuation” coefficient can help identify coronary plaque types.19–22 Others have shown that A-line peaks can be used to discriminate normal from diseased vessel regions.23,24 Automated “pixel-wise” plaque classification was achieved by several groups. Ughi et al.25 incorporated optical attenuation and texture features with a random forest classifier to identify fibrous, calcified, and lipid pixels within an IVOCT image. In addition, this group classified mature and immature neointimal tissue-covering stents.26 Athanasiou et al.27 utilized intensity and texture features to identify calcium, lipid, fibrous, and mixed tissues using k-means clustering and random forest classifier. Zhou et al.28 used edge detection to identify adventitia and calcium tissue, and they further refined the calcium boundary using a level-set approach. In addition, they used wavelet and intensity features with a random forest classifier to find lipid and mixed tissues. In addition to voxel-wise segmentation, some groups have performed A-line plaque classification. Rico-Jimenez et al.24 used a linear discriminant analysis algorithm to automatically identify A-lines containing fibrotic and fibrolipid plaques. Our group compared using a convolutional neural network (CNN) with a fully connected artificial neural network to classify fibrolipidic, fibrocalcific, and other A-lines.29 Abdolmanafi et al.30 assessed using deep learning methods to perform tissue classification23 and to identify pathological formations in IVOCT images. Moreover, Gessert et al.31 used CNNs to identify IVOCT frames that contain plaque. Although all of the aforementioned studies were promising, some limitations exist. (1) In pixel-wise, semantic plaque classification, there is a conundrum in labeling and in evaluation, as the back border is not seen in lipidous plaques (Fig. 1). (2) It is unclear that all reports use a sufficiently large base of support in the image to capture a priori knowledge of vessel wall morphology (e.g., fibrous tissue followed by a lipidous region). (3) Very few studies consider histological verification. (4) Many studies have a limited number of images, limiting the ability to generalize.
We contend that A-line classification approaches have advantages compared with pixel-wise approaches. We consider the A-line a more natural analysis element for IVOCT images, as it can capture important plaque characteristics (e.g., the rapid fall off of the signal in lipid or the sharp edges of a calcified lesion). Moreover, when labeling pixels, analysts have no way of knowing the extent of lipid from an IVOCT image (Fig. 1). This is because the IVOCT signal quickly decays in areas containing highly attenuating tissues, such that the detected IVOCT signal is only visible within tens to a few hundreds of microns beyond the plaque border. Hence, the depth of a lipidous plaque and the pixels to label in a lipidous plaque are quite arbitrary. In addition, for calcium and lipid plaques, pixel-wise segmentation approaches can sometimes produce granular plaque boundaries, which do not coincide with what is observed histologically for such tissues. With A-line plaque classification, we can identify plaque angular boundaries and feed results into previously developed plaque boundaries segmentation algorithms.7,8,32,33 These algorithms implement border smoothness and connectivity constraints to better identify plaque boundaries.
This study includes several important contributions. First, and most important, we evaluated an A-line classification algorithm that uses the most comprehensive set of handcrafted features () to date. We included many features previously used for IVOCT plaque analysis23,26–28 and introduced innovative types, including vessel-lumen morphology, which evaluates the irregular lumen shape often present in calcifications, and three-dimensional (3-D) digital edge and texture, which can encompass information from nearby IVOCT image frames. Second, we created two large, labeled in-vivo and ex-vivo datasets ( pullbacks, image frames), which include a very wide variety of lesions, giving an opportunity to better test generalizability of algorithms. The unique ex-vivo dataset is labeled with the aid of carefully registered, cryo-imaging data,34 reducing potential labeling bias introduced by human expert labeling of in-vivo images. The in-vivo dataset ensures that our work translates to real patient data. Third, we selectively tested the impact of our intuitive feature groups (e.g., morphological and 3-D features) on performance, aiding intuitive interpretation. Fourth, we evaluated classifiers having a common or different set of features for each plaque type. Fifth, we applied a conditional random field classification noise cleaning method previously reported by us29 on en-face views, creating visualizations for rapid evaluation of classification quality. Finally, we demonstrate proof of concept to use A-line classification to automatically initiate plaque segmentations, which enables fully automated fibrolipidic and fibrocalcific boundary assessments.
A-Line Plaque Classification Algorithm
Our A-line plaque classification approach involved several steps. We preprocessed the input IVOCT images to reduce speckle, to correct for catheter optics, and to determine an appropriate tissue region for processing ( Supplementary Material, A.1). Then, we extracted numerous handcrafted features, including vascular lumen morphology, optical physics, A-line peaks, and two-dimensional (2-D)/3-D digital edge and texture. We classified A-lines as fibrolipidic (from the lumen, fibrous tissue followed by lipid), fibrocalcific, or “other.” We used feature selection to choose the best features among the large number of proposed features. We compared two distinct multiclass classification strategies: a three-class (3C) approach and dual binary (DB) classifiers (fibrolipidic and fibrocalcific). The 3C approach looked for fibrolipidic and fibrocalcific A-lines simultaneously; the DB classifiers looked for A-lines from the two groups separately and then merged the results. In both cases, conditional random field postprocessing was used to reduce classification noise.
Classification Feature Extraction
Lumen morphological features were implemented to describe the size and shape of the lumen, which tends to correlate with hemodynamic shear stress and potentially contributes to atherosclerotic plaque formation.35,36 Features were determined from the anatomical view. We computed the “change in lumen area” () along the longitudinal () direction, by comparing the current slice lumen area with each of its adjacent slices. The absolute lumen area of the current frame was not included because it depended more on the vessel under study than on disease. “Lumen eccentricity” was used to assess deviations from a circular shape, as often found in the presence of plaque, compared with very circular normal regions (Fig. 2).
We calculated the “frame lumen eccentricity” (FLE), which is a single value describing lumen eccentricity in the current frame. We also computed the “change in frame lumen eccentricity” across adjacent slices () by comparing the current frame FLE with the FLE of each of its adjacent slices. Individual “A-line lumen eccentricity” (ALE) was determined in an effort to localize A-lines containing plaque (Fig. 3). To calculate ALE, we determined the center of mass and area of the segmented lumen. We then created a perfectly concentric circle having the same area as the lumen, centered at the center of mass. We calculated the signed distance between each A-line lumen boundary point and the corresponding circle boundary along the radius from the center of mass, giving a vector of ALE values.
Because we noticed that large eccentric calcifications often resulted in a flat lumen in our dataset, we computed a best-fit line for lumen boundary points and extracted corresponding features (sum of squared residuals, goodness of fit, and line magnitude) based on the residuals between the points and the fitted line (Fig. 4). We also compared the values for the same A-line position for each frame with the values obtained from its adjacent slices. These features are summarized in Table S1 in the Supplementary Material.
The last lumen morphological feature is “ lumen curvature” (Fig. 5). Although listed in the morphological group, this feature could also be considered in the optical group. A region from an IVOCT image can appear dim, not because of the presence of plaque but because the IVOCT imaging beam is obliquely incident on the tissue surface resulting in light being reflected away from the transducer.37 As a surrogate, we computed the lumen slope in images, where a slope of 0 corresponds to a vertical lumen boundary giving a perpendicular incidence of the beam on the tissue. Large positive or negative slopes corresponded to an oblique angle of incidence.
The optical features implemented were inspired by the physics of IVOCT image acquisition and the tissue optical properties characteristic of each plaque type.38 Optical attenuation was included (Fig. 6) because this feature has been well established in a number of previous studies.19,20,22,26 We used the method developed by Vermeer et al.21 to determine the optical attenuation. Using a discretized differential equation, they solved the optical attenuation at each pixel as
Intensity features for each A-line were extracted from linear data, as we expect tissues with heavily calcified or lipid tissues to have low image intensities.14,16 For intensity and optical attenuation features, we calculated the mean, median, standard deviation (SD), minimum, maximum, skewness, and kurtosis values over the following ranges of tissue, observed immediately after the lumen boundary: 0.0 to 0.25 mm; 0.25 to 0.5 mm; 0.5 to 0.75 mm; 0.75 to 1.0 mm, as well as over the total 1.0 mm.
Additional optical features were included. For each A-line, we obtained the distance between the catheter and the lumen boundary because larger distances could contribute to diminished image intensity. The distance to background intensity describes the depth at which the signal reached a value for background, as determined using Otsu’s method.39 We determined this value by starting at the far depth of the image and searching toward the lumen to find the first pixel above this threshold. This feature should detect highly attenuating fibrolipidic regions, which are consistent with a rapid decay of the IVOCT signal. All optical features used are summarized in Table S2 in the Supplementary Material.
We analyzed prominent peaks in each IVOCT A-line (Fig. 7), which tend to correlate with plaque type.23,24 Briefly, A-lines corresponding to normal tissue have a prominent first and second peak, and the signal tends to monotonically decrease. Fibrocalcific A-lines have several prominent peaks, and the signal does not decay monotonically. Finally, fibrolipidic A-lines tend to have only one prominent peak near the A-line boundary, and then the signal rapidly decays to baseline intensity. We low-pass filtered each A-line using a zero-phase shift finite impulse response filter with order 50 and cut-off frequencies of 0.1 and 0.01 of units (pixel size ). The cut-off frequency affects smoothing of the signal. To identify local peaks, we used the “findpeaks” function in Matlab with a minimum intensity difference required of between a peak and its neighbors. Several features were determined, as described in Table S3 in the Supplementary Material.
Digital edge and texture
We used 2-D digital edge and texture features because calcified plaques tend to have visually pronounced edges and a more prominently textured appearance than lipidous plaques.14,16 Texture features came from the Leung–Malik filter bank,40 which is a multiscale, multiorientation filter bank containing 48 filters, including multiorientation edge, Gaussian, and Laplacian of Gaussian filters. We also used the Schmid filter bank,41 which consists of 13 rotationally invariant filters. Finally, edge gradients were added (i.e., Laplacian, horizontal and vertical Sobel, and horizontal and vertical Prewitt). The parameters for each kernel are presented in Table S4 in the Supplementary Material. To process these 2-D features, all helically sampled A-lines over a large vessel segment were merged to form a large 2-D () array, for example, consisting of about for a 100-frame segment. The array was pixel-shifted and only the first 1.0 mm of tissue in the wall was filtered. We referred to this image as . We then convolved this image with our various 2-D digital edge/texture filters to derive our filter responses. We computed the mean, median, SD, minimum, maximum, skewness, kurtosis, and energy values over the following ranges of tissue: 0.0 to 0.25 mm; 0.25 to 0.5 mm; 0.5 to 0.75 mm; 0.75 to 1.0 mm, as well as over the total 1.0 mm.
We implemented the 3-D digital edge and texture features (i.e., Laplacian, Laplacian of Gaussian, Sobel, and Prewitt). To do this, we reshaped our image to form a 3-D matrix and convolved this image with the various 3-D digital/edge filters from our filter bank. Following 3-D convolution, the matrix was reshaped back to the image, and feature values were calculated in the same manner as for the 2-D edge/texture features described above. The descriptions are summarized in Table S5 in the Supplementary Material.
Classification Model Development
We evaluated two multiclass classification approaches. First, we created 3C classification approaches, in which A-lines were categorized into one of three classes: fibrolipidic, fibrocalcific, or other. We evaluated a random forest classifier (3C-RF) and a one-versus-all support vector machine classifier (3C-SVM) with a Gaussian radial basis function kernel. Hyperparameters for the proposed approaches were empirically determined using one fold of the training data. We examined sensitivity, specificity, and overall accuracy metrics since these were ultimately what we wanted to optimize and they could be used for both random forest and SVM. For the 3C-RF approach, we used a maximum number of 20 splits and 30 learners. For the 3C-SVM approach, box constraint was set to 1.0, and a heuristic search approach in Matlab was used to automatically determine the manual kernel scale. For both approaches, feature selection was an important step to reduce overfitting.
As an alternative multiclass classification approach, we used DB classifiers, where one classifier was used to classify A-lines as fibrolipidic versus other, and the other classifier was used to categorize A-lines as fibrocalcific versus other. For the DB approach, compared with the 3C approach, feature selection was carried out independently for each of the two classifiers. Moreover, for each DB classifier, the training data were skewed, such that half of the training data belonged to the plaque type of interest, whereas the remaining half of the data came from the other two categories. With the DB approach, there is a chance that both the fibrolipidic and fibrocalcific classifiers will claim ownership of an A-line. This situation is addressed as described below following classification noise cleaning. For each DB classifier, we tried using both random forest (DB-RF) and support vector machine (DB-SVM), with the same parameters as for the 3C approach and compared the classification results.
Feature selection was used to identify the best features. For the 3C approaches, we used the minimal-redundancy-maximal-relevance (mRMR)42 method. For the DB approach, we tried both mRMR and the binary Wilcoxon method.43 In both cases, feature selection was used to rank order the features (from best to worst) in each group. Next, we performed forward feature selection, starting with an empty subset and gradually adding features (starting with the highest ranked features) to train the classifiers. We plotted overall classification accuracy and individual sensitivities for each class. Our “stopping rule” for adding features was when the values of these classification performance metrics started to plateau. Finally, we combined the best features from each group to form the total feature subset. Note that each classifier we developed had its own unique set of features.
Classification features were normalized to ensure that each one had an equal effect on the classifier performance. Features were normalized by the mean and SD of the range of values of each feature, as observed over the training set. This is also referred to as the statistical z-score. We compared normalizing each feature across the range of values observed within each individual vessel with normalizing across the range of values observed over all vessels. When normalizing across the ranges observed in all vessels, we normalized in-vivo features and ex-vivo features separately.
Classification noise cleaning
We used a modified version of the conditional random fields (CRF) approach, which was presented in Ref. 44, to perform classification noise cleaning on an en-face view of our data. The advantage of a fully connected CRF is that it establishes pairwise potentials on all pairs of pixels in an image. This means it performs classification noise cleaning by considering how all pairs of pixels in an image interact with one another. This is logical in our case, as we expect plaque types to have volumetric continuity. We implemented CRF on the en-face view of the classification results, as this representation shows the classification results for all A-lines within an IVOCT pullback segment in a single 2-D view. In our implementation, we only made use of the smoothness kernel, as we were not conveying the intensity information to the CRF algorithm. Thus, our implementation of CRF involved optimization of the following equation:Supplementary Material.
In this implementation, we only had to optimize two parameters corresponding to the smoothness kernel: and . We did so using a grid search approach with empirically determined values for each parameter. We set the values for as 0.25:0.25:1. Because controlled the degree of smoothness in both the and directions of the en-face image, each direction had to be optimized separately. In the direction, we set the values for as 0.5:0.5:1.5. Similarly, in the direction, we set the range of values for as 0.5:0.5:1.5. These parameters were optimized by applying the classifier to all data from one of the folds in our training set. Because we used the efficient inference in fully connected CRF implementation,44 the algorithm used an iterative mean field approximation to the CRF distribution. In our implementation, we simply set the number of iterations to 100.
For the 3C results, we only had to use CRF processing once. For the DB approaches, we applied CRF processing to the fibrolipidic classification results and the fibrocalcific classification results separately. Finally, the cleaned fibrolipidic and fibrocalcific results were merged. In the event that an A-line was classified as both fibrolipidic and fibrocalcific, we classified the A-line based on the plaque type with the maximum probability from the initial classification.
Creation of Training and Validation Database
We used both a clinical in-vivo and an ex-vivo dataset, validated by 3-D cryo-imaging/histology. In-vivo IVOCT pullbacks were obtained from the University Hospitals Cleveland Medical Center imaging library.45 All pullbacks were imaged before the interventional procedure. IVOCT images were collected using a frequency domain OCT system (Ilumien Optis; St. Jude Medical, St. Paul, Minnesota). A total of 6556 image frames across 49 patient pullbacks were analyzed. Each polar image consisted of either 448 or 496 A-lines, 968 pixels along each A-line, and 16 bits of grayscale data per pixel. The ex-vivo dataset consisted of 440 IVOCT images from 10 segments, taken from 10 pullbacks. Acquisition of this dataset has been described previously.34 IVOCT image acquisition for the ex-vivo data was performed using a LightLab C7-XR Fourier Domain IVOCT Intravascular Imaging System (C7-XR; LightLab Imaging Inc., Massachusetts). In this dataset, each polar image consisted of either 1008 or 504 A-lines, 984 pixels along each A-line, and 16 bits of grayscale data per pixel. Each ex-vivo IVOCT image in our dataset had a corresponding registered cryo-image. Registration between IVOCT pullbacks and corresponding cryo-image volumes was performed using the registration method developed by Prabhu et al.34 Labeling of both in-vivo and ex-vivo images is described in the Supplementary Material (A.4).
A-line Plaque Classification Experiments
Classification training using in-vivo and ex-vivo datasets
Classification algorithms were developed using fivefold cross validation across training data. A held-out test set was also used to evaluate classification performances. Training data consisted of 4819 in-vivo images across 41 patients and 440 ex-vivo images across 10 cadaver coronary arteries. The held-out test set consisted of 1737 in-vivo images across 22 segments from eight patients. Each training fold was divided to ensure that all data from a particular patient were entirely contained within one fold. No patient data from the held-out test set were contained in any of the training folds, and vice versa.
We performed a fivefold cross validation across the training data. For each fold, 45,000 A-lines were used for training, and 30,000 A-lines were used for testing. The training and testing A-lines were balanced for each class (1/3 fibrolipidic, 1/3 fibrocalcific, and 1/3 other for the 3C approaches; 1/2 fibrolipidic/fibrocalcific and 1/2 equally sampled from the remaining two classes for the DB approaches). We report confusion matrices containing the classification sensitivities across all folds on the training data. In addition, we evaluated results on a held-out dataset. In this case, we trained each classifier using 45,000 A-lines that were equally sampled across all five training folds, and applied the classifier to all the data in the held-out (imbalanced) test set and computed performance metrics.
To assess the added value of our morphological and 3-D edge/texture features, we evaluate overall classification accuracy both with and without inclusion of these features. We assess performance of the best 3C and binary classifier on the held-out test set. In addition, because fibrocalcific angle assessment is critical for PCI treatment planning, we perform Bland–Altman analysis between the automated and the ground-truth methods for fibrocalcific plaque angle using our best classifier.
Transition of classification output to segmentation algorithms
We demonstrated proof of concept for streamlining A-line plaque classification results into existing semiautomated algorithms for fibrocalcific and fibrolipidic boundary segmentation.7,32,33 We selected one fibrocalcific case and one fibrolipidic case from our held-out in-vivo test set to assess implementation. For each vessel, we only display one plaque type. We input lumen, guidewire, back boundary segmentation, and angular plaque boundaries that were automatically determined by our classifier. The resulting plaque boundaries and volume renderings across the images were assessed.
Classification Software Implementation
The computation software was developed on a Dell Precision T7610 with an Intel® Xeon® CPU E5-2650 v2 with a 2.60-GHz processor. The computer operated 64-bit Windows 7 Professional with 128 GB of RAM and an NVIDA Quadro K4000 video card. Image analysis and classification software was performed in Matlab® 2016B (Mathworks, Natick, Massachusetts). The Matlab code for the CRF approach described in Sec. 2.2 can be found in the following GitHub repository: https://github.com/johannesu/meanfield-matlab/.
A-Line Classification Results
We determined the optimal features and tested the performance of each one of the 10 combinations of classifiers and feature selection schemes that we developed. Feature selection was performed to find the optimal number of features for each classifier. The results are shown for the 3C-SVM classifier in Fig. S2 in the Supplementary Material. After feature selection, we performed fivefold cross validation and testing on our held-out dataset. The best 3C approach was the 3C-SVM. Using fivefold cross validation on the training data, the 3C-SVM classifier gave an overall classification accuracy of 74.60%, with percentage sensitivity/specificity for each class: other (76.90/88.77), fibrolipidic (88.33/83.78), and fibrocalcific (69.31/93.10). Moreover, on the held-out dataset the 3C-SVM classifier gave an overall classification accuracy of 76.52%, with percentage sensitivity/specificity for each class: other (81.43/89.59), fibrolipidic (94.48/87.32), and fibrocalcific (74.82/95.28). The confusion matrices and the table containing these results are shown in Fig. S3 and Table S1 in the Supplementary Material.
For the DB approaches, we found that the DB-SVM classifier using mRMR feature selection worked best for fibrolipidic classification and that the DB-SVM classifier using Wilcoxon feature selection worked best for fibrocalcific classification. When performing fivefold cross validation of the training data, the DB-SVM classifier using mRMR feature selection for fibrolipidic classification gave an overall classification accuracy of 80.14%, with percentage sensitivity/specificity for each class: other (80.59/79.68) and fibrolipidic (79.68/80.59). Moreover, on the held-out dataset, the DB-SVM classifier had an overall classification accuracy of 85.46%, with percentage sensitivity/specificity for each class: other (80.31/90.60) and fibrolipidic (90.60/80.31). Moreover, when performing fivefold cross validation of the training data, the DB-SVM classifier using Wilcoxon feature selection for fibrocalcific classification gave an overall classification accuracy of 80.03%, with percentage sensitivity/specificity for each class: other (87.90/72.03) and fibrocalcific (72.09/87.90). Moreover, on the held-out dataset, the DB-SVM classifier had an overall classification accuracy of 81.99%, with percent sensitivity/specificity) for each class: other (90.59/73.38) and fibrocalcific (73.88/90.59). The confusion matrices and the tables containing these results are shown in Figs. S4 and S5 and Tables S2 and S3 in the Supplementary Material.
In addition, we assessed the utility of including morphological and 3-D features for A-line plaque assessment (Fig. 8). We used the best 3C, binary fibrocalcific, and fibrolipidic classifiers and observed the effect of removing our 3-D and morphological features on overall classification accuracy on our held-out test set. For all classifiers, we found a reduction in overall classification accuracy following the removal of both 3-D and morphological features, suggesting the utility of incorporating both feature groups for fibrolipidic and fibrocalcific plaque classification.
Classification Noise Cleaning
We integrated the output from the best-performing classifiers from Secs. 4.1 and 4.2 into the CRF classification noise cleaning approach described in Sec. 2.2 (Fig. 9). For the 3C-SVM model, grid searching suggested the following optimal parameters: as 0.75, in the direction as 1.5, and in the direction as 1.5. For the fibrocalcific DB-SVM using Wilcoxon feature selection, we set , in the direction, and in the direction as 0.5, 1.5, and 1.5, respectively. For the fibrolipidic DB-SVM using mRMR feature selection, we set , in the direction, and in the direction as 0.5, 1.5, and 1.5, respectively.
For the 3C-SVM approach, CRF cleaning improved the overall classification accuracy to 81.58%, with percentage sensitivity/specificity for each class: other (81.43/89.59), fibrolipidic (94.48/87.32), and fibrocalcific (74.82/95.28). Improvement in classification performance after CRF cleaning was also observed for the DB classifiers. For the fibrolipidic classifier, CRF cleaning improved the overall classification accuracy to 85.52%, with percentage sensitivity/specificity for each class: other (84.20/94.58) and fibrolipidic (94.48/84.20). For the fibrocalcific classifier, CRF cleaning improved the overall classification accuracy to 91.25%, with percentage sensitivity/specificity for each class: other (95.71/76.26) and fibrocalcific (76.26/95.71). These results are also displayed in Fig. S6 and Table S9 in the Supplementary Material.
We finally merged the cleaned results for the fibrolipidic and fibrocalcific classifiers to perform an apples-to-apples comparison of their performance with that from the 3C classifier (Fig. 10). After merging, for the DB approaches, we achieved an overall accuracy of 80.19%, with percentage sensitivity/specificity for each class: other (79.90/89.31), fibrolipidic (93.63/85.03), and fibrocalcific (73.52/96.19). Thus, the performances of the two approaches were nearly identical, with a slight edge given to the 3C-SVM classifier. The fact that the two approaches obtained similar performances yields credence to the same underlying feature set from which the two approaches were developed.
Successful and Challenging Classification Cases
Overall, our classification approach worked very well in the majority of cases. We show results from the cleaned 3C-SVM classification approach. Fibrocalcific cases with sharp borders, a textured appearance, and very eccentric lumens were easily classified. Figure 11(a) shows a fibrocalcific lesion with a textured appearance and visually sharp luminal and abluminal boundaries. Figure 11(b) shows a fibrocalcific lesion with a very eccentric lumen, textured appearance, but only luminal boundaries that were visibly sharp. In both cases, however, our classifier gave near-perfect assessments of the lesion angular boundaries. Fibrolipidic cases in which the signal initially peaks and then decreases were also correctly classified. Figure 11(c) shows a fibrolipidic lesion with a sharp peak near the luminal boundary followed by a rapid decay of the signal. Figure 12(d) shows a near-360-deg fibrolipidic lesion with a characteristic initial peak followed by a precipitous decay of the signal. In both fibrolipidic lesions, a near-perfect agreement of plaque angular boundaries between manual and automated assessments was observed.
Because fibrocalcific angle assessment is critical for clinical treatment planning, we assessed agreement across all frames in our held-out test set between manual and automated analysis (Fig. 12). Bland–Altman analysis of the fibrocalcific angle demonstrated a very strong correlation between the expert and automated assessments. On average, the difference between the automated and manual assessments was only . Only 141 of the 1737 images assessed in the held-out test set were outside the 95% confidence interval. No apparent bias was observed regarding plaque angle, which suggested that we can accurately predict the angle of both small and large calcified lesions.
Across all the 1737 images in our held-out test set, we only confronted a few challenging cases (Fig. 13). However, we believe that such cases would be difficult for an analyst as well. Figure 14(a) shows an image that contains both fibrocalcific and fibrolipidic A-lines. The classifier correctly identifies both lesions. However, one part of the lesion referred to as fibrocalcific by the analysts is designated as other by the classifier. This is because a clear back border is not visible in this region, and it is possible that this region of the plaque is mixed. Figure 14(b) shows a lesion that was judged to be fibrolipidic by the analysts. However, the classifier designates one portion of the lesion as fibrocalcific because this region does not decay clearly, and the fibrocalcific portion does appear to have some textured appearance and a slightly eccentric shape following the initial signal peak. In both cases, it is possible that this case was either misclassified by our experts or contained some combination of both lipidic and calcific plaque.
Transition of Classification Output to Segmentation Methods
We assessed the feasibility of streamlining the A-line classification results into existing segmentation methods for one representative fibrolipidic case and one representative fibrocalcific case from our held-out test set (Fig. 14). We used the cleaned results from the 3C-SVM classifier. Fibrocalcific and fibrolipidic boundaries are shown in Figs. 14(1) and 14(2), respectively. The results demonstrated the feasibility of combining classification output with existing segmentation methods for fully automated plaque classification and segmentation.
In this study, we used numerous handcrafted features and expansive clinical in-vivo and ex-vivo validation datasets (totaling images) to develop automated methods to identify fibrolipidic and fibrocalcific IVOCT A-lines. We incorporated features developed in previous studies that assessed tissue optical properties,21,22 A-line peaks,23,24 and 2-D texture features.25–27,46 In addition, we developed features based on vascular lumen morphology and 2-D/3-D digital edge and texture. We investigated two strategies for multiclass classification: a 3C approach and DB classifiers. The 3C approaches tried to simultaneously detect fibrolipidic and fibrocalcific A-lines. The DB approaches tried to detect fibrolipidic and fibrocalcific A-lines independently and then merge results together. Our 3C approach using a multiclass SVM classifier and mRMR feature selection with a common set of features for fibrolipidic and fibrocalcific A-lines had the best performance. Following classification noise cleaning (Fig. 13), this approach had an overall classification accuracy of 81.58% for all A-lines in a held-out test set ( images), with percentage sensitivity/specificity for each class: other (81.43/89.59), fibrolipidic (94.48/87.32), and fibrocalcific (74.82/95.28). In addition, fibrocalcific angle detection between manual and automated assessments over our entire test set as assessed using Bland–Altman analysis was (Fig. 12). Again, this indicates strong performance of our automated approach. In addition, we demonstrated proof of concept for streamlining classification output with existing fibrolipidic and fibrocalcific boundary segmentation methods (Fig. 14). Fully automated IVOCT plaque characterization could facilitate live-time stent treatment planning and offline assessment of drug and biologic therapeutics. The automated results could be input into finite element models to predict the clinical outcomes of using stents in certain lesions.47,48 Thus, from A-line segmentation results, we have an opportunity to give recommendations on proper stent sizing, location, and potential plaque modifications. Moreover, the software provides an opportunity for offline assessment of drug and biologic therapeutics. That is, the software developed could help facilitate analysis of plaque progression/regression in offline assessments of patient data.
We believe that our approach of combining A-line classification with segmentation methods is advantageous compared with pixel-wise classification. This is because IVOCT data are acquired as a series of one-dimensional A-lines, which makes A-line analysis a natural means of assessing IVOCT images. Moreover, pixel-based classification is severely compromised in areas containing highly attenuating tissue types (Fig. 1). Furthermore, pixel-based classification can produce granular borders, which do not reflect the plaque boundaries that are histologically observed. Using A-line classification with 3-D classification noise cleaning, we locate the general location of plaque. Then, we apply plaque boundary segmentation to the location to find the exact location and to quantify plaque burden. Not only does this approach produce biologically observe plaque boundaries, but it also mimics how IVOCT image assessment was performed by our analysts.
Incorporation of morphological and 3-D features into our classification approach improved the overall classification accuracy (Fig. 11). Morphological features were inspired by previous studies, which suggest that vascular lumen morphology changes coincides with plaque incidence.35,36 Digital 2-D/3-D edge features accounted for the difference in edge strength observed between the fibrolipidic and fibrocalcific boundaries.14,16 The 3-D features accounted for the biologically observed, volumetric continuity of blood vessels and associated plaques. The improvement in overall classification accuracy by incorporating these feature groups suggests that future studies would benefit by considering such features for IVOCT plaque classification. Several groups, including our own, are investigating deep learning approaches for IVOCT image analysis and plaque classification.23,29,30 Hybrid methods, which combine such deep learning results with the handcrafted features used in our study, could further improve classification performance.
We varied our multiclass classification strategy, choice of classifier, and feature selection algorithms to optimize the classification performance (Figs. S2–S6 in the Supplementary Material). The decision to vary such parameters was based on previous studies43 that suggested that optimizing these parameters affects classification performance. In the end, we merged our DB classifiers to obtain a multiclass result, which we compared with the results of 3C classification (Fig. 10). We found nearly identical performance using these approaches. The fact that all approaches performed similarly is evidence of the robustness of our feature set.
Classification noise cleaning, which combines CRF and morphological image-processing operations, smoothed classification results and improved classification true positive rates by (Fig. S6 in the Supplementary Material). This processing was performed on the en-face () view of data, which is an innovative way of viewing volumetric classification results for a vessel in a single 2-D view. Thus, our method ensured that classification results made sense from a biological standpoint, considering that plaques demonstrate volumetric continuity. We used a CRF model44 for classification noise cleaning. This algorithm considers the classification probabilities but also enables incorporation of intensity information into determining cleaned classification results. In our approach, we only considered classification probabilities. Future studies can investigate whether incorporating en-face intensity data improves classification performance. In previous work, we generated intensity-based IVOCT en-face views49 to facilitate stent segmentation. Such views could be incorporated into the CRF model.
Classification models were developed using both in-vivo and ex-vivo training data. This is advantageous because it instills confidence in our training data and annotation labels. Labels in ex-vivo data were obtained from accurately registered, cryo-imaging data, facilitating accurate labeling. The use of in-vivo data ensures that our classification approach is applicable to real patient data. Although our classification approach performed well in the majority of cases, we noticed several challenging cases in our held-out test set (Fig. 14). We believe that these challenging cases were potentially mixed lesions. Expansion of our ex-vivo training database could help with classifying such cases.
A number of software implementation optimizations were necessary while developing our classification approach. This study involved machine learning using very expansive datasets. In total, A-lines were analyzed, each as an individual sample. For each A-line, we extracted a total of features before feature selection, most of which were of data type double. Thus, although the computer we used had 128 GB of memory, we needed to carry out frequent memory clearing when performing computations on the entire data set. For feature extraction, it was imperative to preallocate data matrices and perform computation on portions of data at a time. Matrix reshaping and concatenation to generate our image and 3-D matrix facilitated the computational extraction of digital edge and texture features. When performing classification model development and fivefold cross-validation training, it was necessary to subsample data. These computational limitations could be circumvented in the future using computers with vast amounts of memory. Finally, although the classification time on precomputed features for the 3C-SVM classifier was , classification feature extraction is a rate-limiting step. We could expedite feature extraction by experimenting with further feature reduction to see whether we can use fewer features to generate comparable classification results. Improving our hardware resources by increasing computer memory and using more powerful graphics-processing units can further reduce computation time.
IVOCT is a promising intravascular imaging modality for assessing coronary plaque burden to perform stent treatment planning. Our algorithm automatically classifies fibrocalcific and fibrolipidic IVOCT A-lines, using a very large number of handcrafted features and is developed using expansive in-vivo and ex-vivo, cryo-image validated datasets. Use of our 3-D and lumen morphology features improved classification accuracy, suggesting the importance of these intuitive features. A-line classification results were streamlined to initiate plaque segmentation. The study results suggest that our classification approach is a viable step toward fully automated IVOCT plaque classification and quantification for both live-time treatment planning and offline assessment of drug and biologic therapeutics.
This project was supported by the National Heart, Lung, and Blood Institute through the U.S. National Institutes of Health (NIH) Grant Nos. R21HL108263, R01HL143484, and R01HL114406, by NIH Construction Grant No. C06 RR12463, and the Choose Ohio First Scholarship. These grants were attained via collaboration between Case Western Reserve University and University Hospitals of Cleveland. The veracity guarantor, Chaitanya Kolluru, affirms to the best of his knowledge that all aspects of this paper are accurate.