Human blood contains different type of cells; however, red blood cells (RBC) or erythrocyte are the most abundant cell type. RBC contains hemoglobin, which binds to either oxygen or carbon dioxide. This allows oxygen to be transported to tissues and organs and carbon dioxide to be taken away during microcirculation. The biconcave shape of the erythrocyte is extremely important in the functionality of RBCs. It allows the membrane to have a high surface area to volume (SAV) ratio facilitating large reversible elastic deformation of the RBC while squeezing through the tiny capillaries.12.–3 According to its importance, it has drawn considerable attentions into the pathology research in the clinical relevant blood diseases. Pathological disorders can modify RBCs and lead to significant alterations in its original shape.4 The consequences of modified RBC often are observed as clinical symptoms ranges from obstruction of capillaries and restriction of blood flow to tissues to necrosis and organ critical damages.220.127.116.11.–9 Also, counting cell types in the blood sample is another important task for investigating clinical status, which can be evaluated by well-known methods such as complete blood count or RBC distribution width, which are part of cytometry field. Because an automated cell counter samples and counts so many cells, the results are reliable in most of the cases.10 However, certain abnormal cells in the blood may not be identified correctly, requiring manual review and identification of any abnormal RBCs the instrument could not categorize. This information can be very helpful regarding identifying the cause of a patient’s anemia. Abnormal increase or decrease in RBC counts as revealed in a complete RBC count may indicate that you have an underlying medical condition that calls for further evaluation.
In the case of RBC, biconcaves are a substantial type in a healthy person, but there are other RBCs types with the different percentage varying between healthy and unhealthy persons. It has been shown that the percentages of different types of RBCs will be distinct according to the RBC diseases type.10 Accordingly, it is essential to measure the percentage of each RBC type in a blood sample consists of multiple RBCs for diagnosis and drug testing subjects. Typically, the diagnosing is performed by a human expert, and it shows some drawbacks, such as time-cost consuming and inaccuracy. Generally, experts visualize the sample in the images through a microscope based on their subjective knowledge from the viewpoint of intensity, morphology, texture, and so on based features. Usually, small-scale differences in the features are overlooked by human eyes especially for the border-line diagnostic scenario.
However, the situation has changed completely by the emergence of automatic classification algorithms. These techniques have been applied to problems in biology and have shown promising results for automatic recognition and classification of various micro-organisms.11,12 Among the classification methods, the pattern recognition neural network (PRNN) has been suggested in nonlinear classification problems, such as RBC classification and counting.1314.–15 The fundamental benefit of artificial neural network (ANN) is nothing, but it does not use any mathematical model since ANN learns from data sets and identifies patterns in a sequence of input and output data without any previous assumptions about their type and interrelations. Also, ANN eliminates the drawbacks of the conventional methods by extracting the wanted information using the input data.
In conventional RBC classification problems, the experts deal with two-dimensional (2-D) erythrocyte images obtained by conventional microscopes and cameras.13,1617.18.–19 These methods generally have good performance but most of them have a significant number of features since they need to discriminate groups by utilizing 2-D features. However, in the case of RBC, which is transparent or semitransparent, we cannot take advantage of conventional intensity-based microscopes. Therefore, we believe that to obtain a satisfied level of accuracy, the classification and recognition should also take into account the three-dimensional (3-D) shape of RBCs. Among the techniques that can provide 3-D images of transparent or semitransparent cells, digital holographic microscopy (DHM) has shown promising results.2021.22.23.–24 Also, the DHM technique has been utilized in the classification of RBC using 2-D and 3-D features since DHM can provide quantitative phase image.2526.–27 In studying RBC, DHM enables the measurement of 3-D features, such as mean corpuscular volume, surface area, SAV ratio, functionality factor, sphericity index, and sphericity coefficients.28,29 Chemical parameters of MCH and MCHSD can also be obtained due to DHM. Accordingly, we believe that any automated RBC classification that can distinguish different RBC types accurately should take into accounts the benefits of DHM imaging technique.
In this study, four main types of RBC shapes, biconcave (doughnut shape), flat discs, stomatocyte, and echinospherocyte are interested RBCs for the quantitative determination of the percentage of RBC types in multiple human RBCs. The reason for differentiating doughnut-shaped and flat-disc RBCs is that it can help in the applications of separating old cells versus young cells. It has been shown that during the stages of biconcave-echinocyte transformation in so-called storage lesion, biconcave cells become flat discs (loss of ATP results in a stiffer cytoskeleton that pulls the bilayer) after a few weeks of storage in blood bank.2918.104.22.168.–34 Transfusion of these old samples might have critical consequences according to the previous studies.35,36
At first, RBCs are visualized by off-axis DHM and the quantitative phase images are reconstructed by the numerical algorithm.37,38 Then, single RBCs are extracted from images with multiple RBCs using the watershed algorithm.39 At the next step, following 2-D features of projected surface area (PSA), perimeter, radius, elongation, and PSA to perimeter rate are extracted. In this paper, we have ignored extracting 2-D features related to the inner section of RBC, unlike the previous method proposed by Refs. 25, 26, since flat-disc and echinospherocyte RBCs do not have the inner section. Also, volume, surface area, SAV ratio, average RBC thickness, sphericity index, sphericity coefficient and functionality factors, and MCH and MCH surface density (chemical properties of RBC) are extracted from single RBCs. The latter feature-set is related to the morphological and biochemical properties of RBC 3-D profile. Along with the 3-D features, two new features related to the ring section of RBC are introduced. These features add significant information to the classification model and increase the discrimination power of the classifier. Then, each feature set is fed into PRNN, separately, and the classification results are compared using 10-fold cross validation (CV) technique. Since we are involved in a classification model with nonlinear decision boundary, we have decided to use PRNN strategy. In PRNN, the training algorithm is Bayesian regulation back-propagation, which updates the weights according to Levenberg–Marquardt optimization technique and the activation function for midlevel layers is hyperbolic tangent sigmoid.
Finally, to propose the best feature set, concerning both 2-D and 3-D features, sequential forward feature selection (SFFS) is utilized here. It has been shown that the best performance of a classification model can be achieved by selecting the most informative features and remove noisy ones that are either redundant or irrelevant.40 Indeed, reducing the number of features can shorten training time, reduce the complexity of classifier, and simplify the model for interpretation goals. SFFS technique tries to select a subset of variables that best predict the data by sequentially selecting features until there is no improvement in the prediction.
In this paper, we have extracted 108 biconcave RBCs from a healthy sample stored for 1 day in the blood bank, 106 samples of stomatocyte shape from a sample with predominantly of stomato cells, 38 samples of flat-disc shape, and 71 samples of echinospherocyte shape for training and testing PRNN. Flat-disc and echinospherocyte cells are extracted from RBC samples stored in the blood bank for 40 days and 57 days, respectively. Performance comparison is evaluated by calculating misclassification rate of 10-fold CV technique. It is often claimed that leave-one-out-cross-validation (LOOCV) has higher variance than -fold CV, and that it is because the training sets in LOOCV have more overlap. This makes the estimates from different folds more dependent than in the -fold CV and, hence, increases the overall variance.41
Our experimental results demonstrate that the PRNN trained by 3-D features gives a good performance in classifying and counting RBCs in multiple human RBCs in an automated manner in comparisons with the 2-D features. In addition, we introduce the best set of features that combines 2-D and 3-D features to improve the RBCs classification accuracy. We believe that the final feature set evaluated with the presented neural network classification strategy can provide better discrimination results.
This paper is organized as follows. Section 2 explains the general scheme of the off-axis DHM to image RBCs and RBCs preparation. Section 3 explains 2-D and 3-D features extracted in this research and gives a short review about them. In Sec. 4, we will focus on designing the PRNN and counting different RBCs in multiple RBCs, automatically. The experimental results and discussions are provided in Sec. 5. Finally, the conclusion is presented in Sec. 6.
Off-Axis Digital Holographic Microscopy and RBC Preparation
Off-Axis Digital Holographic Microscopy
Figure 1 shows the off-axis DHM based on the Mach–Zehnder interferometer geometry.42 In this optical setup, light from a coherent source (HeNe laser diode source ) is split into reference and object beams by a beam splitter. The object beam is transmitted through the RBC sample and magnified by a NA microscope objective and interferes with the reference beam. The interference patterns between the diffracted object beam and the reference beam are recorded onto a CCD camera. The reconstruction of the RBC wavefront is obtained from the recorded hologram by using the numerical algorithm described in Refs. 37 and 38.
After reconstructing RBC phase images, single RBCs are extracted by the maker-controlled watershed segmentation algorithm.39 Figure 2 shows a reconstructed phase image of a sample consisting of biconcave, flat-disc, and stomatocyte shape RBCs. This image shows that in a single sample it is possible to see RBC with different morphologies.
The RBCs of healthy laboratory personnel were obtained through the Laboratoire Suisse d’ Analyse Du Dopage, CHUV and stored at 4°C during the storage period. The DHM measurements were performed several days after the blood was collected from the laboratory personnel. A total of 100 to of RBC stock solution was suspended in a high-efficiency particulate air (HEPA) buffer at 0.2% hematocrit for predominantly stomatocyte and discocyte-shaped RBCs while at a concentration of for predominantly echinocyte-shaped RBCs. A total of of the erythrocyte suspension was diluted to of the HEPA buffer and introduced into the experimental chamber, including two cover slips separated by spacers 1.2 mm thick. The cells were incubated for 30 min at a temperature of 37°C before mounting on the chamber on the DHM stage. All experiments were performed at room temperature (22°C).
After segmentation step and extracting many single RBCs, features can be extracted. We first start with 2-D features. Following features are extracted in 2-D case.
Elongation of the RBC is a measure of the ratio of width to length for oblong RBCs. It can be computed from the chain code by summing the number of each type of elements 0 to 7 and combining 0 and 4, 1 and 5, 2 and 6, and 3 and 7.16 Average and STD values of the above features are in the agreement with previously reported values (data not shown here).25,26
Since some 3-D features require thickness of the RBC, we first need to convert phase image into thickness image. Accordingly, the thickness value for each pixel of with phase value in a phase image can be expressed as28,2929
3-D features listed in Table 1 are extracted in this paper. We only give a short description of the eight features related to the morphological properties of RBC, but interested readers can refer to Refs. 28 and 29 for the detail of each feature. Regarding the calculation of three features (F9-F11), we obtained many points over the ring section of RBC by applying two techniques. First, we estimate the ring section (blue triangles in Fig. 3) by calculating the radius of a circle having the area of the projection of RBC on plane (the ring is around three-fourth of the RBC radius). Then, we update the position of each point on estimated ring (blue triangles) by finding the thickest point in a neighbor (red stars in Fig. 3). From now on we name the red stars as RPs. The single green point shows the center of RBC regarding the calculation of sphericity coefficient. In this study, we have decided to obtain 30 RP points. The reason is that the total number of points on the ring section for the stomatocyte RBCs is between 90 and 120 points according to our calculation. Also, since we are looking at a neighborhood around the blue triangles to find the point on the ring (red stars), there is high possibility of overlapping between red stars. For example, Fig. 4 shows the position of red stars and blue triangles when we increase the number of points. We can see that by increasing the number of red stars many of them overlap.
3-D features descriptions.
|3-D-F1||AT||[ is the thickness at ’th pixel]|
|3-D-F3||TVS area||Surface area of the upper view of erythrocyte calculated by dividing surface into small triangles and then the summation of each triangle’s surface area|
|3-D-F4||TVS area volume ratio|
|3-D-F5||Total surface area (SA)|
|3-D-F6||Surface area volume ratio (SAV)|
|3-D-F7||Sphericity index (SI)|
|3-D-F8||Functionality factor (FF)|
|3-D-F9||Sphericity coefficient (SP)||( and are thickness values in RBC center and ring section, respectively)|
|3-D-F10||STD of thickness in ring section||This feature measures variation in the thickness of RPs in the ring of RBC|
|3-D-F11||Upper side of the ring/lower side of ring||This feature is calculated by dividing four maximum RPs by four minimums|
|3-D-F12||Mean hemoglobin (MCH)||( is the average phase value, is the wavelength of the light source of the setup, and is a constant known as the specific refraction increment related to the protein concentration)|
|3-D-F13||MCH surface density (MCHSD)|
We believe that earlier 3-D features can distinguish among different RBCs since they are related to the 3-D profile of RBC. The statistic -test using two-sample Kolmogorov–Smirnov test revealed that some of these features are independent (data not shown). Table 2 shows average and STD values of each feature for each RBC type.
3-D features (mean±STD).
|Average thickness ()|
|TVS area ()|
|TVS area volume ratio ()|
|Total SA ()|
|SA volume ratio ()|
|Sphericity index (SI)|
|Functionality factor (FF)|
|Sphericity coefficient (SP)|
|STD of thickness in ring section|
|Upper side of the ring/lower side of ring|
Pattern Recognition Neural Network
ANNs are highly simplified mathematical models of biological neural networks having the ability to learn and provide meaningful solutions to the problems with high-level complexity and nonlinearity. The ANN approach is faster compared to its conventional techniques, robust in noisy environments, and can solve a wide range of problems. Due to the advantages, ANNs have been used in numerous applications.1314.–15 A typical ANN is presented in Fig. 5. An important application of neural networks is pattern recognition that can be implemented by using a feed-forward neural network with a specific training function and specific function in the output layer. During training, the network is trained to associate outputs with input patterns. When the network is used, it identifies the input pattern and tries to output the associated output pattern.
The information processing in the ANN strategy starts from the input layer to the hidden layer (from one hidden layer to another one if there are more than one) and from the last hidden layer to the output layer. A synaptic weight is assigned to each link to represent the relative connection strength of two nodes at both ends in predicting the input-output relationship. (the output) of any node , is given as
For output layer in PRNN, the activation function is softmax transfer function (normalized exponential) that can be interpretable as posterior probabilities for a categorical target variable. It is highly desirable for those outputs to lie between zero and one and to sum to one. The purpose of the softmax activation function is to enforce these constraints on the outputs. Let the input to each output unit be , , where is the number of classes. Then the softmax output is43
Experimental Results and Discussion
In this experiment, 108 RBCs are labeled as biconcaves, 106 RBCs labeled as stomatocytes, 38 RBCs are labeled as flat-disc, and 71 RBCs labeled as echinospherocytes. Four samples of each group are shown in Fig. 6. Simulations are all executed on a 64-bit Windows 7 computer with a 3.60-GHz Intel Core i7-4790 CPU, 8 GB RAM, and 8 cores. The performance of the classification model is assessed by utilizing 10-fold CV check. Put it simply, the data set is divided into 10 subsets, and the test is repeated 10 times. Each time, one of the 10 subsets is used as the test set and the other 9 subsets are put together to form a training set. Then, the average misclassification error across all 10 trials can show the overall misclassification error. PRNN consists of one input, output layer, and three hidden layers. Number of neurons in each hidden layer is 5, 10, and 5, respectively. Neuron numbers are obtained by trial-and-error technique. All the simulations codes are implemented in MATLAB® 2014.
Comparison Between 2-D and 3-D Features
In the case of 2-D features (Table 3), 10-fold CV revealed that the total misclassification rate is significantly high. In detail, misclassification of each group is: flat-disc 64%, stomatocyte 13.4%, biconcave 32.3%, and echinospherocyte are 4.2%. Only echinospherocyte RBCs can be accurately classified by taking into account 2-D features while other categories have significant error. According to the confusion matrix, PRNN confuses between biconcaves and flat-discs by using 2-D features (data not shown). In contrast, 3-D features explained in Table 1 generate more accurate and interesting results. According to the results of 10-fold CV, misclassification rates are 0%, 1.6%, 3.2%, and 0% for flat-disc, stomatocyte, biconcave, and echinospherocyte RBCs, respectively. Table 4 shows the classification error for 2-D and 3-D features. It is noted that the presented neural network classification strategy was used to evaluate the discrimination power of the feature set based on 3-D morphological properties of RBC against 2-D features. The classification results obtained with the neural network demonstrate that the 3-D features can be more effective in RBC classification than the 2-D features.
2-D feature descriptions.
|2-D-F1||PSA||( is pixel size in holographic image and is number of pixels within RBC)|
|2-D-F2||Perimeter (Pr)||Length of the RBC boundary|
|2-D-F4||Elongation (El)||Orientation of chain code in the cell boundary|
|2-D-F5||Radios ()||Radius is estimated by considering the radius of a circle having the area of the PSA|
Misclassification results of 2-D and 3-D features.
In another experiment, we evaluated the normalized mutual information between each feature in 2-D and 3-D features.40 In the case of 2-D features, it turns out, as shown in Table 5, that there is big mutual information among some features. Mutual information is the amount of information that two features share. If the mutual information between the two features is large (small), the two features are closely (not closely) related. If the mutual information becomes zero, the two features are independent. For example, 2-D-F1 has significant mutual information with features 2-D-F2, 2-D-F4, 2-D-F5, and 2-D-F6 (see first row of Table 5). Therefore, this feature is statistically redundant and cannot add significant information.
Normalized mutual information between 2-D features.
Combining 2-D and 3-D Features and Select the Best Feature-Set
We believe that the best classification model should take into account not only 3-D features but also 2-D features. However, it is worth mentioning that not any feature can add significant information to the classification model. Therefore, we believe that the performance of any classification model can be enhanced by taking advantages of feature selection (FS) strategy. In FS, we seek to find the best set of the features that has the strongest ability (or strong) to distinguish each class. Generally speaking, FS preserves the original features intact; features deemed unimportant/irrelevant/redundant are simply eliminated from further consideration while selecting only those features that significantly contribute to the classification problem. Therefore, FS can reduce the number of features (variables) of the classification problem and make the model simpler (or less complex) and shorten training time.40 We also implemented FS in this research by using SFFS technique. Generally speaking, in SFFS features are sequentially added to an empty candidate set until the addition of further features does not decrease the criterion. It has two components of an objective function, called the criterion, and a sequential search algorithm. In former, common criteria are misclassification rate for classification objects (similar in this paper) and mean squared error for regression models. A sequential forward search algorithm adds features from a candidate subset while evaluating the criterion. Since an exhaustive comparison of the criterion value at all subsets of an -feature data set is typically infeasible (sometimes feasible but time-consuming), sequential searches grow the candidate set. It turns out that the following features of average RBC thickness (3-D-F1), top view surface (TVS) area volume ratio (3-D-F4), sphericity coefficient (3-D-F9), the upper side of the ring divided by lower side of the ring (3-D-F11), and perimeter (2-D-F2) can better classify multiple RBCs in this research. Our experimental results also show that adding more features does not add significant discrimination ability to the final classification model. As it has been mentioned earlier, SFFS is responsible for adding or removing features from the final feature set. One component of SFFS is the objective function, which here is the misclassification rate. SFFS starts from an empty set and add features one by one to the set and evaluates the misclassification rate. If there is a significant change in the objective function (misclassification rate) then the feature can be added to the final feature set (see Table 6).
Misclassification rate after adding each feature to the feature set.
|Features added to the set||Criterion value (misclassification rate)|
|Average RBC thickness (3-D-F1)||0.252396|
|RBC perimeter (2-D-F2)||0.134185|
|TVS area-volume ratio (3-D-F4)||0.0670927|
|Sphericity coefficient (3-D-F9)||0.0607029|
|The upper side of the ring divided by lower side of the ring (3-D-F11)||0.0511182|
|Adding next feature to the feature set (sixth feature)||0.479233|
|Adding next feature to the feature set (seventh feature)||0.479233|
|Adding next feature to the feature set (eighth feature)||0.479233|
According to Table 6 after adding the 7th and 8th features, the misclassification rate never changes. It means that they have no contribution to the final feature set. Adding the 6th feature decreases the misclassification rate marginally but we still did not consider it as part of the feature set since we wanted to keep the final feature set as small as possible (only five features).
Figure 7 shows the data distribution RBCs for each selected feature. We can see that the distribution of average RBC thickness of echinospherocyte RBCs has almost no overlap with other distributions [Fig. 7(a)]. Table 7 shows misclassification rate of the final feature-set and PRNN approach.
Misclassification results for the best feature set obtained by sequential features selection technique.
|Best feature set||0%||0.9%||3.1%||0%|
The confusion matrix of the test set shows that PRNN sometimes confuses stomatocytes and echinospherocytes because there are cases in which RBC has a morphology similar to both stomatocyte and echinospherocytes (see Fig. 8). According to Fig. 8, we can see that RBC is similar to both stomatocyte and echinospherocytes morphology and posterior probability for belonging to stomatocyte and echinospherocyte categories are 0.33 and 0.66, respectively. Even for a human examiner, it can be difficult to put it in the correct category.
In another experiment, we tried to count different types of RBCs in five quantitative phase images with multiple RBCs. Images are inspected visually by the human inspector and then results are presented. In Fig. 9(a) stomatocyte RBCs are dominant (flat discs 0/80, stomatocytes 58/80, biconcave 8/80, and echinospherocyte 14/80) and in other images biconcave RBCs [Fig. 9(b) has following flat discs 2/27, stomatocytes 5/27, biconcave 20/27, and echinospherocyte 0/27] and echinospherocytes are dominant [Fig. 9(c) shows RBC types of flat-disc 0/49, stomatocytes 5/49, biconcaves 6/49, and echinospherocytes 38/49, and Fig. 9(d) shows RBC types of flat-disc 0/63, stomatocytes 3/63, biconcaves 1/63, and echinospherocytes 59/63]. Figure 9(e) shows an image with 40 days storage time with many flat-disc RBCs (flat-disc: 16/36, stomatocytes: 7/36, biconcaves: 5/36, and echinospherocytes: 8/36). The numbers in parenthesis show the number of each morphology obtained by the human inspector. At first, each image is segmented into many RBCs since feature extraction should be applied at the single cell level. Then, the percentage of the different types of RBCs in the RBC phase images is calculated (see Fig. 9). As expected, the classifier showed that at the first sample stomatocyte RBCs are dominant. In contrast, in the second and fifth figures biconcave and flat-disc RBCs are the major types. Third and fourth figures illustrate that echinospherocytes are dominant RBCs. Although there is a small error in counting nondominant RBCs, the main and important thing is counting and reporting the dominant type for further investigation.
According to the results, the proposed feature set and classifier can automatically count and categorize different types of RBCs in human RBCs by taking advantages of 2-D and 3-D profiles of RBC. The classifier is helpful to assess RBC-related abnormality since the ratio of the different types of RBCs is associated with certain types of diseases. There are some disadvantages in using NN technology regarding the classification such as there is no formula for the number of nodes and hidden layers. Generally, these numbers in the network are problem dependent and are decided by the trial-and-error method.
We believe that DHM, by providing the quantitative phase images, allows for doing different analysis especially classification goals. For example, someone can easily classify echino-echino-spherocyte from the rest of cells easily by using a conventional binary support vector machine (SVM) classifier and one or two features. Figure 10(a), e.g., shows the density of average cell thickness of echinospherocytes against other RBCs. Any binary classifier can be used to separate these two groups. Figure 10(b) shows the scattering of two groups based on average thickness (AT) value and surface area, and an SVM classifier with its boundary region.
Automatic classification of different types of RBCs in human RBC is a challenging and important problem for pathological diagnosis. In particular, as far as RBCs are concerned, classification based on their 3-D profile is highly relevant. Another issue is that since demands on laboratories are ever increasing and funding and staffing levels are generally below the desired level, the implementation of a system that shortens staff time, cost-effective, and noninvasive is greatly desired. In this paper, we have presented and assessed the use of PRNN applied to the 2-D and 3-D features of RBCs obtained through DHM in order to categorize and count biconcave, stomatocyte, flat-disc, and echinostomatocyte RBCs in an RBC sample with multiple types. Six 2-D features and 13 3-D features have been extracted and classification results are compared right after. Our experimental results show that the 3-D features have more useful information in RBC classification. In addition, FS shows that average RBC thickness, TVS area volume ratio, sphericity coefficient, the upper side of the ring divided by lower side of the ring, and RBC perimeter can better classify RBCs into the desired categories. The experimental results and the performance imply that the final feature set can help in classifying and counting RBCs, which is substantially important in analyzing RBC abnormality and shape-related diseases.
This work was supported by the Basic Science Research Program through the National Research Foundation of Korea (NRF), which is funded by the Ministry of Science, ICT & Future Planning (NRF-2015K1A1A2029224) and the research funds from Chosun University, 2016.
Keyvan Jaferzadeh is a PhD student in the Computer Engineering Department at Chosun University. He received his BS degree in software engineering and MS degrees in mechatronic engineering in 2006 and 2010, respectively. His current research interests include image processing, digital holography, image compression, and machine vision.
Inkyu Moon received his BS degree in electronics engineering from SungKyunKwan University in Korea in 1996 and his PhD in electrical and computer engineering from the University of Connecticut in the United States in 2007. He joined Chosun University in Korea in 2009 and is currently a professor at the School of Computer Engineering. His research interests include digital holography, biomedical imaging, and optical information processing. He is member of IEEE, OSA, and SPIE.