This paper presents an end-to-end framework for automatically detecting and segmenting blood cells including normal red blood cells (RBCs), connected RBCs, abnormal RBCs (i.e. tear drop, burr cell, helmet, etc.) and white blood cells (WBCs). Our proposed system contains several components to solve different problems regarding RBCs and WBCs. We first design a novel blood cell color representation which is able to emphasize the RBCs and WBCs in separate channels. Template matching technique is then employed to individually detect RBCs and WBCs in our proposed representation. In order to automatically segment the RBCs and nuclei from WBCs, we develop an adaptive level set-based segmentation method which makes use of both local and global information. The detected and segmented RBCs, however, can be a single RBC, a connected RBC or an abnormal RBC. Therefore, we first separate and reconstruct RBCs from the connected RBCs by our suggested modified template matching. Shape matching by inner distance is later used to classify the abnormal RBCs from the normal RBCs. Our proposed method has been tested and evaluated on different images from ALL-IDB,10 WebPath,24 UPMC,23 Flicker datasets, and the one used by Mohamed et al.14 The precision and recall of RBCs detection are 98.43% and 94.99% respectively, whereas those of WBCs detection are 99.12% and 99.12%. The F-measure of our proposed WBCs segmentation gets up to 95.8%.