This paper presents a new approach to the segmentation of the microscopic nuclei images. First, for segmentation of the
cell nuclei from background, the adaptive local thresholding is used. A threshold for adaptive local thresholding is
estimated by using the gaussian mixture model and maximizing the likelihood function of gray value of cell images.
After nuclei segmentation, overlapped nuclei and isolated nuclei need to be classified for exact nuclei separation. For
nuclei classification, this paper extracted the morphological features of the nuclei such as compactness, smoothness and
moments from training data. For overlapped nuclei classification, this paper uses a Bayesian network with three
probability density functions for evidence at each node. The probability density functions for each node are modeled
using the three morphological features. After nuclei classification, segmenting of overlapped nuclei into isolated nuclei is
necessary. Since watershed algorithm has the problem of over-segmentation, we find makers from each overlapped
nuclei and apply watershed algorithm with the proposed merging algorithm. The experimental results using microscopic
nuclei images show that our system can indeed improve segmentation performance compared to previous researches, because we performed nuclei classification before separating overlapped nuclei.