Delineation of the subcortical nucleus in MR images is prerequisite for advanced radiotheraphy, surgical planning and morphometric analysis. However, it is always difficult to implement such a complicated work. We proposed a novel framework of 3D active shape model (ASM) based segmentation of the subcortical nucleus in MR images. Firstly, the most representative one of all samples represented by the segmented MR volumes is selected as the template and triangulated to generate a triangulated surface mesh. Then, free form deformation is used to establish dense point correspondences between the template and the other samples. A set of consistent triangle meshes are obtained to build the model by a statistical analysis. To fit the model to a MR volume, the model is initialized with Talairach transformation and the edge map around the model is extracted using watershed transform. An algorithm of robust point
matching is used to find a transformation matrix and model parameters to transpose the model near the target nucleus and match the model to the target nucleus, respectively. The proposed framework was tested on 18 brain MR volumes. The caudate, putamen, globus pallidus, thalamus, and hippocampus were selected as the objects. In comparison with manual segmentation, the accuracy (Mean±SD) of the proposed framework is 0.90±0.04 for all objects.