The key to successful spectral un-mixing is indicating number of endmembers and their corresponding spectral signatures. Nevertheless, correctly estimate the number of end members without a priori knowledge is a very hard task because pixels in a hyperspectral image are always contain a mixture of the several reflected spectra. Currently, Noise Whitened Harsanyi, Farrand, and Chang (NWHFC) and hyperspectral signal subspace identification by minimum error (HySime) are two well-known methods for estimating the number of endmembers. However, in practice, because NWHFC requires fixing the false-alarm probability and HySime needs estimate noise of each spectral band, these two methods may not only ignore small objects but also can’t identify endmembers. In this paper, assuming endmembers in a hyperspectral image can be modeled by convex geometry. We propose a threestage process to estimate the number of endmembers. At the first stage, principal component (PC) is used to transform original image to low-dimensional components for speeding up algorithm execution. At second stage, successive volume maximization (SVMAX) is used to obtain vertex using convex properties. At the third stage, spectral angle mapper (SAM) is used to compute similarity measures among vertex, and minimum SAM value represents vertex separation. Repeat the second and third stages by increasing transformed component dimensions until reach a predefined criteria. The number of endmembers of the image is the vertex with maximum of the vertex separations Finally, the proposed method is applied to synthetic and real AVIRIS and HYDICE hyperspectral data sets for estimating the number of endmembers. The results demonstrate that the proposed method can be used to estimate more reasonable and precise number of endmembers than the two published methods.