Native fluorescence spectra play important roles in cancer detection. It is widely acknowledged that the emission spectrum of a tissue is a superposition of spectra of various salient fluorophores. However, component quantification is essentially an ill-posed problem. To address this problem, the native fluorescence spectra of normal human very low (LNCap), moderately metastatic (DU-145), and advanced metastatic (PC-3) cell lines were studied by the selected wavelength of 300 nm to investigate the key fluorescent molecules such as tryptophan, collagen and NADH. The native fluorescence spectra of cancer cell lines at different risk levels were analyzed using various machine learning algorithms for feature detection and develop criteria to separate the three types of cells. Principal component analysis (PCA), nonnegative matrix factorization (NMF), and partial least squares fitting were used separately to reduce dimension, extract features and detect biomolecular alterations reflected in the spectra. The scores corresponding to the basis spectra were used for classification. A linear support vector machine (SVM) was used to classify the spectra of the cells with different metastatic ability. In detection of signals coming from tryptophan and NADH with observed data corrupted by noise and inference, a sufficient statistic can be obtained based on the basis spectra retrieved using nonnegative matrix factorization. This work shows changes of relative contents of tryptophan and NADH obtained from native fluorescence spectroscopy may present potential criteria for detecting cancer cell lines of different metastatic ability.