Hyperspectral images (HSI) have a strong ability in information expression, and sparse subspace clustering (SSC) model for HSI have become very popular in recent years. Due to polarization information has a good performance on the edges and roughness of materials, adding polarization information into HSI clustering can give better results. In this paper, a fast spectral-polarized sparse subspace clustering (FSP-SSC) algorithm combining hyperspectral information and polarized information is presented. Furthermore, a new framework in the manner of sampling-clustering-classifying is used to reduce the computational complexity of the algorithm: firstly, super pixels which are segmented form original images by simple linear iterative clustering (SLIC) are sampled; then the samples are clustered by solving the optimization problem considering both of hyperspectral information and polarized information; after that, we can get the final cluster results by classifying non-sampled super pixels into the clusters based on the sampled super pixels. Some experiments have been performed to demonstrate the accuracy, efficiency and potential capabilities of proposed algorithm.