Hyperspectral image (HSI) is a three-dimensional data cube containing two spatial information dimensions and one spectral information dimension. The spectral vectors of different classes may have similar tendency and value that may bring about negative influences on classification. It is, therefore, important to introduce signal preprocessing techniques in the spatial domain to improve classification accuracy of HSIs. Assuming that local pixels in HSI have some correlations with each other, this paper proposes a spatial filtering model based on adaptive manifold (AM) for HSI. The AM for spatial filtering emphasizes the similar neighboring pixels and is robust to resist the noisy points with fast speed. The rich information in the filtered data is effective for improving the performance of the subsequent classification. The filtered data are classified by an extreme learning machine (ELM). The experimental results indicate that the framework built based on AM and ELM provides competitive performance. Specifically, by classifying the filtered data, the average accuracy of ELM can be improved as high as 30.54%, while performing tens to hundreds times faster than those state-of-the-art classifiers.