We present a framework based on the development of adaptive scalable kernel (ASK) for hyperspectral image classification, which can achieve an excellent status in removing insignificant details and defending crucial features. The proposed method consists of three steps. First, the spectral feature extraction based on interval gradient and a fast morphological filter is used to reduce the high dimensionality. Second, a powerful spatial structure extraction method based on adaptive scale kernels is adopted to enhance the performance of structure-preserving filtering. Depending on patch-based statistics, this model identifies small-scale texture from large-scale structure and finds an optimal per-pixel smoothing scale. Third, the obtained spectral structure feature maps are classified with the large-margin distribution machine. The experimental results show that the proposed spatial structure extraction method based on ASK achieves the state-of-the-art performance in terms of classification accuracy and computational efficiency.