Soil texture is a primary parameter affecting the soil–water retention characteristics, plant nutrient storage, and carbon sequestration capability of a soil. A rapid, nondestructive, and economical detection method needs to be developed. A near-infrared hyperspectral imaging (HSI) system with a wavelength range of 874 to 1734 nm was used to measure the soil samples from Songyang County, Zhejiang Province, China. Totally, 198 samples were classified into four different texture groups: clay, clay loam, loam, and sand, based on the International Soil Science Society standard. The optimal wavelengths were selected by the method of competitive adaptive reweighted sampling. The texture features (contrast, correlation, energy, and homogeneity) of the images at the optimal wavelengths were extracted by the gray-level co-occurrence matrix. The results showed that both the spectral and image texture information could classify soil texture to some extent. The model established by the combination of optimal wavelengths and texture features achieved the highest correct classification rate of 87.9%. It was concluded that HSI was an efficient technique to classify the soil texture into four groups of clay, clay loam, loam, and sand.