The extensive development of coal resources has resulted in the formation of tensile fissures, disrupting soil structure and impacting soil quality. Monitoring soil nutrients in such areas is crucial for agricultural productivity. This study focused on the tension fissure area near the 3522 working face of Huaibei coal mine, Anhui province, China. We analyzed the variability characteristics of soil organic matter (SOM) in the fissure zone and assessed the potential of using soil spectra to estimate SOM. The fractional order differentiation (FOD) method was employed to preprocess the spectral data. Traditional correlation coefficient (CC) and coupled algorithms correlation coefficient—successive projection algorithm (CC-SPA) and correlation coefficient—competitive adaptive reweighted sampling (CC-CARS) were used for band extraction. Finally, partial least squares regression (PLSR) and support vector machine regression (SVR) were applied to establish the SOM estimation model. The results show that the development of fissures decreases the SOM content, and using spectroscopy to estimate SOM content in the fissure zone has greater potential. FOD can capture subtle spectral features. As the fractional order increases gradually from 0 to 1, spectral features are enhanced, but further increasing the fractional order introduces minor noise peaks, limiting the accuracy of SOM estimation. Feature selection algorithms effectively extract sensitive spectra related to SOM. The coupling algorithm CC-CARS obtains good estimation results with fewer band variables extracted. After 0.8 order differentiation, the SVR model constructed using feature wavelengths extracted by the CC-CARS algorithm performs best, with an R2 of 0.87 for the test set. The study can provide some technical references for the rapid and accurate estimation of SOM in coal mining fissure zones.