In the agricultural field, optical remote sensing technology plays an important role in crop monitoring or production estimation. However, the widespread distribution of clouds and rain limits the application of optical remote sensing. Synthetic aperture radar (SAR) has been widely used for studies of oceans, atmosphere, land, and space exploration, as well as by the military due to its all-weather nature, penetration to surface and cloud layers, and diversity of information carriers. However, it is difficult to classify ground objects with high accuracy based on SAR data. Considering the features of these two datasets, we proposed a framework to improve crop classifications in cloudy and rainy areas based on the optical-SAR response mechanism. Specifically, this method is designed to train a parametric analytic model in the area using both kinds of datasets and applied in the area with only SAR data to obtain the optical time-series features. Then crops from the second area were classified by the long-short-term memory network. As an example, the parametric analytic model in Lixian County was studied and was applied to Xifeng County to classify the crops with the OA of 61%, which had proved the robustness of the method.