Clouds are one of the major factors leading to the failure of optical remote sensing observations on Earth, and reconstructing the information loss caused by clouds in optical images has attracted significant attention. Existing methods primarily utilize multi-temporal optical images as references, which makes it challenging to reflect real-time information. Due to the cloud-penetrating capability of synthetic aperture radar (SAR) imaging, SAR images can serve as reference images to capture real-time information during the cloud removal process. We propose a cloud removal method, DS2-SO-CR, that leverages multi-source data fusion using a conditional generative adversarial network (CGAN) combined with a residual network to effectively remove clouds from multi-cloud images. We used a CGAN model to obtain SAR-to-optical images, which are then input into a fusion residual network along with SAR images and cloudy optical images. This significantly enhances the richness and accuracy of the texture in the cloud-removed images. To ensure the preservation of cloud-free regions in cloudy images during the fusion process, we added a cloud detection component to the fusion network. The proposed method was validated on the SEN12MS-CR and HN-TS datasets, achieving the following accuracy metrics: PSNR of 30.48, SSIM of 0.892, MSE of 0.038, LPIPS of 0.146, and FID of 37.53. This study provides a reference for cloud removal in remote sensing imagery of cloudy regions, as well as support for monitoring geological disasters, vegetation, and land cover changes.