An effectual method is proposed to remove impulse noise from corrupted color images. The spiking cortical model (SCM) is adopted as a noise detector to identify noisy pixels in each channel of color images, and detected noise pixels are saved in three marking matrices. According to the three marking matrices, the detected noisy pixels are divided into two types (type I and type II). They are filtered differently: an adaptive median filter is used for type I and an adaptive vector median for type II. Noise-free pixels are left unchanged. Extensive experiments show that the proposed method outperforms most of the other well-known filters in the aspects of both visual and objective quality measures, and this method can also reduce the possibility of generating color artifacts while preserving image details.