How to choose effective fusion frames and how to obtain effective fusion coefficients are key problems in image fusion. A novel image fusion scheme is presented based on multiscale decomposition and directional filter banks (DFBs). First, contrast pyramid (CP) decomposition is used for each level of each original image. Then, DFBs are constructed for filter each image. Furthermore, a kind of evolution computation method—the immune clonal selection (ICS) algorithm—is introduced to optimize the fusion coefficients for better fusion products. By applying this technique to fusion of infrared thermal and visual light images, simulation results clearly demonstrate the superiority of this new approach. Fusion performance is evaluated through subjective inspection, as well as objective performance measurements. Experimental results show that the fusion scheme is effective and the fused images are more suitable for further human visual or machine perception.