Infrared images are widely used in the practical applications to capture abundant information. However, it is still challenging to enhance the infrared image by the visual image. In this paper, we propose an effective method using bidirectional similarity. In the proposed method, we aim to find an optimal solution from many feasible solutions without introducing intermediate image. We employ some priori constraints to meet the requirements of image fusion which can be detailed to preserve both good characteristics in the infrared image and spatial information in the visual image. In the iterative step, we use the matrix with the square of the difference between images to integrate the image holding most information. We call this matrix the bidirectional similarity distance. By the bidirectional similarity distance, we can get the transitive images. Then, we fuse the images according to the weight. Experimental results show that, compared to the traditional image fusion algorithm, fusion images from bidirectional similarity fusion algorithm have greatly improved in the subjective vision, entropy, structural similarity index measurement. We believe that the proposed scheme can have a wide applications.