Currently, most exposure fusion algorithms are put forward on the assumption that the source images are aligned prior to fusion. As a result, some artifacts, such as haloing, may be caused due to the slight misalignment in the source images. In order to reduce the influence induced by the misalignment, a novel shift-invariant and rotation-invariant steerable pyramid-based exposure fusion (SPBEF) algorithm is proposed. It can combine multiple color images of a scene with different exposures to one image with high quality. First, instead of processing of R, G, and B channels separately, the chrominance information of the scene is obtained by the average image of the median two images. The strategy can greatly reduce the computational complexity. Second, the luminance images of source images are then transferred to frequency domain and are fused to one luminance image using different fusion rules in different frequencies. In this process, fusion is performed in a hierarchical fashion. Last, the final color image is generated by combining the data of the fused luminance image and the chrominance information. Experiments show that SPBEF can give comparative or even better results compared to other exposure fusion algorithms, as well as other traditional pyramid-based fusion algorithms.