Dyadic displacements of an image can be regarded as a special type of permutations of pixel addresses. This property can be used to encrypt an image without information loss, damage, or addition. In this paper, a new optical image encryption technique based on dyadic permutations and holographic interference is proposed. First, we adopt holographic interference to record both the phase and amplitude of the Fourier transform of an input image as the intensity information. Then the recorded phase is processed through dyadic permutations based on exclusive-OR (XOR) operations. The original image can be successfully recovered by using the inverse procedure.