This paper proposes a novel simple image encryption and decryption technique based on the principle of interference and a binary-phase computer-generated hologram (BPCGH). To obtain a stable interference pattern, a technique involving a photorefractive material is also proposed. The proposed scheme is composed of digital encryption and optoelectronic decryption. For the encryption process, a BPCGH that reconstructs the original image perfectly is designed, using a simulated annealing algorithm, and the resulting hologram is regarded as the image to be encrypted digitally. In addition, a reference image with key information is created using a random two-phase generator. The hologram is then encrypted using the reference image and the interference rule. For the decryption process, the interference intensity is obtained by interfering the reference image and the encrypted image, binarized with software, and then transformed into phase information. Finally, the original image is recovered by an inverse Fourier transformation of the binary-phase information. During this process, since the intensity information generated by the interference of the two images is very sensitive to external vibrations, a stable interference pattern is achieved using a self-pumped phase-conjugate mirror made of the photorefractive material BaTiO3.