2.1.

Image Steganography

The purpose of image steganography is to send the stego-images like innocent normal images to the receiver and avoid the secret data being noticed by the attacker. As can be seen in Fig. 2, the process of image steganography is to embed the secret data into the cover image and then arrive at the stego-image that steganalysis such as spatial rich model (SRM)²⁴ and maxSRMd2,²⁵ which are two feature-based classifiers, and Xu-Net,²⁶ which is a convolutional neural network classifier, tries to distinguish the stego-image from the cover image.

Fig. 2

The process of image steganography and steganalysis.

The least significant bit (LSB) replacement is a classical steganography algorithm that replaces the LSBs of the cover image with the secret data bits.²⁷ However, the LSBs of the pixel of the cover image take up a small part of the cover image, so the capacity is limited. Meanwhile, if we modify more bits of the pixel of the cover image to get a larger capacity, the possibilities to be detected by the attacker are higher. For higher capacity and higher undetectability, Pevný et al.²⁸ introduced HUGO, which provided a larger capacity while having equal safety compared with LSB matching.

Considering the relationship between the content of the image itself and the size of the secret image, adaptive steganography selects the edge²⁹ or texture³⁰ areas of the cover image as the embedding location as much as possible to strengthen the invisibility of stego-images. Li et al.³¹ proposed a cost function for spatial image steganography that used a high-pass filter to locate the less predictable parts in an image, and then utilized two low-pass filters to make the low-cost values more clustered; it achieved better performance on resisting SRM steganalysis over HUGO²⁸ and S-UNIWARD.³⁰

Deep learning image steganography methods can reach a higher capacity and have higher undetectability by not only the traditional feature-based steganalysis model but also deep learning steganalysis models over the traditional steganography algorithms. The capacity is measured by bits-per-pixel (bpp), which is the average number of bits concealed into each pixel of the cover image.³² An ideal image steganography method could embed as much secret data as possible into the cover image without being detected by steganalysis methods. For example, Ref. 33 described that 0.2, 0.3, 0.4, and 0.5 bpp of secret data were embedded into the image from the BOSSbase³⁴ dataset using three algorithms. As illustrated in Table 1, the detection error rates of three steganalysis methods SRM,²⁴ maxSRMd2,²⁵ and Xu-Net²⁶ on the deep learning model in Ref. 33 are all higher than on traditional image steganography methods such as HILL³¹ and S-UNIWARD,³⁰ which shows that deep learning-based image steganography has superiority in escape steganalysis.

Table 1

Different steganalysis models detection error rate of different image steganography methods.

2.2.

Image Encryption

Image encryption keeps image content invisible until someone has the correct key. As we can see in Fig. 3, image cryptography involves encrypting the plaintext image to the ciphertext image by the encryption key and decrypting the ciphertext image to the decrypted image by the decryption key. Cryptanalysis involves cracking the cryptography to get the encryption/decryption keys or the secret image; these includes ciphertext only attacks,³⁵ known-plaintext attacks,³⁶ chosen plaintext attacks,³⁷ and chosen ciphertext attacks.³⁸

Fig. 3

The process of image cryptography and cryptanalysis.

Chaos is sensitive to initial states and possesses a complex and unpredictable long-term behavior,³⁹ so it is widely used in image encryption. Bentoutou et al.⁴⁰ introduced an efficient image encryption method based on chaotic maps and the Advanced Encryption Standard. Naim et al.⁴¹ described a satellite image encryption algorithm based on the linear-feedback shift register generator, SHA 512 hash function, hyperchaotic systems, and Josephus problem.

Based on the characteristic of optical systems,^42–44 cellular automata,^45–47 quantum,^48–50 and DNA computing,^51–53 there are many applications in image encryption.

In most deep learning image encryption methods, deep learning plays a supporting role, which gives full play to their special advantages. End-to-end deep learning image encryption is a new research direction and has achieved security that is the same as or even better than the traditional encryption methods. Table 2 compares three different image encryption methods in ciphertext image entropy; correlation coefficients of two horizontal, vertical, diagonal adjacent pixels; the number of pixels change rate (NPCR), and the unified averaged changed intensity (UACI) on satellite images, and the best values are shown in bold. As we can see in Table 2, the end-to-end deep learning image algorithm in Ref. 54 has better values in correlation coefficients and NPCR than the other two traditional image encryption methods. Furthermore, in other indexes, the deep learning method also has a similar value to the other two methods.

Table 2

Ciphertext image security performance comparisons of different image encryption schemes.

2.3.

Image Authentication

Checking the image identity or image integrity is the target of image authentication. Image watermarking is a significant technique for authenticating images. As can be seen in Fig. 4, the process of image watermarking is like image steganography. The watermark is embedded into the container and arrives at the watermarked image. Watermark extraction or detection involves extracting or detecting and further authenticating the watermark. From the perspective of watermark visibility, an image watermark can be divided into the visible watermark⁵⁵ and the invisible watermark,⁵⁶ whereas from the perspective of watermark robustness, an image watermark can be classified by fragile watermark,⁵⁷ semifragile watermark,⁵⁸ and robust watermark.⁵⁹ Furthermore, from the mode of watermark extraction, an image watermark can be itemized into blind,⁶⁰ semiblind,⁶¹ and nonblind extraction.⁶²

Fig. 4

The scheme of image watermarking and watermark extraction or detection.

Compared with the traditional image watermark methods, the deep learning methods have better imperceptibility and robustness. Table 3 describes the peak signal-to-noise ratio (PSNR) and normalized correlation (NC) between the extracted watermark from the watermarked image under multiattacks and the original watermark. Suppose that $x$ and $y$ are two images with the size of $M\times N$ and $(i,j)$ is the pixel location in an image. The PSNR is calculated as Eq. (1), and NC is computed as Eq. (2), where if $a=b$, then $f(a,b)=1$, and otherwise, $f(a,b)=0$:⁶³

Table 3

Comparisons of different methods facing multiattacks.

As can be seen in Table 3, when watermarked images face the attacks like median filtering of $3\times 3$, salt and pepper noise with noise level $\delta$ of 0.005, and JPEG compression with quality factor 20, the deep learning method in Ref. 63 reaches almost all of the highest watermark quality assessment values, which are shown in bold, while it also obtains the most complete watermark with respect to no attack.

3.1.

Learning to Generate Cover Images

As can be seen in Fig. 5, using deep convolutional generative adversarial networks (DCGAN),⁶⁴ Volkhonskiy et al.⁶⁵ proposed steganographic generative adversarial networks (SGAN), which was trained on the Celebrities dataset. Liu et al.⁶⁶ generated cover images and embedded secret messages using the LSB algorithm to deceive the steganography analyzer. Similar to SGAN, secure steganography based on generative adversarial networks (SSGAN)⁶⁷ used wasserstein generative adversarial networks (WGAN)⁶⁸ as the cover images generator and Gaussian-neuron convolutional neural networks (GNCNN)⁶⁹ as the steganography analyzer trained on the CelebA⁶⁶ database to improve the training performance and image quality. They opened the field of container generation, although they lacked considerations in steganography properties.

Fig. 5

The structure of SGAN and SSGAN.

3.2.

Learning to Generate Stego-Image

As can be seen in Fig. 6, the deep learning model in stego-image generation consists of three parts: the generator translates the cover image and secret data to the stego-image, the restorer recovers the secret data from the stego-image, and the steganography analyzer determines if the input image is either the stego-image or the cover image.

Fig. 6

The framework of the deep learning model in stego-image generation.

Hayes and Danezis⁷⁰ simulated a communication scenario in which three neural networks trained on the CelebA dataset engaged: Alice and Bob communicated by concealing a secret message in the carrier image, and Eve eavesdropped on the image and distinguished the embedded image from the innocent image. Baluja⁷¹ designed three neural networks: a preparation network gained the features of the secret image, a hiding network concealed the extracted features into the cover image across all available bits, and a revealing network reconstructed the secret image from the stego-image. Chattopadhyay et al.⁷² adopted a multistage feed-forward artificial neural network to complete image steganography. Zhu et al.⁷³ transferred the input message and cover image to a discriminator-indetectable image by a neural network and recovered the secret message from the encoded image by the other neural network simultaneously. Hu et al.⁷⁴ trained a generator to translate a carrier-like image deceiving the discriminator using the vector calculated by the secret information and an extractor to reconstruct the vector and map it to the secret message reversely. The neural networks were trained on the CelebA and Food-101⁷⁵ datasets. However, the disadvantages of DCGAN result in some drawbacks. For example, some generated stego-images were not sufficiently natural, the size of the stego-image was small, and the steganography capacity was not big adequate. Zhang et al.⁷⁶ hid arbitrary binary data in images using generative adversarial networks (GAN). Wang et al.⁷⁷ utilized a secret information preprocessing method, Inception-ResNet block, GAN, and perceptual loss to improve the undetectability, imperceptibility, and capacity, although the model had certain limitations on the length of secret information. Zhang et al.³² used an inception-module-based neural network to embed a secret gray image into an image with the same size, which was the $Y$ channel of the cover image; a neural network to recover the secret image from the $Y$ channel of the stego-image, which tried to minimize a mean square error (MSE), structural similarity (SSIM), and multiscale SSIM mixed loss function; and another neural network to judge whether the input image was the stego-image or not. Duan et al.⁷⁸ employed the residual learning block in the hiding network to directly generate a stego-image that looked like the cover image and designed the reveal network to recover the secret image.

Table 5 estimates PSNR and SSIM indices between the cover image and stego-image obtained by SteGAN,⁷⁰ HiDDeN,⁷³ SteganoGAN,⁷⁶ and HidingGAN⁷⁷ with embedding capacities of 0.4, 0.4, 4.4, and 4 bpp, respectively, in the COCO⁷⁹ dataset; ISGAN³² with embedding capacity of 8 bpp; Ref. 71 with embedding capacity of close to 24 bpp in the ImageNet⁸⁰ dataset; and Ref. 78 with embedding capacity of 23.8 bpp in the ImageNet, LFW,⁸¹ or Pascal-VOC⁸² datasets. The invisibility requires the stego-image to be similar to the cover image. And the higher the values of PSNR and SSIM are, the more similar the two images are. PSNR estimates the error between corresponding pixels of two images, but it does not considered the human visual characteristics; the SSIM⁸³ measures the image similarity from comparisons of luminance, contrast, and structure.

Table 5

Comparison of different deep learning methods in PSNR and SSIM between stego-images and cover images on different image datasets.

In practice, these two indices should both be considered. Table 6 compares the steganography capacity and cover image size of different deep learning methods. As can be seen in Fig. 7, the method proposed by Ref. 71 has the highest PSNR, SSIM, and capacity values, and its SSIM value is close to the theoretical maximum value 1, which is the best stego-image imperceptibility and the highest embedding capacity among all compared methods.

Table 6

Comparison of different deep learning methods in capacity and cover image size.

Fig. 7

Histogram comparison of different deep learning methods in PSNR and SSIM between the stego-image and the cover image and embedding capacity.

The deep learning methods generate the stego-image and recover the secret data automatically while having a large embedding capacity and a good performance in undetectability. But they do not recover the secret data 100% when the stego-image does not suffer any attacks. Deep learning methods are not completely accurate secret data extraction algorithms in steganography although they perform well in real complex communication situations. The secret data extraction accuracy is limited by the characteristics of the neural network. Furthermore, the fixed input and output size of deep learning model limits the size of the image that it could process.

3.3.

Learning Embedding Change Probabilities for Every Pixel in the Cover Image

As illustrated in Fig. 8, Tang et al.⁸⁴ trained a generator to learn the probability map and then embedded the secret message simulated by the ternary embedding simulator (TES), which was presented by a neural network; finally the obtained modification map was added to the cover image to achieve the stego-image that attempts to deceive the steganography analyzer. On the basis of Ref. 84, Yang et al.³³ presented a U-Net-based⁸⁵ generator and replaced the TES with the double-tanh function, which does not need to be trained. In addition, six high-pass filters were integrated into the steganography analyzer. Similar to Refs. 33 and 84, Tang et al.⁸⁶ employed reinforcement learning to learn the embedding policy, which performed pixel-level actions and rewards.

Fig. 8

The structure of deep learning model in probability map generation.

The above three methods were tested in embedding 0.1, 0.2, 0.3, 0.4, and 0.5 bpp of secret data into the image from the BOSSbase dataset, and SRM was used to detect them. As illustrated in Table 7, the detection error rates of Ref. 86 are all higher than the other compared algorithms and have the best undetectability.

Table 7

Detection error rate of different image steganography methods using SRM.

3.4.

Coverless Image Steganography

The coverless image steganography does not change the cover image but transmits it directly, so it does not easily raise the suspicion of the attacker, which can effectively resist steganalysis from a new angle. As shown in Fig. 9, Duan et al.⁸⁷ generated the stego-image directly using two generators and two discriminators trained on the CelebA dataset. Generator 1 translates the secret image to the cover image, and discriminator 1 distinguishes the stego-image from the cover image. Furthermore, generator 2 translates the stego-image to the secret image, and discriminator 2 determines whether the input image is generated or not. Luo et al.⁸⁸ showed a coverless image steganography scheme based on multiobject recognition. Liu et al.⁸⁹ retrieved images according to the features DenseNet⁹⁰ trained on the ImageNet⁸⁰ dataset and generated feature sequences using discrete wavelet transform (DWT) coefficients, which presented the secret data and had good robust and security performance in resisting most image attacks. Zhang et al.⁹¹ discussed a diverse image style transfer network trained on the Microsoft COCO dataset⁷⁹ using multilevel noise encoding with the image style transfer results presenting different codewords. However, the images with particularly dense textures were not suitable for datasets of steganography. Zhou et al.⁹² created a dictionary that defined the objects and corresponding codes and used a faster region-based convolutional neural network trained on the PASCAL VOC 2007⁸² and VOC 2012⁹³ datasets to detect objects in stego-images that would extract the secret message. Duan et al.⁹⁴ trained an improved Wasserstein GAN model on the LFW dataset⁸¹ and transmitted and input the disguised images with the generator outputting the images similar to the secret images in visual mode.

Fig. 9

The structure of deep learning model in stego-image direct generation in Ref. 87.

3.5.

Steganalysis Methods Based on Deep Learning

A deep learning model can automatically learn image features, which contributes to better classification precision. Utilizing the advantages of the deep learning model, Qian et al.⁶⁹ applied a single neural network called GNCNN, which employed six convolutional layers for extracting the image features, three fully connected layers for classifying, and a Gaussian function as the activation function for separating the cover and stego-signals. Xu et al.²⁶ proposed Xu-Net, which took absolute values of elements in the feature maps generated from the first convolutional layer to force the model to take into account the sign symmetry²⁴ that existed in noise residuals and constrained the range of data values with the saturation regions of tanh in the first two convolutional layers, while constraining the strength of the model using 1 × 1 convolutions in the last three convolutional layers to avoid overfitting. Yang et al.⁹⁵ incorporated selection-channel awareness into a modified Xu-Net architecture trained on the BOSSbase v1.01 dataset,³⁴ which applied large weights to features learned from complex texture regions and small weights to features learned from smooth regions. Zeng et al.⁹⁶ convolved the images with a set of kernels, then calculated the different quantized and truncated features, and finally used the CNN model trained on the ImageNet dataset to process the extracted features for JPEG steganalysis. Jian et al.⁹⁷ presented an image steganalysis CNN-based model called Ye-Net, which was initialized with the filters used in calculating residual maps in SRM and integrated them with the truncated linear unit to suit the distribution of embedding signals (with low signal-to-noise ratio) and selection channel awareness. Yedroudj-Net⁹⁸ involved a predefined convolutional layer for extracting the noise component residuals and adopted scale operations in the last three convolutional layers. Zhu-Net⁹⁹ applied $3\times 3$ convolutional kernels to reduce the number of parameters and model the features in a small local region, used separable convolution to enhance the stego-signal-to-noise ratio, and employed spatial pyramid pooling to aggregate the local features for strengthening the representation ability of features. SR-Net¹⁰⁰ adopted an expanded front part in a deep residual neural network without max-pooling operations to minimize the use of heuristics and externally enforced elements. GBRAS-Net¹⁰¹ used filter banks in the preprocessing phase and depth-wise, separable convolution, and skip connections in the feature extraction phase. Liu et al.¹⁰² constructed DFSE-Net with diverse filter parts that combined three different scale convolution filters that could process information diversely and squeeze-and-excitation parts that could enhance the effective channels out from diverse filter parts. Based on an efficient $\theta$-nondominated sorting genetic algorithm-III, Iskanderani et al.¹⁰³ described a densely connected CNN (DCNN) for image steganalysis. $\theta$ NSGA-III was utilized to tune the initial parameters of the DCNN model. It could control the accuracy and $f$-measure of the DCNN model by utilizing them as the multiobjective fitness function. Singhal and Bedi¹⁰⁴ proposed multiclass blind steganalysis that included 60 layers to record demographic features and residual mapping to retain weak stego-signals generated by embedding payload and thus making classification easier and utilized a high-pass filter to preprocess images.

Table 8 and Fig. 10 illustrate the detection accuracy of different deep learning steganalysis algorithms in multiple steganography conditions, which consist of using the WOW¹⁰⁵ steganography algorithm with 0.2 and 0.4 bpp embedding capacity and using the S-UNIWARD³⁰ steganography algorithm with 0.2 and 0.4 bpp embedding capacity on the BOSSbase dataset. Compared with other algorithms, GBRAS-Net¹⁰¹ achieves the highest steganalysis accuracy.

Table 8

Comparison of detection accuracy of deep learning steganalysis methods for WOW and S-UIWARD at embedding capacity of 0.2 and 0.4 bpp.

Fig. 10

Steganalysis accuracy comparisons of the deep learning steganalysis techniques against WOW and S-UNIWARD algorithms with the embedding capacity of 0.2 and 0.4 bpp.

Despite the image steganography method based on deep learning achieving an impressive result in embedding capacity and stego-image quality, these methods have a lack of robustness. The robustness of image steganography algorithms based on deep learning needs to be further discussed and improved in future studies. Furthermore, the input and output of the deep learning model are fixed, so the deep learning models only process the images of a fixed size. Developing a model that could process images of various sizes in the field of image steganography is a meaningful task.

4.1.

Image Compression in Image Encryption Algorithms

Image compression can increase the efficiency of image encryption by reducing the size of data. Chen et al.¹⁰⁶ utilized a deep learning model trained on a color image dataset¹⁰⁷ to compress and reconstruct the plaintext image and compound the chaotic system to encryption. Hu et al.¹⁰⁸ employed stacked autoencoder for compression and chaotic logistic map to encrypt the compressed vector. Suhail and Sankar¹⁰⁹ presented an application of image compression and encryption using an autoencoder and chaotic logistic map. Selvi et al.¹¹⁰ suggested a competent adaptive sigma filterized synorr certificateless signcryptive Levenshtein entropy coding-based deep neural learning technique trained on a dataset of chest x-ray images to develop the image encryption and compression (Table 9).

Table 9

Some deep learning models for image cryptography tasks.

4.2.

Image Resolution Improvement or Denoising in Image Encryption Algorithms

Zhang et al.¹¹¹ employed ghost imaging as a transmission and encryption mode and a CNN that was trained on the IAPR TC-12 benchmark¹¹² to improve the recovered image resolution. Chen et al.¹¹³ utilized a dilated deep CNN denoiser trained on the Waterloo exploration database,¹¹⁴ which improved the resolution of the fractional Fourier transform-based decrypted images and resistance against multiclass attacks.

4.3.

Image Object Detection and Classification in Image Encryption Algorithms

Zhao et al.¹¹⁵ applied the multitask cascaded convolution network (MTCNN) to seek five key feature points of human faces and then adopted a combination of chaotic logic diagrams and Rivest Cipher 4 stream ciphers to encrypt eigenvalues. At the same time, the face coordinates generated by MTCNN and user passwords were hash-converted and double-encrypted by a hash table, which reduced the size of the images to be encrypted. Alqaralleh et al.¹¹⁶ employed elliptic curve cryptography with an optimal key generated by hybridization of grasshopper with the fruit fly optimization algorithm, then used the neighborhood indexing sequence with burrow wheeler transform to encrypt the hash values, and finally adopted a deep belief network to classify the existence of the disease. Asgari-Chenaghlu et al.¹¹⁷ described a technique based on YoloV3 object detection and chaotic image encryption that had the ability of automatic image encryption on both full or user-selected regions.

4.4.

Image Private Key Generation in Image Encryption Algorithms

Li et al.¹¹⁸ trained a CNN on the CASIA iris database version 4.0¹¹⁹ to extract the feature of the iris image, then employed the RS error correcting code to encode the feature vector, and calculated the encryption key that was adopted to encrypt the plaintext image by the XOR operation. As can be seen in Fig. 11, Ding et al.¹²⁰ trained a GAN trained on Montgomery County’s chest x-ray¹²¹ dataset to generate the private key and input a seed image that had a large key space, pseudorandomness, a one-time pad, high sensitivity to change, and resistance to different kinds of attacks; then the bit-wise XOR algorithm was adopted as an encryption and decryption algorithm. Jin and Kim¹²² illustrated a method based on deep neural network learning without sharing the preshared key between systems and created and used the key that was variably used through the symmetric key encryption system of the 3D cube algorithm, which provided good security. Maniyath and Thanikaiselvan¹²³ proposed a robust deep neural network trained on an SIPI image database that generated a secret key resistive of multiattacks and applied a chaotic map to encrypt the image without any negative effect on image quality. Erkan et al.¹²⁴ mechanized a CNN trained on the ImageNet database to generate sensitive keys and then produced initial values and controlled parameters for the hyperchaotic log-map; thus they obtained a diverse chaotic sequence for image encryption. Fratalocchi et al.¹²⁵ decoupled the design of the physical unclonable functions from the key generation and trained a neural structure to learn the mapping between the key and the physical unclonable function, which could address the shortcomings of unreliability and weak unpredictability of cryptographic keys.

Fig. 11

The structure of the deep learning model in image private key generation and image encryption in Ref. 120.

4.5.

End-to-End Image Encryption

As can be seen in Fig. 12, the main end-to-end image encryption deep learning scheme consists of the encryption network that generates a random-like ciphertext image, the decryption network that reconstructs the plaintext image, and the discriminator that distinguishes the ciphertext images from the pixel-random images.

Fig. 12

The main scheme of the deep learning model in end-to-end image encryption.

Furthermore, cycle-consistent generative adversarial network (Cycle-GAN)¹²⁶ has a good performance in image style transfer, in which the process of image encryption is regarded as translating the usual images to images with randomly distributed pixels. Thus the neural network structure of Cycle-GAN was widely used as the encryption or decryption network in end-to-end image encryption methods based on deep learning. The neural network structure, described in Fig. 13, down-samples the input, then extracts the feature map through nine residual blocks,⁹ and finally up-samples the feature map to output the image with the objective style. One style transfer process of Cycle-GAN can be seen in Fig. 14, where the generator translates the original image to the generated image with objective style and the discriminator distinguishes whether the image is a real image.

Fig. 13

The generator neural network architecture of Cycle-GAN.

Fig. 14

One style transfer process of Cycle-GAN.

Li et al.¹²⁷ demonstrated an optical image encryption learning scheme based on Cycle-GAN that was trained by the plaintext-ciphertext training set of satellite images in which the ciphertext images were encrypted by double random phase encoding. Ding et al.¹²⁸ employed Cycle-GAN trained on a dataset of chest x-rays¹²¹ to encrypt and decrypt medical images as a style transfer task. In addition, a neural network is projected to gain the interested object from the ciphertext image. Bao et al.¹²⁹ constructed an encoder–decoder and discriminator framework trained on the Corel-1000 dataset¹³⁰ to imitate the process of image scrambling and reconstruction in which the parameters of the encoder and decoder are different. However, the cipher pixels were not uniformly distributed, the decrypted images quality and the generalization ability of model were not good, and the plaintext and ciphertext image sensitivities were weak. Bao and Xue⁵⁴ analyzed the causes of the weak avalanche effect in the neural network of Cycle-GAN and integrated the traditional diffusion algorithm into the Cycle-GAN-based image encryption methods trained on a dataset of satellite images scraped from Google Maps to enhance the avalanche effect, although its decryption performance was not well.

Table 10 compares the image encryption methods illustrated in Refs. 106, 109, 117, 120, 127128.–129, and 54 in PSNR and SSIM values between recovered decrypted images and plaintext images, the key space, and the encryption speed per plaintext image with $256\times 256$ resolution. Table 11 compares the image entropy, correlation coefficients of two horizontally, vertically, and diagonally adjacent pixels of the ciphertext image obtained by different methods, NPCR values, and UACI values when these methods face a chosen plaintext attack, especially, changing 1% of pixels of the plaintext image in Refs. 128 and 129. As can be seen in Tables 10 and 11, the end-to-end key generation and image encryption methods based on the deep learning model have a large key space and a quick encryption efficiency. However, correlation coefficients of two horizontally, vertically, and diagonally adjacent pixels of the ciphertext image and the resistance to the chosen plaintext attack need to be further improved generally.

Table 10

Comparison of PSNR and SSIM values between the decrypted image and the plaintext image, key space, and encryption efficiency in different deep learning methods.

Table 11

Comparison of different deep learning methods in image entropy, correlation coefficients of two horizontally, vertically, and diagonally adjacent pixels of the ciphertext image, NPCR values, and UACI values facing a chosen plaintext attack.

Some problems exist with end-to-end key generation and image encryption based on the deep learning model. For example, the histogram of the generated key and ciphertext image is not seriously randomly distributed, and as can be seen in Table 11, the UACI values facing chosen ciphertext or plaintext attacks is not high in general. Although Ref. 54 proved that the employment of traditional diffusion techniques to Cycle-GAN-based image encryption methods could enhance the ability against chosen plaintext attack, the complexity of the image encryption methods also increased. Using a deep learning method to take the place of the diffusion algorithm is a possible solution to reduce the time costs. Then the decrypted image quality could be further enhanced. In addition, the deep learning model needs a lot of training time, and the huge amount of model calculations results in a poor encryption/decryption speed. Thus the training and encryption/decryption efficiency of the end-to-end image encryption techniques based on Cycle-GAN could be further improved. Furthermore, because the model is trained according to a specific dataset, the generalization ability of the encryption and decryption model should be further discussed, analyzed, and improved. However, the automatic generation of keys and ciphertext images using deep learning methods has the advantages of convience, large key space and reduced reliance on complex cryptography design knowledge. How to realize the image cryptography on an even more profound level using deep learning models and thus making a breakthrough is still a potential research direction in the future.

4.6.

Image Cryptanalysis Method Based on Deep Learning

Some progress of image cryptanalysis using deep learning has been made, especially in optical image encryption. The deep learning model is trained with large ciphertext and corresponding plaintext images to learn the ability to crack the optical image encryption method.

Because of the nonlinear operation of phase truncation, the cryptography based on phase truncated Fourier transforms (PTFT)¹³¹ has high robustness against existing attacks.

Figure 15 illustrates the encryption processes of PTFT, assuming that the original image is $f(x)$ $\mathrm{FT}(·)$ and $\mathrm{IFT}(·)$ are the Fourier transform and inverse Fourier transform, respectively. The Fourier transform is given in Eq. (3). Equation (4) gives the phase truncation operation $\mathrm{PT}(·)$. Suppose that $R1(x)$ and $R2(u)$ are a pair of independent random phase masks; the encryption procedure of ciphertext image $g(x)$ is seen in Eqs. (5) and (6):¹³¹

Fig. 15

Process diagram of phase truncation Fourier transform encryption.

Xu et al.¹³² proposed a deep learning method to attack the PTFT encryption system¹³¹ and used a dataset with pairs of plaintext images on the MNIST handwritten dataset¹³³ and corresponding ciphertext images constructed through the PTFT encryption system to train residual network,⁹ which automatically learned the decryption characteristics of the encryption system by reducing the MSE between the decrypted images obtained by the deep learning model with the secret image as shown in Fig. 16. However, the quality of recovered images obtained by this method was not good.

Fig. 16

The training process of deep learning decryption network for phase truncation Fourier transform encryption.

Hai et al.¹³⁴ trained a neural network to crack the random phase encoding-based optical cryptosystems.¹³⁵ Wu et al.¹³⁶ trained a model that involved a module for obtaining the features of the ciphertext image and a module for recovering the plaintext image according to the obtained features with a large numbers of ciphertext-plaintext image pairs to attack the modified diffractive-imaging-based image encryption¹³⁷ cryptosystem. Chen et al.¹³⁸ demonstrated a CNN trained with a large amount of ciphertext image data encrypted by the joint transform correlation structure and its corresponding plaintext image, directly converting the ciphertext image to the original plaintext image.

Deep learning also shows a certain ability to detect or crack other encryption methods. He et al.¹³⁹ mapped the ciphertext images encrypted by the chaos-based image encryption algorithm demonstrated in Ref. 140 into the low-dimensional space and then regenerated visually consistent decrypted images utilizing a deconvolutional generator.

5.1.

Image Forgery Detection

Bondi et al.¹⁴¹ employed a CNN trained on the Dresden image database¹⁴² to extract characteristic camera model features, which were analyzed through iterative clustering techniques for image tampering detection and localization using characteristic footprints left on images by different camera models. Elaskily et al.¹⁴³ exploited a CNN trained on the MICC-F220,¹⁴⁴ MICC-F2000,¹⁴⁴ MICC-F600,¹⁴⁵ and SATs-130¹⁴⁶ datasets to extract features from images to detect the copy-move forgery. Diallo et al.¹⁴⁷ presented a camera identification CNN model trained with a mixture of different qualities of compressed and uncompressed images on the Dresden dataset¹⁴² for image forgery detection. Patil and Jariwala⁶³ carried out the intensive and incremental learning phase and then implemented a hybrid CNN to detect the image and video forgery. Bappy et al.¹⁴⁸ introduced a manipulation localization method using resampling features, long-short-term memory cells, and an encoder–decoder network to segment out manipulated regions from nonmanipulated ones. Xiao et al.¹⁴⁹ suggested a splicing forgery detection algorithm with diluted adaptive clustering and a coarse-to-refined CNN trained on the CASIA,¹⁵⁰ COLUMB,¹⁵¹ and FORENSICS¹⁵² datasets, which cascaded a coarse CNN and a refined CNN and extracted the differences in the image properties between untampered and tampered regions from image patches with different scales. However, the detection only focused on a single tampered region in an image owing to a restriction of the postprocessing approach. Zhang and Ni¹⁵³ employed a cross-layer intersection mechanism to dense U-Net⁸⁵ for image forgery detection and localization. Biach et al.¹⁵⁴ described an encoder using an architecture that was topologically the same as that of Resnet-50;⁹ it analyzed the discriminating characteristics between the manipulated and nonmanipulated regions, and a decoder localized the manipulated regions. However, there were a few poorly detected images, especially on the NIST’16¹⁵⁵ dataset. Moulin and Goel¹⁵⁶ derived locally optimal statistical tests for identifying forgeries and showed a procedure for learning a forgery detector trained on the CIFAR-10¹⁵⁷ and MNIST handwritten datasets. To combat image recapture attacks such as recapturing high-quality images from high-fidelity liquid crystal display screens, Zhu et al.¹⁵⁸ described a recaptured image detection method based on CNN in which the local binary patterns coding coded maps were extracted as the input (Table 12).

Table 12

Some deep learning models for image authentication tasks.

$F_1$-score takes both false negatives and false positives into account. Table 13 shows that Ref. 154 achieves the highest $F_1$-score and has the best performance in image forgery detection among all compared methods.

Table 13

F1-score comparisons of different image forgery detection methods on the CASIA151 and NIST’16156 datasets.

Although deep learning has achieved good results in image forgery detection for several types of forgery, there is shortage of large and perfect datasets that include images tampered by methods for more types of forgery and research studies on deep learning forgery detection methods for more complete forgery types.

5.2.

Watermarked Image Generation

To dynamically adapt image watermarking algorithms, deep learning-based image watermarking schemes have attracted increased attention, and experiments and estimation results have confirmed the advantages of the deep learning mechanisms in image watermarking. Vukotić¹⁵⁹ investigated a new family of transformations based on deep learning networks that were useful in image watermarking. As can be seen in Fig. 17, Kandi et al.¹⁶⁰ proposed an autoencoder CNN for watermark embedding and extraction; the attack layer simulated different attacks, and the strength factor controlled the level of watermarked images robustness versus imperceptibility. Fierro-Radilla et al.¹⁶¹ demonstrated a zero-watermarking algorithm in which features of the image were gained by the CNN and combined with the watermark sequence using the XOR operation. Ahmadi et al.¹⁶² described two fully CNNs with the residual structure trained on the CIFAR-10 and Pascal VOC2012⁹³ datasets for watermarks embedding and extraction. Mun et al.¹⁶³ exploited a reinforcement learning trained on the BOSSbase dataset for robust and blind watermarking. Zhong et al.¹⁶⁴ introduced an encoder to encode the watermark and input the result into an embedder with the cover image to reach the watermarked image, with the encoder and embedder being two CNNs trained on the ImageNet and CIFAR¹⁵⁷ datasets. Zhang et al.¹⁶⁵ exploited an embedding network and an extractor network to embed and gain the watermark, respectively, and a surrogate network to boost the watermark, revealing ability of an extractor network; these were trained on the PASCAL VOC⁸² and Chestx-ray8¹⁶⁶ datasets. However, the method was not robust enough to some preprocessing techniques such as random cropping and resizing.

Fig. 17

The watermarking framework of Ref. 160.

Table 14 compares the bit error rate (BER) for the method recovering watermark information against normal digital image manipulations (including 20% cropping, 5% pepper and salt noise, and lossy JPEG compression with factor 10), capacity, and the PSNR between the container image and the watermarked image gained by Refs. 161162.163.–164. The higher the BER value is, the more robust the watermark recovery ability of the methods is. The higher the PSNR values between the container image and the watermarked image is, the more imperceptible the method is.

Table 14

Comparison of different deep learning techniques in PSNR between watermarked images and container images, capacity, and BER for the methods recovering watermark information against normal digital image processing operations.

As can be seen in Table 14, different methods have their distinctive advantages. For example, Ref. 161 has better resistance in JPEG compression with factor 10, whereas Ref. 162 has a higher PSNR index between watermarked images and original images, and Ref. 164 has a better resistance in 5% salt and pepper noise and 20% cropping and larger capacity. As a result, a deep learning method in image watermarking that has a high resistance to multiattacks and a good imperceptibility of the watermarked image while having a large capacity needs to be further developed in future research.

5.3.

Image Watermark Extraction and Detection

Li et al.¹⁶⁷ extracted a watermark image from a single-frame watermarked hologram by a conditional GAN trained on the fashion-MNIST¹⁶⁸ and MNIST handwritten datasets. Huynh-The et al.¹⁶⁹ exploited a blind image watermarking framework based on a deep convolutional encoder–decoder network watermark extraction model trained with various attacked watermarked images on the BOSSbase v1.01 database. Li et al.¹⁷⁰ embedded a processed watermark image into the block discrete cosine transform component and used a cooperative neural network to recognize the suspected watermark signal with a single hologram, which improved the transmission efficiency. Hayes et al.¹⁷¹ learned a transformation-resilient watermark detector trained on the CIFAR-10 and ImageNet datasets to detect watermarks that could be employed in various carriers such as in the image, audio, and video domains. Chen et al.¹⁷² performed a simulated process to generate a large number of distorted watermarks and then collected them to form a training dataset to train a CNN model that could accurately identify the watermark copyright.

5.4.

Image Watermarking Attack

Wang et al.¹⁷³ trained a watermark faker based on U-Net trained on the Caltech256¹⁷⁴ dataset with the input being an original image and the output being a fake watermarked image after preprocessing; a set of paired images of original and watermarked images was obtained by the targeted image watermarking algorithms. However, this method did not perform well at generating the watermark in the frequency domain. Hatoum et al.¹⁷⁵ employed a fully CNN trained on th BOSSbase dataset to denoise watermarked images and destroy the watermarks while preserving a satisfied quality of the denoised images. Sharma and Chandrasekaran¹⁷⁶ implemented an enhanced hybrid watermarking scheme using DWT and singular value decomposition methods and proposed an adversarial attack based on a CNN-based autoencoder scheme trained on the CIFAR-10 database that could produce a perceptually close image.

5.5.

Image Watermark Removal

As can be seen in Fig. 18, for the visible watermark removal task, deep learning models often consist of two parts:¹⁷⁷ watermark detection and removal. First, the deep learning model detects the watermark object in the watermarked images and then removes the watermark object from the watermarked image to obtain the watermark-less image. Gandelsman et al.¹⁷⁸ proposed a coupled “Deep-Image-Prior” network to remove the image watermark that needed no training examples other than the input image/video. Hertz et al.¹⁷⁹ trained on the Microsoft COCO val2014 dataset⁷⁹ and learned to separate the visual motif from the image by estimating the visual motif matte and reconstructing the latent image for blind removal of both opaque and semitransparent motifs. Li et al.¹⁸⁰ suggested a watermark processing framework using the conditional GAN trained on a large-scale visible watermark dataset¹⁷⁷ and the PASCAL VOC2012 dataset for visible watermark removal in a real-world application. The generated watermark-less image had photorealistic quality but not good performance in standard quantitative evaluation metrics such as PSNR. Jiang et al.¹⁸¹ presented a watermark removal structure consisting of a watermark extraction network that removed the watermark in the watermarked image and an image inpainting network that inpainted the image for a watermark-less image. The two networks were trained on the PASCAL VOC2012 and places2¹⁸² datasets. Cun and Pun¹⁸³ introduced a multitask feature extractor and a watermarked region smoother incorporated with multiple perceptual losses trained on the VAL2014 subset of the MSCOCO⁷⁹ dataset and a dataset of logos to simulate the procedure of image watermark detection, removal, and refinement. However, when the detection failed or the textures in the watermark and background were similar, the network could not remove the watermark perfectly.

Fig. 18

The general process of deep learning methods for watermark removal.

To remove the image watermark generated by the deep learning model, many scholars have proposed their methods from different perspectives. Shafieinejad et al.¹⁸⁴ focused on backdoor-based watermarking, removingd the watermark fully by just relying on public data and proposing an attack that detected whether a model contained a watermark trained on th MNIST and CIFAR-10 datasets. Chen et al.¹⁸⁵ exploited a unified watermark removal framework based on fine-tuning and incorporated it with an adaption of the elastic weight consolidation algorithm and unlabeled data augmentation. William et al.¹⁸⁶ described a neural network “laundering” algorithm to remove black-box backdoor watermarks from neural networks trained on the MNIST and CIFAR-10 datasets.