Recent advances in deep-learning-enhanced photoacoustic imaging

Abstract. Photoacoustic imaging (PAI), recognized as a promising biomedical imaging modality for preclinical and clinical studies, uniquely combines the advantages of optical and ultrasound imaging. Despite PAI’s great potential to provide valuable biological information, its wide application has been hindered by technical limitations, such as hardware restrictions or lack of the biometric information required for image reconstruction. We first analyze the limitations of PAI and categorize them by seven key challenges: limited detection, low-dosage light delivery, inaccurate quantification, limited numerical reconstruction, tissue heterogeneity, imperfect image segmentation/classification, and others. Then, because deep learning (DL) has increasingly demonstrated its ability to overcome the physical limitations of imaging modalities, we review DL studies from the past five years that address each of the seven challenges in PAI. Finally, we discuss the promise of future research directions in DL-enhanced PAI.


Introduction
Photoacoustic imaging (PAI) is a noninvasive and radiation-free biomedical imaging modality that provides high spatial resolution, deep penetration, and great optical absorption contrast by synergistically combining optics and acoustics. 1PAI is based on the photoacoustic (PA) effect, in which optical energy from a pulse laser is converted into acoustic energy waves by the light absorption characteristics of biomolecules. 2The initial pressure of a generated PA wave can be calculated as where p 0 denotes the initial pressure, Γ is the Gruneisen coefficient, μ a is the optical absorption coefficient, η th is the efficiency of heat conversion from the optical absorption, and F is the optical fluence.Because acoustic scattering in biological tissues is several orders of magnitude lower than light scattering, PAI can obtain biomolecule information based on the absorption contrast of light at a depth of several centimeters. 3,4[7][8][9][10][11][12][13] In particular, oxygen saturation (sO 2 ), an important index for evaluating various diseases, is calculated through HbO and HbR values. 142][23][24] Not limited to endogenous chromophores, the high molecular sensitivity of PAI enables molecular imaging when exogenous contrast agents are administered. 4][30] Photoacoustic microscopy (PAM) and photoacoustic computed tomography (PACT) are the main PAI modalities.2][33][34][35][36] Optical resolution PAM (OR-PAM), which implements focused optical illumination on the acoustic focal area, shows high spatial resolution (a few micrometers) and has been applied to investigate small biological structures. 37,38Acoustic resolution PAM (AR-PAM) uses a less tightly focused optical beam than OR-PAM, but its acoustic focus is smaller than its laser focus. 39AR-PAM achieves deeper light penetration (up to several centimeters, compared to the 1 mm depth in OR-PAM) despite its lower spatial resolution (tens/hundreds of micrometers), defined by the acoustic focus.PACT uses multiple detection positions to simultaneously reconstruct an image in 2D or 3D.It provides hundreds of images with micrometer-level spatial resolution at imaging depths ranging in the tens of millimeters.1][42] Images are created by reconstruction algorithms, 43 such as delay-and-sum (DAS), 44,45 delay-multiplyand-sum (DMAS), 46 backprojection (BP), 47 Fourier beam forming, 48 time reversal (TR), 49 and model-based methods. 50,51AI has gained widespread recognition as a promising biomedical imaging modality for preclinical and clinical studies.However, to fully realize PAI's great potential, the seven challenges listed in Table 1 and illustrated in Fig. 1 must be addressed to further enhance the image quality and expand PAI's applications: (1) The first challenge, overcoming limited detection capability, arises because most PAI systems are still constrained by such factors as restricted bandwidth, a limited detection view, and sampling sparsity. 592) The second challenge is to compensate for low-dosage light delivery.The PACT systems based on LEDs or laser diodes are portable and cost-effective alternatives to bulky and expensive solid-state laser systems.However, their low-dosage light delivery provides only a low signal-to-noise ratio (SNR), which can affect image quality.In the case of OR-PAM, fast scanning with high repetition rates is necessary for certain applications such as recording brain-wide neuronal activities. 60However, to ensure laser safety, low laser dosages are required, resulting in reduced SNRs and decreased image qualities.
(3) The third challenge is to improve the accuracy of quantitative PA imaging.Accurately determining physiological parameters remains a demanding task due to the complex and nonlinear nature of light absorption and scattering. 614) The fourth challenge is to optimize or replace current reconstruction methods, whose inherent limitations compromise their accuracy and effectiveness in generating high-quality images.
(5) The fifth challenge is to address the problems posed by tissue heterogeneity.Local variations in the acoustical properties of biological tissue can lead to inconsistencies in the reconstructed PA images, resulting in artifacts that degrade the accuracy of quantitative measurements derived from the images. 626) The sixth challenge is to improve the classification and segmentation accuracy of PA images.The limited availability of annotated PAI data sets has hindered the development of automated image classification and segmentation, resulting in either continued reliance on manual delineation by expert physicians or the adaptation of traditional methods from other imaging modalities.(7) In addition to the six challenges mentioned earlier, there are still specific issues, such as motion artifacts, limited spatial resolution, electrical noise, image misalignment, accelerating superresolution imaging, and achieving digital histologic staining, which are also important for PAI studies.To ensure a comprehensive understanding of the challenges in PAI, these specific issues are categorized as a seventh challenge.These seven challenges are summarized in Table 1.
Overcoming these challenges is important because relying solely on hardware improvements will not be enough to resolve them.It will require significant investments of time and resources to find effective solutions.Deep learning (DL) plays a crucial role in advancing the field of medical and bioimaging by not only addressing the inherent limitations of imaging systems but also by driving substantial improvements in classification and segmentation performance.In recent years, DL has gained significant traction in PAI research, leading to remarkable breakthroughs and achievements.This comprehensive review article provides an in-depth analysis of diverse methodologies and outcomes showcasing the utilization of DL techniques to effectively address the seven challenges encountered in PAI, as previously outlined.

Principles of DL Methods
DL is a subset of machine-learning algorithms that encompasses supervised learning, unsupervised learning, and reinforcement learning.Supervised learning is a modeling technique that establishes a correlation between input data and their corresponding ground truth (GT).This approach is commonly utilized in DL-enhanced medical imaging, where high-and low-quality images can be paired.On the other hand, unsupervised learning identifies specific patterns hidden within data, without the use of labeled examples or a priori known answers.Lastly, in reinforcement learning, an algorithm maximizes the final reward by learning through rewards obtained as a result of performing specific actions in a particular environment.A notable example of such a learning algorithm is AlphaGo, the first computer program to beat a human champion Go player. 63Next, we explain the basic structure of DL network and the basic operating principle of DL training.In addition, we introduce the representative DL architectures that are most widely used in the field of image and video: convolution neural network (CNN), U-shaped neural network (U-Net), and generative adversarial network (GAN) architecture.

Artificial Neural Networks
Artificial neural networks (ANNs) draw inspiration from biological neural networks, wherein different stimuli enter neurons through dendrites and are transmitted to other cells through axons once a threshold of activation is achieved [Fig.2(a)].
In ANNs, an artificial neuron is a mathematical function conceived as a biological neuron.This function multiplies the various inputs by each weight, sums them, and adds the deviation.This sum is then sent to a specific activation function and produces an output [Fig.2(b)].This artificial neuron is expressed as where σ is the activation function, x is the input, w is the weight, b is the variance, and y is the output.The calculated output acquires nonlinearity through such activation functions (σ) as sigmoid, hyperbolic tangent, and rectified linear unit. 64Without such an activation function, it would be pointless to build a deep model, since a linear transformation would occur regardless of how many hidden layers are present.ANNs typically have an input layer, a hidden layer, and an output layer, each comprising multiple units.The number of hidden layers between the input and output determines whether the ANN is a simple neural network or a deep neural network (DNN) [Fig.2(c)].Formulaically, the two networks in Fig. 2(c) are described as (3) and

Backpropagation
Backpropagation serves as the fundamental principle for training DL models.In order to grasp the concept of backpropagation, it is necessary to first understand forward propagation.Forward propagation involves sequentially passing an input value through multiple hidden layers to generate an output.For instance, given an input x, a weight w, a variance b, and an activation function σ, the DNNs' forward propagation on Fig. 2(c) is represented as where y Ã represents the value predicted by DNNs.The process of finding the optimized w and b variables that minimize the loss, which is the difference between the predicted result and the actual y, is called training.The loss is calculated for all training data sets.As loss-function metrics, image data sets such as PA images commonly employ the structural similarity index measure (SSIM) and peak-signal-to-noise ratio (PSNR).Backpropagation refers to the process of transmitting the loss back to the input stage using the chain rule. 65,66This process determines the weight w that yields the minimal loss function value.Most DL models adopt the gradient descent technique throughout this procedure. 67By calculating the gradient at a specific w and continually updating w, the loss function's minima can be estimated using the following equation: The learning rate, a hyperparameter that determines the variable's update amount in proportion to the calculated slope, is set before the learning process and remains unchanged.The number of hidden layers and their dimensions are also hyperparameters.In the following section, we introduce representative ANN architectures commonly used in biomedical imaging, including PAI.

CNN
CNNs, the most basic DL architecture, have received significant attention in the field of PAI due to their extensive use in image processing and computer vision.CNNs were developed to extract features or patterns in local areas of an image.A convolution operation is a mathematical process that measures the similarity between two functions.The convolution operation in image processing is the process of calculating how well a subsection of an image matches a filter (also referred to as a kernel) and is used for things such as edge filtering.In CNN, the network is trained by learning this filter, which is used to extract image features, as a weight.The encoder-decoder CNN architecture is a relatively simple network structure capable of performing image-to-image translation tasks [Fig.3(a)]. 68nitially, the input image undergoes a series of downsampling and convolution operations.Throughout this process, the dimensions of the image progressively decrease while the number of image channels, representing an additional dimension, increases.
This results in a bottleneck in the representation, which is subsequently reversed through a sequence of upsampling and convolutional operations.The bottleneck enforces the network to encode the image into a compact set of abstracted variables (also referred to as latent variables) along the channel dimension.The predicted image is synthesized by decoding these variables in the second half of the network.

U-Net
A U-Net [Fig.3(b)] is a CNN-based model that was originally proposed for image segmentation in the biomedical field. 69The U-Net is composed of two symmetric networks: a network for obtaining overall context information of an image and a second network for accurate localization.The left part of the U-Net is the encoding process, which encodes the input image to obtain overall context information.The right part of U-Net is the decoding process, which decodes the encoded context information to generate a segmented image.The feature maps obtained during the encoding process are concatenated with up-convolved feature maps at each expanding step in the decoding process, using skip connections. 70This enables the decoder to make more accurate predictions by directly conveying important information in the image.As a result, the U-Net architecture has shown excellent performance in several biomedical image segmentation tasks, even when trained on a very small amount of data, due to data augmentation techniques.

GAN
A GAN is a type of generative model that learns data through a competition between a generator network and a discriminator network [Fig.3(c)]. 71,72The generator network generates the fake data, and the discriminator network tries to distinguish the real data from the fake data.To deceive the discriminator, the generator aims to generate data that look as realistic as possible, while the discriminator attempts to distinguish the real data from the realistic fake data.Through this competition, both networks learn and improve iteratively, resulting in a generator that can generate increasingly realistic data.As a result, GANs have been successful in generating synthetic data that are very similar to real data, making them useful for applications such as data augmentation and image synthesis.These three representative networks are summarized in Table 2. 3 Challenges in PAI and Solutions through DL

Overcoming Limited Detection Capabilities
In PAI, optimal image quality requires a broadband US transducer and dense spatial sampling to enclose the target. 43,61,73owever, real-world scenarios introduce limitations, such as limited bandwidth, limited view, and data sparsity.DL methods have been used as postprocessing techniques to overcome these limitations and enhance the PA signals or images, reducing artifacts.This section provides an overview of studies utilizing DL methods as PAI postprocessing methods.The DL-based image reconstruction studies are discussed separately in Sec.3.4.

Limited bandwidth
The bandwidth of US transducer arrays is limited compared to the natural broadband PA signal (from tens of kilohertz to a hundred megahertz). 74Although optical detectors of PA waves have expanded the detection bandwidth, manufacturing high-density optical detector arrays and adopting them to PACT remains a technical challenge. 75o solve the limited-bandwidth problem, Gutte et al. proposed a DNN with five fully connected layers to enhance the PA bandwidth [Fig.4(a)]. 76The network takes a limited-bandwidth signal as input and outputs an enhanced bandwidth signal, which is then used for PA image reconstruction using DAS.To train the network, the authors generated numerical phantoms using the k-Wave toolbox 79 to create pairs of full-bandwidth and limited-bandwidth PA signals.The synthesized results from the numerical phantoms demonstrated an enhanced bandwidth that is like that of images obtained from the full-bandwidth signal [Fig.4(a)].

Limited view
PA image quality is reduced by the scant information provided by the limited coverage angle of the PA signals detected by the US transducer. 3This problem is commonly encountered in PACT systems, particularly in linear US array-based systems and is referred to as the "limited view" problem. 75,80Researchers have addressed this problem to some extent by developing reconstruction methods with iterative methods. 80Recent studies based on the fluctuation of the PA signal of blood flow or microbubbles 81,82 show another effective solution, but they need a number of single images to reconstruct one fluctuation image, which compromises the temporal resolution.
Deng et al. 83 developed DL methods using U-Net and principal component analysis processed very deep convolutional networks (PCA-VGG) 84 while Zhang et al. 85 designed a dual domain U-Net (DuDoUnet) incorporating reconstructed images and frequency domain information.In addition to utilizing the U-Net architecture, researchers have also explored the use of GAN networks, which have garnered attention due to their ability to preserve high-frequency features and prevent oversmoothing in images.Lu et al. proposed a GAN-based network called the limited view GAN (LV-GAN). 77Figure 4(b) shows the architecture of LV-GAN, which consists of two networks: the generator network responsible for generating high-quality PA images from limited-view images, and the discriminator network designed to distinguish the generated images from the GT.To ensure accurate and generalizable results, the LV-GAN was trained using both simulated data generated by the k-Wave toolbox and experimental data obtained from a custom-made PA system.The results presented in Fig. 4(b), using ex vivo data, demonstrate the ability of LV-GAN to successfully reconstruct high-quality PA images in limited-view scenarios.The quantitative analysis further confirms that LV-GAN outperforms the U-Net framework, achieving the highest retrieval accuracy.
The combination of a postprocessing method with direct processing using PA signals is considered as another approach to reduce artifacts in limited-view scenarios.Lan et al. 78 designed a new network architecture, called Y-net, which reconstructs PA images by optimizing both raw data and reconstructed images from the traditional method [Fig.4(c)].This network has two inputs, one from raw PA data and the other from the traditional reconstruction.It combines two encoders, each corresponding to one of the input paths, with a shared decoder path.The training data were generated by the k-Wave toolbox with a linear array setup.The public vascular data set 86 was used to generate PA signals.They compared the proposed method with conventional reconstruction methods [e.g., DAS and time reversal (TR)] and other DL methods such as U-Net.In in vitro and in vivo experiments, the proposed method showed superior performance to the other methods, with the best spatial resolution.

Sparsity
To achieve the best image quality, the interval between two adjacent positions of the transducer or array elements must be less than half of the lowest detectable acoustic wavelength, according to the Nyquist sampling criterion. 87In sparse sampling, the actual detector density is lower than this requirement, introducing streak-shaped artifacts in images. 74Sparse sampling can also result from a trade-off between image quality and temporal resolution, which is sometimes driven by system cost and hardware limitations. 88o remove artifacts caused by data sparsity, Guan et al. 89 added additional dense connectivity into the contracting and expanding paths of a U-Net.Farnia et al. 90 combined a TR method with a U-Net by inserting it in the first layer.Guo et al. 91 built a network containing a signal-processing method and an attention-steered network (AS-Net).Lan et al. 92 proposed a knowledge infusion GAN (Ki-GAN) architecture that combines DAS and PA signals for reconstruction from sparsely sampled data.DiSpirito et al. 93 compared various CNN architectures for PAM image recovery from undersampled data of in vivo mouse brains. 94They chose a fully dense U-Net (FD U-Net) with a dense block, allowing PAM image reconstruction using just 2% of the original pixels.Later, they proposed a new method based on a deep image prior (DIP) method 95 to solve this problem without pretraining or GT data.

Combinational limited-detection problems
Previous studies have primarily tackled individual issues in isolation, neglecting the simultaneous occurrence of multiple limited-detection challenges in PA systems. 74However, researchers have recently focused on utilizing a single NN to address two or three limited-detection problems concurrently, leading to promising advancements in this area.
For linear array, Godefroy et al. 96 incorporated dropout layers 97 into a modified U-Net and further built a Bayesian NN to improve the PA image quality.Vu et al. 98 built a Wasserstein GAN (WGAN-GP) that combined a U-Net and a deep convolutional GAN (DCGAN). 99The network reduced limited-view and limited-bandwidth artifacts in PACT images.For a ring-shaped array, Zhang et al. 100 developed a 10-layer CNN, termed a ringarray DL network (RADL-net), to eliminate limited-view and under-sampling artifacts in photoacoustic tomography (PAT, also known as PACT) images.Davoudi et al. 101 proposed a U-Net network to improve the image quality from sparsely sampled data from a full-ring transducer array.They later updated their U-Net architecture 102 to operate on both images and PA signals.Awasthi et al. 103 proposed a U-Net architecture to achieve superresolution, denoising, and bandwidth enhancements.They replaced the softmax activation function in the final two layers of the U-Net for segmentation with an exponential linear unit. 104Schwab et al. proposed a network that combined the BP with dynamic aperture length (DAL) correction, which they called DALnet 105 to address the limited-view and undersampling issues in the 3D imaging PACT system.
One of the notable achievements in applying DL to the 3D-PACT system was made by Choi et al. 52 They introduced a 3D progressive U-Net (3D-pUnet) as a solution to address limitedview artifacts and sparsity caused by clustered-sampling detection, as shown in Fig. 4(d).The design of their network was inspired by the progressive growth GAN, 106 which utilizes a progressively increasing procedure to optimize a U-Net.In their 3D-pUnet, subnetworks were trained sequentially using downsampled data from the original high-resolution volume data, gradually transferring knowledge obtained from each progressive step.
The training data set consisted of in vivo experimental data from rats, and the results demonstrated superior performance compared with the conventional 3D-U-Net method.Interestingly, they demonstrated that the 3D-pUnet trained cluster-sampled data set also works in sparsely sampled data sets.The proposed approach was also applied to predict dynamic contrast-enhanced images and functional neuroimaging in rats, achieving increased imaging speed while preserving high image quality.In addition, they demonstrated the ability to accurately measure physiological phenomena and enhance structural information in untrained subjects, including tumor-bearing mice and humans.
All the research reviewed in this section is summarized in Table 3.

Compensating for Low-Dosage Light Delivery
Pulsed laser sources, such as an optical parametric oscillator laser system with a Nd:YAG pumped laser, are commonly used in PACT systems to achieve deep penetration with a high SNR, but those laser systems are bulky and expensive. 2,107In recent years, researchers have explored compact and less expensive alternatives, such as pulsed-laser diodes 107 and light-emitting diodes (LEDs). 108While these alternatives have shown promising results, their low pulse energy results in a low SNR, requiring frame averaging to increase image quality.Unfortunately, this method comes at a cost, as it reduces imaging speed.Furthermore, in dynamic imaging, frame averaging can cause blurring or ghosting due to the movement of the object being imaged.To address these problems, DL methods can be applied to enhance image quality in situations where the light intensity is low.
One of the representative works for LED-based systems was achieved by Hariri et al. 53 They proposed a multilevel waveletconvolutional NN (MWCNN) that could map the low-fluence PA images to high-fluence PA images from an Nd:YAG laser system.This approach helps to eliminate the background noise while preserving the structures of the target, as shown in Fig. 5(a).Phantom and in vivo studies were conducted to assess the performance of their model.The MWCNN demonstrated a significant improvement in contrast-to-noise ratio (CNR) with up to a 4.3-fold enhancement in the phantom study and a 1.76-fold enhancement in the in vivo study.These results highlight the practicality of the proposed method in real-world scenarios.Singh et al. 111 and Anas et al. 112 proposed a U-Net and a deep CNN-based approach to improve the image quality with a similar system setup.Anas et al. later introduced a recurrent neural network (RNN) 113 to further improve the system's performance. 114o enhance the image quality in a pulsed-laser-diode PA system, Rajendran et al. 109 proposed a hybrid dense U-Net (HD-UNet) [Fig.5(b)].To train the network, they generated simulated data using the k-Wave toolbox, and evaluated the model with both single-and multi-US transducer (1-UST and multi-UST-PLD) PACT systems, using both phantom and in vivo images.Compared with their previous system, the HD-UNet improved the imaging speed by approximately 6 times in the 1-UST system and 2 times in the multi-UST-PLD system.To address the challenges of balancing laser dosage, imaging speed, and image quality in OR-PAM, Zhao et al. 110 proposed a multitask residual dense network (MT-RDN) that performs image denoising, superresolution, and vascular enhancement [Fig.5(c)].The network comprises three subnetworks, each using an independent RDN framework and assigned a supervised learning task.The first subnetwork     processes the data of input 1 (i.e., 532 nm data) to obtain output 1, and the second subnetwork processes the data of input 2 (i.e., 560 nm data) to obtain output 2. These outputs are then combined and processed by subnetwork 3, and the differences between the outputs and the GT are compared.
To train the network, input images were undersampled at half-per-pulse laser energy of the GT, while the GT images were sampled at the full ANSI per-pulse fluence limit.To evaluate the performance of the proposed method, U-Net and RDN were used.The MT-RDN method achieved a 16-fold reduction in laser dosage at 2 times data undersampling and a 32-fold reduction in dosage at 4 times undersampling compared to the GT images.
All the research reviewed in this section is summarized in Table 4.

Improving the Accuracy of Quantitative PAI
Quantitative photoacoustic imaging (qPAI) quantifies molecular concentrations in biological tissue using multiwavelength PA images, enabling the estimation of various endogenous and exogenous contrast agents and physiological parameters, such as sO 2 . 61However, qPAI presents significant challenges due to the wavelength-dependent nature of light absorption and scattering, leading to varying levels of light attenuation across different wavelengths. 2,115Thus, it is hard to accurately determine the fluence distribution, which is nonlinear and complex in biological tissues.Early research in qPAI assumed constant optical properties of biological tissue and uniform parameters such as the scattering coefficient throughout the imaging field. 61However, recent studies have shown that these assumptions lead to errors, especially in deep-tissue imaging. 116Model-based iterative optimization methods have been developed to address this issue and provide more accurate solutions. 117But these methods are time-consuming and sensitive to quantification errors. 118A new approach called eigenspectral multispectral optoacoustic tomography (eMSOT) has been proposed to improve qPAI accuracy. 116MSOT formulates light fluence in tissues as an affine function of reference base spectra, leading to improved accuracy in qPAI.However, it requires ad hoc inversion and has limitations in scale invariance.
Researchers have pursued multiple avenues to extract fluence distribution information from multiwavelength PA images using DL architectures.Cai et al. 119 introduced ResU-Net, which adds a residual learning mechanism to the U-Net.Chang et al. 120 developed DR2U-Net, a fine-tuned deep residual recurrent U-Net.Luke et al. 121 combined two U-Nets to create a new network called O-Net, which segments blood vessels and estimates sO 2 .A novel DL architecture that contains an encoder, decoder, and aggregator was introduced by Yang et al. 122 termed called EDA-Net.The encoder and decoder paths both feature a dense block, while the aggregator path incorporates an aggregation block.Gröhl et al. 123 designed a nine-layer fully connected NN that directly estimates sO 2 from PA images.All showed much high accuracy in estimating sO 2 distributions or other molecular concentrations compared with linear unmixing.
One of the representative results is from Ref. 124.Researchers built two separated convolutional encoder-decoder type networks with skip connections, termed EDS to solve this problem in 3D conditions [Fig.6(a)].One network was trained to output images of sO 2 from 3D-image data and the other network was trained to segment vessels.By leveraging the spatial information present in the 3D images, the 3D fully convolutional networks could produce precise sO 2 maps.Besides getting more accurate sO 2 results, these networks were able to handle limiteddetection capabilities, such as limited-view artifacts, and showed promise for producing accurate estimates in vivo.
Researchers have also employed DL methods to recover the absorption coefficient from reconstructed PA images.Chen et al. 126 proposed a U-Net-based DL network to recover the optical absorption coefficient and Grohl et al. 127 adapted a U-Net to compute error estimates for optical parameter estimations.A notable contribution was made by Li et al. in a recent study. 54hey addressed the challenge of insufficient data-label pairs in qPAI by introducing two DNNs, depicted in Fig. 6(b).First, they introduced a simulation-to-experiment end-to-end data translation network (SEED-Net) that provides GT images for experimental images through unsupervised data translation from a simulation data set.They then designed a dual-path network based on U-Net (QPAT-Net) to reconstruct images of the absorption coefficient for deep tissues.The QPAT-Net outperformed the previous QPAT method 128 in simulation, ex vivo, and in vivo, with more accurate absorption information and relatively few errors.
Another seminal study was done by Zou et al. 125 They developed the US-enhanced U-Net model (US-Unet), which combines information from US images and PA images to reconstruct the optical absorption distribution [Fig.6(c)].They implemented a pretrained ResNet-18 to extract features from US images of ovarian lesions.
This feature information was incorporated into a U-Net structure designed to reconstruct the optical absorption coefficient.The U-Net was trained on simulation data and subsequently tested on a phantom, blood tubes, and clinical data from 35 patients.The US-Unet outperformed both the U-Net model without US features and the standard DAS method in phantom and clinical studies, demonstrating its potential for improving accuracy in clinical PAI applications.
0][131] To this end, Madasamy et al. 132 compared the compensation performance of different DL models.The models tested included U-Net, 69 FD U-Net, 89 Y-Net, FD Y-Net, 78 deep residual U-Net (deep ResU-Net), 133 and GAN. 134Results showed the robustness of all DL models to noise and their effectiveness; FD U-Net showed the best performance.qPAI requires an unmixing process, which can be achieved through linear or model-based methods. 115Durairaj et al. 135 proposed an unsupervised learning approach using an initialization network and an unmixing network.Olefir et al. 136 introduced DL-eMSOT, combining eMSOT with a bidirectional RNN and CNN blocks for accurate sO 2 estimation and faster calculations.
All the research reviewed in this section is summarized in Table 5.

Optimizing or Replacing Conventional Reconstruction Algorithms
In PACT, the acoustic inverse problem involves reconstructing the PA initial pressure from raw data.Several reconstruction methods have been developed, including BP, 47 FB, 48 DAS, 44 DMAS, 46 TR, 49 and model-based methods. 51However, each method has limitations, and either to enhance existing reconstruction techniques or to directly reconstruct PA images using NNs, researchers have turned to DL methods.Various DL methods have been developed to convert PA raw data into images.One such method, called Pixel-DL, proposed by Guan et al., 137 uses pixel-wise interpolation followed by an FD U-Net for limited-view and sparse PAT image reconstruction [Fig.7(a)].The Pixel-DL model was trained and tested using simulated PA data from synthetic, mouse brain, lung, and fundus vasculature phantoms.It achieved comparable or better performance than iterative methods and consistently outperformed other CNN-based approaches for correcting artifacts.
To direct reconstruct PA images, Waibel et al. 139 introduced a modified U-Net that includes additional convolutional layers in each skip connection.Antholzer et al. 140 proposed a direct reconstruction process, based on a U-Net and a simple CNN, that can resolve limited-view and sparse-sampling issues.Lan et al. 141 proposed a modified U-Net, termed DU-Net, to reconstruct PA images using multifrequency US-sensor raw data.A noteworthy study of this topic is an end-to-end reconstruction network developed by Feng et al., 138 termed Res-U-Net [Fig.7(b)].They integrated residual blocks into the contracting and symmetrically expanding path of U-Net and added a skip connection between the input of raw data and the output of images.The training, validation, and test data sets were synthesized using the k-Wave toolbox.In digital phantom experiments, the Res-UNet showed performance superior to other reconstruction methods [Fig.7(b)].
Another representative work was done by Tong et al. 55 They proposed a novel two-step reconstruction process with a feature projection network (FPnet) and a U-Net [Fig.7(c)].The FPnet converts PA signals to images and contains several convolutional layers to extract features.There is one max pooling layer for downsampling and one full connection layer for domain transformation.The U-Net performs postprocessing to improve image quality.The resulting network, trained using numerical simulations and in vivo experimental data, outperformed other approaches to handle limited-view and sparsely sampled experimental data, exhibiting superior performance on in vivo experiments [Fig.7(c)].
In addition, Yang et al. 142 introduced recurrent inference machines (RIM), an iterative PAT reconstruction method using convolution layers.Kim et al. 143 employed upgUNET, a U-Net model with 3D transformed arrays for image reconstruction.Hauptmann et al. 144 proposed DGD, a deep gradient descent algorithm, outperforming U-Net and other model-based   methods.They also introduced fast-forward PAT (FF-PAT), a modified version of DGD, which addressed artifacts using a small multiscale network. 145ll the research reviewed in this section is summarized in Table 6.

Addressing Tissue Heterogeneity
Biological tissues are acoustically nonuniform, making it crucial to use a locally appropriate speed of sound (SoS) value for accurate PA reconstruction.SoS mismatch or a discontinuity in hard textured tissue can create acoustic reflection and imaging artifacts 74 that make it hard to detect the source of the PA signal, which is especially troublesome for interventional applications.DL methods have been used to detect point sources, remove reflections, and mitigate the difficulties presented by acoustic heterogeneity.
Highly echogenic structures can cause a reflection of a PA wave to appear to be a true signal, 146 which makes it hard to find point targets or real sources in PAI.Reiter et al. 147 trained a CNN to identify and remove reflection noise, locate point targets, and calculate absorber sizes in PAI.Later, Allman et al. 148 found Fast-RCNN 149 to be more effective than VGG16 84 for source detection and artifact elimination.Shan et al. 150 incorporated a DNN into an iterative algorithm to correct reflection artifacts, achieving superior results compared with other methods. 151,152eon et al. 56 proposed a generalized solution to mitigate SoS aberration in heterogeneous tissue by DL.They proposed a hybrid DNN model, named SegU-net, based on U-Net and SegNet 153 [Fig.8(a)].The architecture is similar to SegNet, but has an additional connection between the encoder and decoder through concatenation layers, like U-Net.The training data were generated using the k-Wave toolbox with different SoS values.They tested the model with phantoms with homogeneous media and in heterogeneous media.The proposed method showed better results than the multistencil fast marching 155 method and automatic SoS selection. 156It not only resolved the SoS aberration but also removed streak artifacts in images of healthy human limbs and melanoma.
All the research reviewed in this section is summarized in Table 7.

Improving the Accuracy of Image Classification and Segmentation
As PAI gains increasing attention in clinical studies, more accurate classification and segmentation methods are necessary to improve the interpretation of PA images.Image segmentation extracts the outline of objects within an image and identifies the distinct parts of the image that correspond to these objects. 158,159mage classification predicts a label for an image, identifying its content.In this section, we focus on segmentation and classification techniques. 158 Segmentation and classification are widely used in image postprocessing, and transfer learning has been utilized to take advantage of pretrained DNNs.Zhang et al. 160 used DL models AlexNet 161 and GoogLeNet 162 for PA image classification, outperforming support vector machine (SVM).Jnawali et al. 163 employed transfer learning with Inception-ResNet-V2 164 for thyroid cancer detection and introduced a deep 3D CNN for cancer detection in multispectral photoacoustic data sets. 165oustakidis et al. 166 developed SkinSeg for identifying skin layers in raster-scan optoacoustic mesoscopy (RSOM) images, evaluating decision trees, 167 SVM, 168 and DL algorithms.Nitkunanantharajah et al. 169 achieved good classification performance with ResNet18 164 on RSOM nail fold images.
PACT has demonstrated its great potential for human vascular imaging in several clinical studies. 15However, segmentation of the vascular structures, particularly the vascular lumen, is still accomplished through manual delineation by expert physicians, which is not only time-consuming but also subjective.To address this issue, Chlis et al. 154 proposed a sparse UNet (S-UNet) for automatic vascular segmentation on MSOT images.GT is obtained from binary images extracted from MSOT images, based on consensus between two clinical experts [Fig.8(b)].The MSOT raw data from six healthy humans' vasculature were acquired using a handheld MSOT system, and they were split into training, validation, and test sets.The S-Unet showed performance similar to other U-Net methods, but with smaller parameter sizes and the ability to select wavelengths, indicating its potential for clinical application. 170n addition, Lafci et al. 171 proposed a U-Net architecture to accurately segment animal boundaries in hybrid PA and US (PAUS) images.Boink et al. 172 proposed a learned primal-dual (L-PD) algorithm based on a CNN to solve the reconstruction and segmentation problem simultaneously.Ly et al. 57 introduced a modified U-Net DL model for automatic skin and vessel segmentation in in vivo PAI.The U-Net architecture showed the best performance.
All the research reviewed in this section is summarized in Table 8.

Overcoming Other Specified Issues
In addition to the general challenges to PAI mentioned above, researchers encounter several more specific problems with their imaging systems.Collectively, these specific issues constitute a seventh challenge, and among them we have identified six representative categories: motion artifacts, limited spatial resolution, electrical noise and interference, image misalignment, slow accelerating superresolution imaging, and achieving digital histologic staining.
Motion artifacts caused by breathing or heartbeats can significantly reduce image quality in PAM and PA endoscopy (PAE or intravascular PA, IVPA).To address this issue, researchers have presented various breathing artifact removal methods, 173,174 and DL methods have recently been proposed as a potential solution.Chen et al. 175 introduced a CNN approach with three convolutional layers to address motion artifacts and pixel dislocation in in vivo rat brain images.Zheng et al. 176 proposed MAC-Net, a network based on VGG16 GAN 134 and spatial transformer networks (STN), 177 to suppress motion artifacts in IVPA.Both methods demonstrated successful improvement in image quality.
OR-PAM can penetrate ∼1 mm deep in biological tissue, limited by light scattering.AR-PAM, which does not use focused light, can penetrate up to several centimeters, but it has a lower spatial resolution than OR-PAM.Researchers have applied DL to enhance the spatial resolution of AR-PAM to match that of OR-PAM.Cheng et al. 178 proposed a GAN-based framework called Wasserstein GAN 179 [Fig.9(a)].An integrated OR-and AR-PAM system was built for data acquisition and network training.The generator network takes an AR-PAM image as input and generates a high-resolution image, while the discriminator network evaluates the similarity between the generator's output and the GT image obtained from OR-PAM.Using in vivo mouse ear vascular images, the proposed method was first compared with the blind deconvolution method, and it improved the spatial resolution and produced superior microvasculature images.Furthermore, the proposed method was shown to be applicable to other types of tissues (e.g., brain vessels) and deep tissues (e.g., a chicken breast tissue slice of 1700 μm thickness) that are not easily accessible by OR-PAM.A similar study was implemented by Zhang et al., 181 who combined a physical model and a learning-based algorithm, termed MultiResU-Net.
DL methods have been applied to address noise and interference issues in PA imaging.Dehner et al. 182 developed a discriminative DNN using a U-Net architecture to separate electrical noise from PA signals, improving PA image contrast and spectral unmixing performance.He et al. 183 proposed an attentionenhanced GAN with a modified U-Net generator to remove noise from PAM images, prioritizing fine-feature restoration.Gulenko et al. 184 evaluated different CNN architectures and found that U-Net demonstrated higher efficiency and accuracy in removing electromagnetic interference noise from PAE systems.
To address image misalignment in PAM, Kim et al. 185 utilized a U-Net framework.Their method effectively addressed nonlinear mismatched cross-sectional B-scan PA images during bidirectional raster scanning, resulting in a significant improvement in imaging speed, doubling the speed compared to conventional approaches.
To improve the temporal resolution of superresolution localization imaging, 186,187 hundreds of thousands of overlapping images are traditionally required.However, this process can be time-consuming.To address this problem, Kim et al. 180 proposed a GAN with U-Net based on pix2pix 188 to reconstruct superresolution images from raw image frames [Fig.9(b)].The proposed network can be applied to both 3D label-free localization OR-PAM and 2D labeled localization PACT.The authors trained and validated the network with in vivo data from 3D OR-PAM and 2D PACT images.The proposed method reduced the required number of raw frames by 10-fold for OR-PAM and 12-fold for PACT, resulting in a significant improvement in temporal resolution.
Ultraviolet PAM (UV-PAM) takes advantage of the optical absorption contrast of UV light to highlight cell nuclei, generating PA contrast images similar to hematoxylin and eosin (H&E) labeling. 189DL techniques can be used to digitally generate histological stains using trained NNs based on UV-PAM images, providing label-free alternatives to standard chemical staining methods. 190oktor et al. 191 utilized a DL approach based on GANs to digitally stain total-absorption PA remote sensing (TA-PARS) images, achieving high agreement with the gold standard of histological staining.Cao et al. 192 employed a cycle-consistent adversarial network (CycleGAN) 193 model to virtually stain UV-PAM images, producing pseudo-color PAM images that matched the details of corresponding H&E histology images.In a recent study, Kang et al. 58 combined UV-PAM with DL to generate rapid and label-free histological images [Fig.9(c)].Their proposed method, termed deep-PAM, can generate virtually stained histological images for both thin sections and thick fresh tissue specimens.By utilizing an unpaired image-to-image translation network, a CycleGAN, they were able to process GM-UV-PAM images and instantly produce H&E-equivalent images of unprocessed tissues.This groundbreaking approach has significant implications for the field of histology and may offer an alternative to traditional staining methods.
All the research reviewed in this section is summarized in Table 9.

Discussion and Conclusion
PAI is a rapidly growing biomedical imaging modality that utilizes endogenous chromophores to noninvasively provide biometric information, such as vascular structure and sO 2 .However, as shown in Fig. 1, PAI still faces seven significant challenges: (1) overcoming limited detection capabilities, (2) compensating for low-dosage light delivery, (3) improving the accuracy of quantitative PA imaging, (4) optimizing or replacing conventional reconstruction methods, (5) addressing tissue heterogeneity, (6) improving the accuracy of image classification and segmentation, and (7) overcoming other specified issues.In this review paper, we have summarized DL studies  over the past five years that have addressed these general challenges in PAI.Further, we have discussed how DL can be used to solve several more specific problems in PAI.CNN, U-Net, and GAN have been the most representative networks used in PAI-related research.While some studies use basic architectures to achieve their goals, others modify or develop new architectures to solve particular problems in PAI.These networks can be used in various ways, such as postprocessing reconstructed images with different types of noise or directly reconstructing PA images from the time domain in the image domain.
Furthermore, recent research has aimed to extract more accurate quantitative information by using multiple networks, rather than solely focusing on enhancing image quality with one network.This approach can provide more comprehensive and detailed information, improving the overall performance of PAI.While SSIM is commonly used as the loss function, other metrics, such as PSNR and the Pearson correlation, may be added to improve information extraction and convergence speed.The continued exploration and refinement of these network architectures and loss functions will likely contribute to continued advancements in PAI.
Several obstacles remain.The success of DL approaches in PAI is highly dependent on the availability of high-quality data sets, and there is a scarcity of experimental training data.DL approaches in PAI also lack a standardized PA-image format and publicly available data that are accessible to all groups.Consequently, researchers rely on data generated from experiments or simulations, and even publishing PA data is difficult because there is no standard format.The k-Wave 79 toolbox is the most generally used to generate the PA initial pressure, along with other light transport simulators, such as mcxlab, 194 to generate the light distribution.However, creating reliable simulation data requires GT data from the real world.Commonly, x-ray CT or MRI images of blood vessels and organs are used for PA simulation.Fortunately, there are public data sets of x-ray CT and MRI images, and many groups have used these open data sets to generate PA GTs.However, the varying information obtained by different imaging modalities may not align with PAI.Recently, the International Photoacoustic Standardization Consortium (IPASC) 195 has been working to overcome this challenge by bringing together researchers, device developers, and government regulators to achieve standardization of PAI through community-led consensus building.With the efforts and involvement of IPASC, the generalization ability of DL, which is the fundamental problem in medical imaging field, will increase.
While DL has shown improved image qualities in PAI, there are still concerns regarding its applications in biomedical images.Therefore, the efforts of researchers who aim to advance PAI without applying DL are still valuable.For examples, new restoration algorithms 196 are being developed to enhance the image quality affected by limited-detection capabilities.The development of ultrawide detection bandwidth transducers 197 aims to mitigate the limited bandwidth of traditional US transducers, thereby improving the overall sensitivity and resolution of PAI.Furthermore, specially designed PACT systems with fast-sweep laser scanning techniques offer automatic fluence compensation and motion correction. 129Combining PACT with transmissionmode US-computed tomography enables the mapping of the distribution of SoS, further enhancing PACT image quality. 198oreover, a wide range of exogenous contrast agents has been developed to improve the SNR of PAI or to overcome the resolution limitations. 40][201][202][203][204][205][206][207] PAI's fundamental problems, caused by hardware limitations and the lack of tissue information, are ripe for solution by the information extraction, convergence, and high-speed processing enabled by DL.The result will be new opportunities for PAI to take off as a major imaging modality, opening an exciting era of DL-based PAI.

Fig. 2
Fig. 2 The concept of (a) a biological neural network and (b) an ANN derived from (a).(c) Schematics of a simple neural network and a DNN.

Fig. 4
Fig. 4 Representative studies using DL methods to overcome limited-detection capabilities.(a) A DNN with five fully connected layers enhances bandwidth.(b) LV-GAN for addressing the limited-view problem.(c) A Y-Net generates the PA images by optimizing both raw data and reconstructed images from the traditional method.(d) A 3D progressive U-Net (3D-pUnet) to diminish the effects of limited-view artifacts and sparsity arising from cluster view detection.The images are adapted with permission from Ref. 76, © 2017 SPIE; Ref. 77, © 2020 Wiley-VCH GmbH; Ref. 78, © 2020 Elsevier GmbH; and Ref. 52, © 2021 Wiley-VCH GmbH.BW, bandwidth; DNN, deep neural network; DAS, delay-and-sum; cluster, cluster view detection; full, full view detection.

Fig. 6
Fig. 6 Representative studies to improve the accuracy of quantitative PAI by DL.(a) Convolutional encoder-decoder type network with skip connections (EDS) to produce accurate estimates of sO 2 in a 3D data set.(b) Dual-path network based on U-Net (QPAT-Net) to reconstruct images of the absorption coefficient for deep tissues.(c) US-enhanced U-Net model (US-UNet) to reconstruct the optical absorption distribution.The images are adapted with permission from Ref. 124, © 2020 SPIE; Ref. 54, © 2022 Optica; and Ref. 125, © 2022 Elsevier GmbH.

Fig. 7
Fig. 7 Representative studies to optimize conventional reconstruction algorithms or replace them with DL.(a) Pixel-wise interpolation approach followed by an FD-UNet for limited-view and sparse PAT image reconstruction.(b) End-to-end U-Net with residual blocks to reconstruct PA images.(c) Two-step PA image reconstruction process with FPnet and U-Net.The images are adapted with permission from Ref. 137, © 2020 Nature Publishing Group; Ref. 138, © 2020 Optica; and Ref. 55, © 2020 Elsevier GmbH.

Fig. 8
Fig. 8 Representative DL studies to correct the SoS and improve the accuracy of image classification and segmentation.(a) Hybrid DNN model including U-Net and Segnet to mitigate SOS aberration in heterogeneous tissue.(b) Sparse-UNet (S-UNet) for automatic vascular segmentation in MSOT images.The images are adapted with permission from Ref. 153, CC-BY; Ref. 154, © Elsevier GmbH.

Fig. 9
Fig. 9 Representative studies using DL to solve specific issues.(a) GAN-based framework (Wasserstein GAN) to enhance the spatial resolution of AR-PAM.(b) GAN with U-Net to reconstruct superresolution images from raw image frames.(c) Deep-PAM generates virtually stained histological images for both thin sections and thick fresh tissue specimens.The images are adapted with permission from Ref. 178, © 2021 Elsevier GmbH; Ref. 180, © 2022 Springer Nature; and Ref. 58, © 2021 Elsevier GmbH.BF-H&E, brightfield hematoxylin and eosin staining; DNN, deep neural network.

Table 1
Summary of challenges facing PAI.
Inaccuracy and rough classification and segmentation of PA image3.7Overcomingother specified issues Motion artifacts, limited spatial resolution, electrical noise and interference, image misalignment, accelerating superresolution imaging, achieving digital histologic staining

Table 3
Summary of overcoming the limited detection capabilities with DL approaches.

Table 4
Summary of studies on compensating for low-dosage light delivery.

Table 5
Summary of studies to improve the accuracy of quantitative PAI.

Table 6
Summary of methods to optimize or replace conventional reconstruction algorithms.

Table 7
Summary of methods for addressing tissue heterogeneity.

Table 8
Summary of methods for improving the accuracy of image classification and segmentation.

Table 9
Summary of methods for addressing other specified issues.Yang et al.: Recent advances in deep-learning-enhanced photoacoustic imaging