Characterization and validation of multimodal annihilation-gamma/near-infrared/visible laparoscopic system

Abstract. Minimally invasive robotic surgery using fluorescence-guided images with a video laparoscope has been widely used because of its advantages of small incision, fast recovery time, and efficiency. However, the penetration depth limitation of fluorescence is a disadvantage caused by the absorption and scattering in tissues and blood cells. If this limitation can be overcome by additional imaging modalities, the surgical procedure can be quite efficient and precise. High-energy annihilation-gamma photons have a stronger penetration capability than visible and fluorescence photons. To characterize and validate a multimodal annihilation-gamma/near-infrared (NIR)/visible laparoscopic imaging system, an internal detector composed of an annihilation-gamma detector and an optical system was assembled inside a surgical stainless pipe with an outer diameter of 15.8 mm and an external detector with a dimension of 100×100  mm2 placed at the opposite side of the internal detector. Integrated images of 511-keV gamma rays, NIR fluorescence, and visible light were obtained simultaneously. The 511-keV gamma image could be clearly seen with the acquisition of 5 s, while NIR and visible images could be presented in real time. This multimodal system has the potential for improving the surgery time and the quality of patient care.


Introduction
Compared to conventional open surgery, which utilizes images acquired before the surgery from computed tomography (CT), magnetic resonance imaging (MRI), positron emission tomography (PET), and other diagnostic modalities, minimally invasive robotic surgery (MIRS) has been widely used because of advantages such as less incision, fast recovery time, and the quality of patient care. [1][2][3] Recently, near-infrared (NIR) imageguided surgeries have been used for the detection of sentinel lymph nodes (SLNs) and micro tumors during surgery, because these fluorescence systems provide relatively high sensitivity and superior resolution of <1 mm. [4][5][6][7] They have also been used for colorectal, urological, and gynecological diseases to define the margin of the target to achieve minimal damage of the normal tissue and sensitive nerves. Among various fluorescence dyes for MIRS, a U.S. Food and Drug Administration-approved indocyanine green (ICG) dye with a peak emission wavelength of 820 nm has been used for SLN mapping and medical imaging for other lesions. [8][9][10][11][12] However, the fluorescence has limited penetration depth (below 10 mm) because of its absorption and scattering in tissue such as skin, fat, and blood. [13][14][15] For this reason, it is difficult to find SLNs or other target tissues covered with normal tissue. To overcome the depth limitation, multimodal imaging and hybrid methods have been used with radiation, ultrasound, and other compatible modalities. Highenergy photons, such as x-rays and gamma rays, have relatively longer penetration depths despite radiation exposure, relatively low sensitivity, and poor spatial resolution.
Recent works validated the feasibility of intraoperative fluorescence-guided surgery with an image of preoperative singlephoton emission computed tomography/CT, although they did not provide real-time integrated images showing the target margin clearly. [16][17][18] Another study used trimodal imaging probes of ultrasound, photoacoustic, and fluorescence to capture sequential SLN images with a preclinical animal model. 19 Kang et al. 20 studied an endoscopic NIR/gamma/visible imaging system that integrated the images of an SLN in a mouse model using a 99m Tc-nanocolloid and an ICG fluorescence dye with a dichroic beam splitter module and respective charge-coupled diode (CCD) cameras. Recently, the feasibility of PET/NIR hybrid tracers that emit annihilation-gamma rays with higher sensitivity and spatial resolution than single gamma rays has been reported. [21][22][23][24][25][26] With these dual-labeled hybrid tracers, fluorescence-guided surgery can be more convenient and expedited without unexpected excessive bleeding and damage of the normal tissue.
To our knowledge, a multimodal laparoscopic system that detects the annihilation-gamma rays has not been developed for fluorescence-guided surgery. If a multimodal system can display the simultaneous images of gamma rays and optical fluorescence for intraoperative surgery, the surgical procedure will be *Address all correspondence to Seong J. Hong, E-mail: hongseongj@eulji.ac.kr faster and more precise. In this study, we present a multimodal imaging system that shows real-time images of positron annihilation-gamma rays, NIR, and visible light with higher spatial resolution and sensitivity than conventional intraoperative gamma probes.
2 Methods and Materials 2.1 Schematic of the Annihilation-Gamma/ Near-Infrared/Visible Multimodal System Figure 1 shows a schematic diagram of an annihilation-gamma/ NIR/visible multimodal imaging system. When hybrid tracers are accumulated in patients, they emit annihilation-gamma rays in the opposite direction and NIR after absorbing external excitation photons with a range of 760 to 800 nm. If the wavelength band of excitation light is too wide, reabsorption occurs in the higher wavelength band of excitation light. 27 To limit the wavelength band of excitation light to the range of 760 to 800 nm, a commercial ICG filter kit that includes excitation, emission, and dichroic imaging filters was used. The proposed laparoscopic module consisted of an internal detector for gamma ray detection and objective lens gathering optical and NIR photons. The external detector was placed outside patients for coincidence detection of annihilation photons with the internal detector. Scintillation light derived from the annihilation-gamma rays was directly detected by an array of multipixel photon counters (MPPCs). Visible light (500 to 700 nm) and emission NIR (800 to 850 nm) were transmitted through a gradient index (GRIN) relay lens and separated with a dichroic mirror module. The visible and NIR images from CCD cameras and the gamma images from MPPCs were merged and displayed on the monitor. The multimodal imaging system provides the simultaneous images of NIR and visible light in real time and the image of gamma rays within 1 s.

Annihilation-Gamma Detection for 511-keV Gamma Ray
The  Fig. 2(b)]: One type of eight central MPPCs (S13360-2050VE, Hamamatsu, Japan) which has a photosensitive area of 2 × 2 mm 2 and a microcell size of 50 μm, and the other type of peripheral 12 MPPCs (S13190-1015HDR, Hamamatsu, Japan) that has a photosensitive area of 1 × 1 mm 2 and a microcell size of 15 μm. Figure 2(b) shows the MPPC array attached to a custom-designed front-end board. The central 2.7 × 2.7 mm 2 blank area of the internal detector was reserved for a GRIN lens for the optical path of the visible light and NIR,   Fig. 2(c)]. All the four crystal blocks were coupled one to one with four 16 × 16 arrays of MPPCs (S13361-3050-NE04, Hamamatsu, Japan). Figure 2(d) shows the 2 × 2 front-end board containing the 32 × 32 array of MPPCs. The surfaces between crystal array and MPPCs of both detectors were coupled with optical grease (BC-630, Saint-Gobain, France). All sides of the scintillation crystals in the internal detector, except for the surface facing the MPPCs, were wrapped with Teflon tape with a reflectivity of >95% at 550 nm. 28 The crystal surfaces in the external detector, except for the surface facing the MPPCs, were painted with barium sulfate (BaSO 4 ) mixed powder (>99% at 550 nm). 29 Each crystal surface of both detectors was chemically polished. To provide an external power source with each MPPC, a power supply board (BASP-10002, Brightonix, Korea), which has eight output channels, was used, as shown in Fig. 2(e).

Signal Processing of the Annihilation-Gamma Detector
The analog signals from the internal detector were reduced to four position signals (A, B, C, and D) by a discretized positioning circuit (DPC). All the MPPCs in the internal detector were connected to a resistive chain, which generates the four position signals. The analog signals from the external detector were transformed into position signals (X þ , X − , Y þ , and Y − ) by a symmetric charge division (SCD) circuit. In general, the SCD circuit generates more position-sensitive signals than the DPC. However, the SCD circuit is not proper for the internal detector, which requires compact size since the SCD circuit requires more resistors than the DPC. The horizontal (X) and vertical (Y) position signals were calculated by the equations, as shown in Fig. 3. The position signals from both detectors were discriminated by an on-board leading-edge discriminator with a 0.5-V threshold. The discriminated signals were fed into a coincidence logic unit (BASP-10003, Brightonix, Korea) which has 16 trigger input channels, 2 AND output channels, and 8 OR output channels The ADC values were used to calculate the X and Y positions of gamma interaction points within the LYSO crystal array and to obtain the energy information by summing the four position signals. Figure 3 shows the block diagram to process the detected signal from the annihilation-gamma detector.

Analysis of Digitized List-Mode Data
Crystal maps containing the gamma-struck positions were segmented using a Voronoi diagram algorithm implemented in MATLAB 2018b. The two-dimensional peak position of the crystal map was automatically picked and segmented into individual crystal addresses corresponding to the number of crystals. Each segmented area in the Voronoi diagram was assigned to an individual crystal, for which the energy spectrum was obtained. The energy spectrum in each crystal was normalized to compensate the different gain of the photodetector and light yield of each crystal using the following equation: E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 0 1 ; 3 2 6 ; 5 3 5 where N factor is the normalizing factor defined by mean ADC value of the spectrum with the highest energy resolution, T factor is the weight factor of temperature change, and N spect is the normalized energy spectrum from the individual energy spectrum (E crystalID ). After obtaining a lookup table (LUT), which contained the mean photopeak ADC value versus the temperature, T factor at the respective temperature was calculated by dividing the mean photopeak ADC at 20°C.

Evaluation of the Annihilation-Gamma Detector
Because the gain of each MPPC channel varies with the bias voltage and temperature, it is necessary to measure the ADC values by changing these external conditions. To evaluate the intrinsic performance of the external detector, a reference detector was used. The reference detector consisted of a 16-channel MPPCs coupled with a gadolinium aluminum gallium garnet (C&A Corporation, Japan) crystal with a size of 12 × 12 × 9.5 mm 3 . A 22 Na calibration disk source (Eckert & Ziegler, Germany) with an activity of 50 kBq was placed vertically between the reference detector and the external detector. Journal of Biomedical Optics 096008-3 September 2019 • Vol. 24 (9) The distances from the 22 Na source to the reference detector and to external detector were 10 and 300 mm, respectively. All the detectors were placed inside a temperature-controlled chamber at 20°C. A total of 5.1 million prompt (coincidence) events were collected.

Design of Optical Lens, Parts and a Dichroic Mirror Module
To aid the assembly of the optical components such as optical lens and a dichroic mirror, the distance between the optical parts was optimized by using optical simulation software (ZEMAX) [ Fig. 4(a)]. The optical imaging system was made up of a GRIN objective lens (GRINTECH, Germany) with an numerical aperture (NA) of 0.5 and a GRIN relay lens (GRINTECH, Germany) with an NA of 0.1 to transmit the properly focused optical light, because the amount of escaped light was smaller than with a higher NA value. Between the GRIN relay lenses, achromatic doublet lenses (Edmund Optics), which has a diameter of 6.25 mm and effective focal length of 7.5 mm, were placed at the center of optical axis with magnification of 1. A dichroic mirror module consisted of an aspherized achromatic lens with a diameter of 12.5 mm (Edmund Optics), a dichroic mirror (Edmund Optics), and CCD cameras (Ace 800-510, Basler, Germany). The dichroic mirror with a cut-off wavelength of 801 nm (>98% reflection) was placed at the center of a beam splitter box [ Fig. 4(b)]. The CCD cameras with the same pixel size of 4.8 × 4.8 μm 2 were used for the acquisition of the reflected visible light and the passed NIR images. Figure 4(c) shows the transmission curves of the optical filters and the emission spectrum of the ICG fluorescence dye. The transmission curve of the dichroic mirror was in agreement with the emission spectrum of the ICG dye.

2.7
Integration of Annihilation-Gamma, Near-Infrared, and Visible Images The 22 Na calibration disk source with a 1.5-mm diameter and a 3D-printed plastic phantom containing an ICG dissolved in dimethyl sulfoxide (DMSO) were used to test the integration of sequentially acquired images. The hole at the center of the ICG plastic phantom has the diameter of 2 mm and height of 5 mm. Figure 5 shows the procedure of merging optical and gamma images. The NIR image rescaled to the gray level in

Tissue-Like Phantom Test for Penetration Depth Measurement
As shown in Fig. 6(a), the integrated images of the ICGcontained plastic phantom and the 22 Na disk source covered  Journal of Biomedical Optics 096008-4 September 2019 • Vol. 24 (9) with optical scattering material were evaluated to measure the penetration depth of NIR optical photons and to validate the annihilation-gamma rays independent on the optical scattering and absorption. The phantom and the source were located on a support jack, which is 10 mm apart from the center of origin. The acquisition time of the NIR CCD and the digitizer were 120 ms and 30 s, respectively. A tissue-like phantom was fabricated with 89% of water, 10% of gelatin powder from porcine skin (Sigma Aldrich), and 1% by weight of intralipid 20% (JW pharmaceutical, Korea). The mixture was heated to 37°C and stirred for 30 min until completely dissolved. After pouring into a silicon mold, the mixture was cooled for 20 h at room temperature. Figure 6(b) shows the tissue-like phantoms with different thicknesses of 5, 10, 15, and 20 mm, respectively. Only an area of 10 × 10 mm 2 of phantoms was used because the hole of the plastic phantom had a diameter of 2 mm. To measure the signal loss of NIR and the annihilation-gamma ray by the thickness of the tissue-like phantom, the total pixel value at region of interest (ROI) of the signal (red box) and the background (white box) in the obtained images was calculated [ Fig. 6(c)].

Phantom Image Test with 18 F and ICG
To validate the feasibility of the proposed multimodal imaging system, an image of 3D-printed acrylonitrile butadiene styrenelike plastic phantom that contained 18 F radionuclides and an ICG-DMSO mixture was obtained. Figure 7(a) shows the experimental setup of the phantom test. A tunable white light source (White 6500K, ScopeLED) and an excitation source (ML780L3, Thorlabs) with the peak wavelength of 780 nm were connected to a dichroic mirror module that transmits light from these light sources. The 0.25-mm-diameter excitation fibers (Eska-CK10, Mitsubishi, Japan) were assembled into four-layered ring shape (total 1-mm width) and were inserted at the edge of the laparoscopic (internal) detector. As shown in Fig. 7(b), the distance of the source to the internal detector and the external detector was 30 and 90 mm, respectively, to cover the full area of the plastic phantom. Half of the volumes of the holes with diameters of 2.6, 3.5, and 4.6 mm were filled with a mixture of 18 F and ICG. The activity of each hole was 0.78 (4.6 mm hole), 0.45 (3.5 mm hole), and 0.25 MBq (2.6 mm hole), respectively, and each hole has the same activity per volume (0.015 MBq∕μL). Events were collected for 30 s to obtain the gamma image. Acquisition time and other parameters of the digitizer and respective CCD cameras were the same as in Secs. 2.4 and 2.6.   Journal of Biomedical Optics 096008-5 September 2019 • Vol. 24 (9) events because these events are dominantly derived from intercrystal scattering [ Fig. 8(b)]. At the bias voltage of 58 V, without gain normalization, the mean photopeak ADC value and the energy resolution of the internal detector were 823 AE 183 ADC values and 52%, respectively [ Fig. 8(c)]. After gain normalization, the energy resolution of the internal detector was 21%, as shown in Fig. 8(d). Similarly, the ADC values of the external detector were also normalized. Figure 8(f) represents the energy resolution maps of the external detector. The energy resolution of top-left and top-right crystal blocks were slightly worse than that of other crystal blocks. This degradation was mainly caused by light output variation between the crystals, and slightly nonuniform optical coupling condition between the crystal and the MPPCs, and the gain difference of the MPPC channels. For these reasons, the energy spectra of the four crystal blocks of the external detector were widely different from each other so that the mean energy resolution could not be calculated, as shown in Fig. 8(g). With gain normalization, the energy resolution of the external detector was 18% at the bias voltage of 58 V [ Fig. 8(h)]. Figure 9 shows the normalized mean photopeak ADC values and the energy resolution obtained from the annihilation-gamma   Journal of Biomedical Optics 096008-7 September 2019 • Vol. 24 (9) obtained by the optical system inside the laparoscopic module. The bar phantom images acquired by two CCD cameras were clearly visible. Figure 10(c) shows the horizontal and vertical contrast transfer functions (CTFs) at the square ROI corresponding to each size of pattern. Even though the base noise signals were caused by the fiber-optic light sources, the differences in the peak and the valley of the images were clearly shown in all line profiles. There was some mismatched contrast region (vertical redline) because of unproperly calibrated flat-field mask images. The minimum distinguishable bar pattern of the USAF target corresponds to 4 line-pair per millimeter.

Penetration Depth Measurements of
Near-Infrared and Annihilation-Gamma Rays Figure 11 shows the integrated images covered with the tissuelike phantoms having different thicknesses ranging from 0 (left) to 15 mm (right). At the phantom thickness of 10 mm, the pseudo-fluorescence-green-colored NIR image was not clearly seen because of severe scattering by intralipid compounds [ Fig. 11(a)]. However, the images of annihilation-gamma rays were not influenced by the thickness of the scattering material even at the thickness of 15 mm [ Fig. 11(b)]. Figure 12 shows the relative and total pixel intensity curves versus the thickness of the tissue-like phantom. To calculate the total pixel intensity, pixel values in the signal ROI was summed and subtracted by pixel values in the background ROI. The signal and background ROIs have the same size for NIR (100 × 100) and gamma rays (20 × 15). In case of gamma rays, there was no signal in the background ROI. The total pixel intensity of the NIR image significantly decreased by the thickness of the scattering material. Because the value at the thickness of 20 mm was almost the same as the thickness of 15 mm, the result at 20-mm thickness was excluded. The total pixel intensity of the image of annihilation-gamma rays was almost the same over the thickness and not affected by the thickness of the tissue-like phantom.

Simultaneously Acquired 511-keV Gamma Ray,
Near-Infrared, and Visible Phantom Image Figure 13(a) shows the reconstructed 511-keV gamma images using simple backprojection algorithm. The image was clearly seen at the acquisition time of 5 s, and the true coincidence count rate at the distance of 30 mm from the internal detector was 0.31 cps∕kBq. As shown in Fig. 13(e), the acquired gamma image was resized by a factor of 10 and the average pixel   Journal of Biomedical Optics 096008-8 September 2019 • Vol. 24 (9) intensity values on the x axis and the full width at half maximum (FWHM) of the rectangular ROI (white line) were calculated. Figure 13(c) shows the intensity profile of each hot phantom hole. After the quadratic subtraction of the size of phantom holes, the FWHM of the gamma image in the hole of 2.6, 3.5, and 4.6 mm was, respectively, 2.8, 3.0, and 3.8 mm at the acquisition time of 30 s. Figure 14 shows the integrated image of gamma ray, NIR, and visible light of the plastic phantom-contained 18 F and the ICG-DMSO compound solution. The pixel size of the reconstructed image was 1.4 mm, corresponding to a real crystal pitch of the internal detector, and the reconstructed image was resized to the same size of the CCD cameras, as shown Fig. 14(c). After the measurement for 30 s, the detected true coincidence events were reconstructed and were updated to the image generated from the measurement for 1 s. As shown in Fig. 14(f), the image overlaid with the pseudocolored NIR and gamma image was well matched.

Discussion
Recently, optical imaging with fluorescence has become a new paradigm for intraoperative surgery because of its superior sensitivity and spatial resolution. Particularly, NIR has a relatively long penetration depth (>5 mm), compared to other fluorescence types. Because NIR optical images provide information on the clear margins of suspicious lesions, they have been widely used for the detection of various diseases. However, fluorescence has limited penetration depth because of the absorption and scattering by tissues such as lipid, blood, and skin. The multimodal system with radiation would be effective to overcome the limited depth problem of fluorescence, because it has higher penetration depth and the additional possibility to display a molecular image. Latest research shows that the spatial resolutions of single gamma ray imaging systems employing multipinholes exceed those of positron annihilation-gamma ray images. 30 On the other hand, it is challenging to incorporate the multipinhole single gamma imaging system into the imageguided laparoscopy because of spatial constraints. In general, positron annihilation-gamma ray images have an advantage of better image quality than other single gamma ray imagers, except a multipinhole imager, because of their higher spatial resolutions. The most widely used positron-emitting nuclides have relatively short half-lives of 110 min ( 18 F) and 68 min ( 68 Ga). For this reason, molecular-guided surgery and biopsy using PET radiotracers have recently been used for various diseases, such as brain, lymph node, and prostate (prostate-specific antigen negative) cancers, in spite of radiation exposure during radioguided surgery using high-energy gamma rays. 31 Furthermore, according to the research by Povoski et al., 32 occupational radiation exposure is relatively small during intraoperative or perioperative surgical procedures because of the short half-lives of radionuclides. With the mean 18 F-FDG activity of 699 AE 181 MBq injected to the patients during radio-guided surgeries, the mean deep-dose equivalents per hour of exposure were 83 AE 134 and 61 AE 57 μSv∕h for preoperative nurse and surgeon, respectively. The annual occupational exposure limit for adults defined by the International Commission on Radiological Protection (ICRP) is 50; 000 μSv of total effective dose equivalent. 33 According to another report by Heckathorne et al., 34 all the surgical staff including surgeons and scrub nurses received <60 μSv per procedure.
Recently, laparoscopes and endoscopes assembled to a fiberoptic image guide bundle have been developed because of their convenience to be coupled with other optical parts, including optical lenses or laser sources. In our unpublished study, integrated phantom images were obtained by using the first prototype multimodal system based on the glass optical fiber bundle. However, there was significant loss of the scintillation lights of <500 nm (peak emission wavelength of scintillation light was 420 nm for LYSO crystals) through the bundle, which was derived for light absorption by glass materials and the reflection of cladding materials. To overcome the loss of transmitted scintillation lights, the internal detector (MPPCs) was directly coupled with scintillation crystals without the optical fiber bundle. The energy resolution of the internal detector was 15% better than the previous optical fiber-based internal detector. Because the performance of the coincidence detectors was dominantly affected by light collection efficiency, the internal detector in this study had a better true coincidence count rate.
The FWHM of the gamma ray image was found to be 3.8 mm at 4.6-mm-diameter hole using the proposed annihilationgamma/NIR/visible imaging system, and the spatial resolution of the gamma ray images of 5-mm-diameter-cylinder sources acquired by previous endoscopic gamma/NIR/visible module was about 10 mm with the pinhole diameter of 1.5 mm. 35 Journal of Biomedical Optics 096008-9 September 2019 • Vol. 24 (9) Moreover, the minimum acquisition time to acquire the image of radioactive source was six times faster than that of previously developed endoscopic module. Because of high energy (511 keV) and coincidence detection, the proposed annihilation-gamma-ray/NIR/visible multimodal system has the higher sensitivity and the spatial resolution than the single gamma multimodal system not employing the multipinholes. However, incorporating the multipinhole single gamma imaging system into the image-guided laparoscopy seems to be difficult because of spatial constraints.
In this study, we acquired the transaxial image of the reconstructed image matrix that has a slice thickness of 10 mm at the depth of an optical image, because we already knew the position of the phantom. However, it was necessary to obtain the relative position information between the laparoscopic module and the tracers, because the position of the hybrid tracers will be unknown during the surgical procedure. The four-dimensional optical tracker will be adopted for future work, because the position of the laparoscope can be changed during intraoperative surgery. The optical tracker provides the position, distance, and the angular information in near real time, which is suitable for laparoscopic surgery. It is also important to provide a continuous gamma ray image without delay, as well as the sound information that more clearly provides the distribution of radio pharmaceuticals.
For three-dimensional image reconstruction, the lack of angular samples compared to the conventional ring PET scanner can cause distortion in the reconstructed image matrix. 36 It is important to know the actual geometric position reflected by the movement and the rotation of the coincidence detectors. Therefore, if there are limited angular samples, additional methods to compensate for these missing angular samples should be implemented to display the real-time images.

Conclusion
By using the multimodal imaging system, simultaneously integrated annihilation-gamma rays, NIR, and visible images can be obtained. In this study, we proved the feasibility of a multimodal annihilation-gamma/NIR/visible imaging system for intraoperative laparoscopic surgery. In future work, an additional backup study and preclinical test with mouse will be conducted with a completed real-time multimodal imaging system.

Disclosures
The authors declare that they have no competing interests.