2.1.

Schematic of the Annihilation-Gamma/Near-Infrared/Visible Multimodal System

Figure 1 shows a schematic diagram of an annihilation-gamma/NIR/visible multimodal imaging system. When hybrid tracers are accumulated in patients, they emit annihilation-gamma rays in the opposite direction and NIR after absorbing external excitation photons with a range of 760 to 800 nm. If the wavelength band of excitation light is too wide, reabsorption occurs in the higher wavelength band of excitation light.²⁷ To limit the wavelength band of excitation light to the range of 760 to 800 nm, a commercial ICG filter kit that includes excitation, emission, and dichroic imaging filters was used. The proposed laparoscopic module consisted of an internal detector for gamma ray detection and objective lens gathering optical and NIR photons. The external detector was placed outside patients for coincidence detection of annihilation photons with the internal detector. Scintillation light derived from the annihilation-gamma rays was directly detected by an array of multipixel photon counters (MPPCs). Visible light (500 to 700 nm) and emission NIR (800 to 850 nm) were transmitted through a gradient index (GRIN) relay lens and separated with a dichroic mirror module. The visible and NIR images from CCD cameras and the gamma images from MPPCs were merged and displayed on the monitor. The multimodal imaging system provides the simultaneous images of NIR and visible light in real time and the image of gamma rays within 1 s.

Fig. 1

Schematic representation of an annihilation-gamma/NIR/visible multimodal imaging system.

2.2.

Annihilation-Gamma Detection for 511-keV Gamma Ray

The annihilation-gamma detector consisted of two types of detectors. The internal (laparoscopic) detector was inserted into a surgical stainless pipe with an outer diameter of 15.8 mm and a thickness of 0.5 mm. A $6\times 6$ crystal array [Fig. 2(a)] of cerium-doped LYSO (EPIC Crystal, China) with the total dimension of $8.6\times 8.6\times 7.5{\mathrm{mm}}^3$ (crystal size of $1.2\times 1.2\times 7{\mathrm{mm}}^3$ and pitch of 1.4 mm) was directly coupled to a custom-assembled MPPC array [Fig. 2(b)]: One type of eight central MPPCs (S13360-2050VE, Hamamatsu, Japan) which has a photosensitive area of $2\times 2{\mathrm{mm}}^2$ and a microcell size of $50\mu \mathrm{m}$, and the other type of peripheral 12 MPPCs (S13190-1015HDR, Hamamatsu, Japan) that has a photosensitive area of $1\times 1{\mathrm{mm}}^2$ and a microcell size of 15 μm. Figure 2(b) shows the MPPC array attached to a custom-designed front-end board. The central $2.7\times 2.7{\mathrm{mm}}^2$ blank area of the internal detector was reserved for a GRIN lens for the optical path of the visible light and NIR, as shown in Figs. 2(a) and 2(b). The $6\times 6$ crystal array consisted of 32 LYSO crystals because of the central blank area.

Fig. 2

Parts of the annihilation-gamma detector: (a) LYSO:Ce crystal array (internal detector), (b) two types of MPPC array (internal detector), (c) LYSO:Ce crystal array (external detector), (d) $32\times 32$ array of MPPCs attached to four front-end boards of the external detector, (e) power supply board, (f) coincidence logic circuit, and (g) 32-channel waveform digitizer. The power supply board was used to supply the power for all the MPPCs in the system.

The external detector was made up of four blocks of $16\times 16$ LYSO (EPIC Crystal, China) crystal array (crystal size of $2.9\times 2.9\times 20{\mathrm{mm}}^3$ and pitch of $3.0\times 3.0\times 20{\mathrm{mm}}^3$) dimension of $49.0\times 49.0\times 20.2{\mathrm{mm}}^3$ [Fig. 2(c)]. All the four crystal blocks were coupled one to one with four $16\times 16$ arrays of MPPCs (S13361-3050-NE04, Hamamatsu, Japan). Figure 2(d) shows the $2\times 2$ front-end board containing the $32\times 32$ array of MPPCs. The surfaces between crystal array and MPPCs of both detectors were coupled with optical grease (BC-630, Saint-Gobain, France). All sides of the scintillation crystals in the internal detector, except for the surface facing the MPPCs, were wrapped with Teflon tape with a reflectivity of $>95\%$ at 550 nm.²⁸ The crystal surfaces in the external detector, except for the surface facing the MPPCs, were painted with barium sulfate (${\mathrm{BaSO}}_4$) mixed powder ($>99\%$ at 550 nm).²⁹ Each crystal surface of both detectors was chemically polished. To provide an external power source with each MPPC, a power supply board (BASP-10002, Brightonix, Korea), which has eight output channels, was used, as shown in Fig. 2(e).

2.3.

Signal Processing of the Annihilation-Gamma Detector

The analog signals from the internal detector were reduced to four position signals (A, B, C, and D) by a discretized positioning circuit (DPC). All the MPPCs in the internal detector were connected to a resistive chain, which generates the four position signals. The analog signals from the external detector were transformed into position signals (${\mathrm{X}}^{+}$, ${\mathrm{X}}^{-}$, ${\mathrm{Y}}^{+}$, and ${\mathrm{Y}}^{-}$) by a symmetric charge division (SCD) circuit. In general, the SCD circuit generates more position-sensitive signals than the DPC. However, the SCD circuit is not proper for the internal detector, which requires compact size since the SCD circuit requires more resistors than the DPC.

The horizontal ($X$) and vertical ($Y$) position signals were calculated by the equations, as shown in Fig. 3. The position signals from both detectors were discriminated by an on-board leading-edge discriminator with a 0.5-V threshold. The discriminated signals were fed into a coincidence logic unit (BASP-10003, Brightonix, Korea) which has 16 trigger input channels, 2 AND output channels, and 8 OR output channels [Fig. 2(f)]. The four position signals of both detectors were converted to analog-to-digital converter (ADC) values by a 32-channel waveform desktop digitizer (DT5740, CAEN, Italy) with a sampling frequency of 62.5 MHz [Fig. 2(g)]. The ADC values were used to calculate the $X$ and $Y$ positions of gamma interaction points within the LYSO crystal array and to obtain the energy information by summing the four position signals. Figure 3 shows the block diagram to process the detected signal from the annihilation-gamma detector.

Fig. 3

Block diagram of signal processing from the annihilation-gamma detector.

2.4.

Analysis of Digitized List-Mode Data

Crystal maps containing the gamma-struck positions were segmented using a Voronoi diagram algorithm implemented in MATLAB 2018b. The two-dimensional peak position of the crystal map was automatically picked and segmented into individual crystal addresses corresponding to the number of crystals. Each segmented area in the Voronoi diagram was assigned to an individual crystal, for which the energy spectrum was obtained. The energy spectrum in each crystal was normalized to compensate the different gain of the photodetector and light yield of each crystal using the following equation:

2.5.

Evaluation of the Annihilation-Gamma Detector

Because the gain of each MPPC channel varies with the bias voltage and temperature, it is necessary to measure the ADC values by changing these external conditions. To evaluate the intrinsic performance of the external detector, a reference detector was used. The reference detector consisted of a 16-channel MPPCs coupled with a gadolinium aluminum gallium garnet (C&A Corporation, Japan) crystal with a size of $12\times 12\times 9.5{\mathrm{mm}}^3$. A ${}^{22}\mathrm{Na}$ calibration disk source (Eckert & Ziegler, Germany) with an activity of 50 kBq was placed vertically between the reference detector and the external detector. The distances from the ${}^{22}\mathrm{Na}$ source to the reference detector and to external detector were 10 and 300 mm, respectively. All the detectors were placed inside a temperature-controlled chamber at 20°C. A total of 5.1 million prompt (coincidence) events were collected.

2.6.

Design of Optical Lens, Parts and a Dichroic Mirror Module

To aid the assembly of the optical components such as optical lens and a dichroic mirror, the distance between the optical parts was optimized by using optical simulation software (ZEMAX) [Fig. 4(a)]. The optical imaging system was made up of a GRIN objective lens (GRINTECH, Germany) with an numerical aperture (NA) of 0.5 and a GRIN relay lens (GRINTECH, Germany) with an NA of 0.1 to transmit the properly focused optical light, because the amount of escaped light was smaller than with a higher NA value. Between the GRIN relay lenses, achromatic doublet lenses (Edmund Optics), which has a diameter of 6.25 mm and effective focal length of 7.5 mm, were placed at the center of optical axis with magnification of 1. A dichroic mirror module consisted of an aspherized achromatic lens with a diameter of 12.5 mm (Edmund Optics), a dichroic mirror (Edmund Optics), and CCD cameras (Ace 800-510, Basler, Germany). The dichroic mirror with a cut-off wavelength of 801 nm ($>98\%$ reflection) was placed at the center of a beam splitter box [Fig. 4(b)]. The CCD cameras with the same pixel size of $4.8\times 4.8\mu {\mathrm{m}}^2$ were used for the acquisition of the reflected visible light and the passed NIR images. Figure 4(c) shows the transmission curves of the optical filters and the emission spectrum of the ICG fluorescence dye. The transmission curve of the dichroic mirror was in agreement with the emission spectrum of the ICG dye.

Fig. 4

Design of optical lens and other parts: (a) optical simulation, (b) design of a dichroic beam splitter module, and (c) transmission and emission spectrum of the optical filters and ICG fluorescence dye (Edmund Optics). In (c), the cut-off represents the 50% reflection wavelength of the dichroic mirror and the FWHM refers to the full width at half maximum.

2.7.

Integration of Annihilation-Gamma, Near-Infrared, and Visible Images

The ${}^{22}\mathrm{Na}$ calibration disk source with a 1.5-mm diameter and a 3D-printed plastic phantom containing an ICG dissolved in dimethyl sulfoxide (DMSO) were used to test the integration of sequentially acquired images. The hole at the center of the ICG plastic phantom has the diameter of 2 mm and height of 5 mm. Figure 5 shows the procedure of merging optical and gamma images. The NIR image rescaled to the gray level in Fig. 5(a) was changed to an RGB true color image with green-fluorescence pseudocolor for clear identification [Fig. 5(b)]. Because the CCD cameras have the same pixel size and the identification number, only off-axis calibration was carried out to merge the two images [Figs. 5(b) and 5(c)]. The reconstructed gamma ray image having a field of view (FOV) of $100\times 100{\mathrm{mm}}^2$ was cropped to a rectangle (red box) corresponding to the FOV of the respective CCD cameras and resized to $640\times 480\text{pixels}$ by using a two-dimensional cubic spline interpolation algorithm provided with MATALB software [Figs. 5(d) and 5(e)].

Fig. 5

Integration of acquired images: (a) the NIR image, (b) green-colored NIR image, (c) the visible image, (d) the gamma ray image, (e) resized and red-colored gamma ray image, and (f) the integrated gamma/NIR/visible image.

2.8.

Tissue-Like Phantom Test for Penetration Depth Measurement

As shown in Fig. 6(a), the integrated images of the ICG-contained plastic phantom and the ${}^{22}\mathrm{Na}$ disk source covered with optical scattering material were evaluated to measure the penetration depth of NIR optical photons and to validate the annihilation-gamma rays independent on the optical scattering and absorption. The phantom and the source were located on a support jack, which is 10 mm apart from the center of origin. The acquisition time of the NIR CCD and the digitizer were 120 ms and 30 s, respectively.

Fig. 6

Tissue-like phantom test: (a) experimental setup to acquire annihilation-gamma and NIR images, (b) a photo of the gelatin-based tissue-like phantoms with different thicknesses, and (c) ROIs of individual images.

A tissue-like phantom was fabricated with 89% of water, 10% of gelatin powder from porcine skin (Sigma Aldrich), and 1% by weight of intralipid 20% (JW pharmaceutical, Korea). The mixture was heated to 37°C and stirred for 30 min until completely dissolved. After pouring into a silicon mold, the mixture was cooled for 20 h at room temperature. Figure 6(b) shows the tissue-like phantoms with different thicknesses of 5, 10, 15, and 20 mm, respectively. Only an area of $10\times 10{\mathrm{mm}}^2$ of phantoms was used because the hole of the plastic phantom had a diameter of 2 mm. To measure the signal loss of NIR and the annihilation-gamma ray by the thickness of the tissue-like phantom, the total pixel value at region of interest (ROI) of the signal (red box) and the background (white box) in the obtained images was calculated [Fig. 6(c)].

2.9.

Phantom Image Test with ${}^{18}\mathrm{F}$ and ICG

To validate the feasibility of the proposed multimodal imaging system, an image of 3D-printed acrylonitrile butadiene styrene-like plastic phantom that contained ${}^{18}\mathrm{F}$ radionuclides and an ICG-DMSO mixture was obtained. Figure 7(a) shows the experimental setup of the phantom test. A tunable white light source (White 6500K, ScopeLED) and an excitation source (ML780L3, Thorlabs) with the peak wavelength of 780 nm were connected to a dichroic mirror module that transmits light from these light sources. The 0.25-mm-diameter excitation fibers (Eska-CK10, Mitsubishi, Japan) were assembled into four-layered ring shape (total 1-mm width) and were inserted at the edge of the laparoscopic (internal) detector. As shown in Fig. 7(b), the distance of the source to the internal detector and the external detector was 30 and 90 mm, respectively, to cover the full area of the plastic phantom. Half of the volumes of the holes with diameters of 2.6, 3.5, and 4.6 mm were filled with a mixture of ${}^{18}\mathrm{F}$ and ICG. The activity of each hole was 0.78 (4.6 mm hole), 0.45 (3.5 mm hole), and 0.25 MBq (2.6 mm hole), respectively, and each hole has the same activity per volume ($0.015\mathrm{MBq}/\mu \mathrm{L}$). Events were collected for 30 s to obtain the gamma image. Acquisition time and other parameters of the digitizer and respective CCD cameras were the same as in Secs. 2.4 and 2.6.

Fig. 7

Phantom test setup: (a) photograph, (b) geometrical setup of the annihilation-gamma detector, (c) a plastic phantom.

3.1.

Gain Normalization of the Annihilation-Gamma Detector

Figure 8 shows the segmented crystal map, the energy resolution map, and the entire energy spectra from the crystals without and with gain normalization (photopeak alignment). The normalized photopeak ADC values were processed by the algorithm described in Sec. 2.4. The individual crystal segmentation of the internal detector and bottom-left blocks of the external detector performed well, as shown in Figs. 8(a) and 8(e). The other three blocks of the external detector showed similar crystal maps. The events in the central $2\times 2$ crystals of the segmented map of the internal detector were excluded from the true coincidence events because these events are dominantly derived from intercrystal scattering [Fig. 8(b)]. At the bias voltage of 58 V, without gain normalization, the mean photopeak ADC value and the energy resolution of the internal detector were $823\pm 183$ ADC values and 52%, respectively [Fig. 8(c)]. After gain normalization, the energy resolution of the internal detector was 21%, as shown in Fig. 8(d). Similarly, the ADC values of the external detector were also normalized. Figure 8(f) represents the energy resolution maps of the external detector. The energy resolution of top-left and top-right crystal blocks were slightly worse than that of other crystal blocks. This degradation was mainly caused by light output variation between the crystals, and slightly nonuniform optical coupling condition between the crystal and the MPPCs, and the gain difference of the MPPC channels. For these reasons, the energy spectra of the four crystal blocks of the external detector were widely different from each other so that the mean energy resolution could not be calculated, as shown in Fig. 8(g). With gain normalization, the energy resolution of the external detector was 18% at the bias voltage of 58 V [Fig. 8(h)].

Fig. 8

Gain normalization for each crystal: (a) the segmented crystal map (internal), (b) the energy resolution map (internal), (c) the energy spectrum without gain normalization (internal), (d) the energy spectrum with gain normalization (internal), (e) the segmented crystal map of bottom-left detector (external), (f) the energy resolution map (external), (g) the energy spectrum without gain normalization (external), and (h) the energy spectrum with gain normalization (external).

3.2.

Performance of the Annihilation-Gamma Detector

Figure 9 shows the normalized mean photopeak ADC values and the energy resolution obtained from the annihilation-gamma detector depending on the bias voltage and temperature. The mean photopeak ADC values and the energy resolution of top-left (dotted-line) and top-right (one-dot chain line) crystal blocks were slightly worse than the other crystal blocks, as described in Sec. 3.1. Because the maximum overvoltage of the MPPCs is 61 V (kapd1053e.pdf, Spec sheet, Hamamatsu, Japan), the performance of all the MPPCs was evaluated from 56 to 61 V. The best (lowest) energy resolution of the internal detector was 16% at 58.5 V, and the normalized energy resolution of the four crystal blocks of the external detector was 19% at 59 V [Fig. 9(b)]. The bias of 72 V for the 12-edge MPPCs with a photosensitive area of $1\times 1{\mathrm{mm}}^2$ was applied. As shown in Figs. 9(c) and 9(d), the ADC values of the annihilation gamma detector were estimated depending on the external temperature. For simplicity, all crystal blocks in the external detector were applied at bias voltage of 59 V. The applied bias voltage to the internal detector was 58.5 V. The mean photopeak ADC value of both detectors shows the linear dependence on external temperature. Because the data acquired at real-time basis are insufficient to generate true coincidence event, all the mean photopeak ADC values and $N_{\text{factor}}$, as described in Sec. 2.4, were automatically saved as a calibration LUT for continuous data acquisition.

Fig. 9

Performance of the internal and external detectors: (a) mean photopeak ADC value with bias voltage, (b) energy resolution with bias voltage, (c) mean photopeak ADC value with temperature, and (d) energy resolution with temperature. Bottom-left, Bottom-right, Top-left, and Top-right represent the four detector modules of the external detector, respectively.

3.3.

Performance of Acquired Optical Images

Figures 10(a) and 10(b) show the USAF 1951 resolution target images and the smoothed line profiles of selected ROI (red box) obtained by the optical system inside the laparoscopic module. The bar phantom images acquired by two CCD cameras were clearly visible. Figure 10(c) shows the horizontal and vertical contrast transfer functions (CTFs) at the square ROI corresponding to each size of pattern. Even though the base noise signals were caused by the fiber-optic light sources, the differences in the peak and the valley of the images were clearly shown in all line profiles. There was some mismatched contrast region (vertical redline) because of unproperly calibrated flat-field mask images. The minimum distinguishable bar pattern of the USAF target corresponds to 4 line-pair per millimeter.

Fig. 10

Evaluation of the optical images: (a) USAF target image and line profiles (NIR), (b) USAF target image line profiles (visible), and (c) the CTF curve of CCD images.

3.4.

Penetration Depth Measurements of Near-Infrared and Annihilation-Gamma Rays

Figure 11 shows the integrated images covered with the tissue-like phantoms having different thicknesses ranging from 0 (left) to 15 mm (right). At the phantom thickness of 10 mm, the pseudo-fluorescence-green-colored NIR image was not clearly seen because of severe scattering by intralipid compounds [Fig. 11(a)]. However, the images of annihilation-gamma rays were not influenced by the thickness of the scattering material even at the thickness of 15 mm [Fig. 11(b)].

Fig. 11

NIR and gamma images of the phantom covered with the tissue-like phantom: (a) pseudo-green colored NIR images and (b) pseudo-hot-colored annihilation-gamma images. The white circular ring shape of acquired image was caused by reflection of excitation light.

Figure 12 shows the relative and total pixel intensity curves versus the thickness of the tissue-like phantom. To calculate the total pixel intensity, pixel values in the signal ROI was summed and subtracted by pixel values in the background ROI. The signal and background ROIs have the same size for NIR ($100\times 100$) and gamma rays ($20\times 15$). In case of gamma rays, there was no signal in the background ROI. The total pixel intensity of the NIR image significantly decreased by the thickness of the scattering material. Because the value at the thickness of 20 mm was almost the same as the thickness of 15 mm, the result at 20-mm thickness was excluded. The total pixel intensity of the image of annihilation-gamma rays was almost the same over the thickness and not affected by the thickness of the tissue-like phantom.

Fig. 12

Relative and total pixel intensity curves by the thickness of the tissue-like phantom: NIR, annihilation-gamma rays, and noise level of the NIR CCD (only absolute values). Error bars were obtained by estimating Poisson fluctuations of the signals and the error bars other than those of the relative pixel intensities of gamma rays are too small to be visible.

3.5.

Simultaneously Acquired 511-keV Gamma Ray, Near-Infrared, and Visible Phantom Image

Figure 13(a) shows the reconstructed 511-keV gamma images using simple backprojection algorithm. The image was clearly seen at the acquisition time of 5 s, and the true coincidence count rate at the distance of 30 mm from the internal detector was $0.31\mathrm{cps}/\mathrm{kBq}$. As shown in Fig. 13(e), the acquired gamma image was resized by a factor of 10 and the average pixel intensity values on the $x$ axis and the full width at half maximum (FWHM) of the rectangular ROI (white line) were calculated. Figure 13(c) shows the intensity profile of each hot phantom hole. After the quadratic subtraction of the size of phantom holes, the FWHM of the gamma image in the hole of 2.6, 3.5, and 4.6 mm was, respectively, 2.8, 3.0, and 3.8 mm at the acquisition time of 30 s.

Fig. 13

Raw gamma images of acquisition time of (a) 5 s, (b) 10 s, (c) 20 s, and (d) 30 s. (e) The ROI of the acquired image and (f) the intensity profile of the ROI (white line).

Figure 14 shows the integrated image of gamma ray, NIR, and visible light of the plastic phantom-contained ${}^{18}\mathrm{F}$ and the ICG-DMSO compound solution. The pixel size of the reconstructed image was 1.4 mm, corresponding to a real crystal pitch of the internal detector, and the reconstructed image was resized to the same size of the CCD cameras, as shown Fig. 14(c). After the measurement for 30 s, the detected true coincidence events were reconstructed and were updated to the image generated from the measurement for 1 s. As shown in Fig. 14(f), the image overlaid with the pseudocolored NIR and gamma image was well matched.

Fig. 14

Integrated images obtained simultaneously: (a) visible image, (b) NIR image, (c) annihilation-gamma image, (d) visible image overlaid with NIR image, (e) visible image overlaid with annihilation-gamma image, and (f) integrated image.