In vivo near-infrared fluorescence three-dimensional positioning system with binocular stereovision

Abstract. Fluorescence is a powerful tool for in-vivo imaging in living animals. The traditional in-vivo fluorescence imaging equipment is based on single-view two-dimensional imaging systems. However, they cannot meet the needs for accurate positioning during modern scientific research. A near-infrared in-vivo fluorescence imaging system is demonstrated, which has the capability of deep source signal detecting and three-dimensional positioning. A three-dimensional coordinates computing (TDCP) method including a preprocess algorithm is presented based on binocular stereo vision theory, to figure out the solution for diffusive nature of light in tissue and the emission spectra overlap of fluorescent labels. This algorithm is validated to be efficient to extract targets from multispectral images and determine the spot center of biological interests. Further data analysis indicates that this TDCP method could be used in three-dimensional positioning of the fluorescent target in small animals. The study also suggests that the combination of a large power laser and deep cooling charge-coupled device will provide an attractive approach for fluorescent detection from deep sources. This work demonstrates the potential of binocular stereo vision theory for three-dimensional positioning for living animal in-vivo imaging.


Introduction
In-vivo optical sensing technique has been proved to be an attractive technique. [1][2][3] It has the capability of performing quantitative and qualitative studies on a cellular or molecular level. [4][5][6] Fluorescence is a versatile and useful tool for living animal in-vivo imaging. 7 Fluorescence is the emission of light from an excitation source. Fluorescence occurs when an orbital electron of a molecule relaxes to its ground state by emitting a photon of light after being excited to a higher quantum state by some type of energy. 8 The advantage of in-vivo optical imaging is cost-effective and easy to operate. 9 The photon absorption rate of most biological tissues is comparatively low in the near infrared (NI) spectral range (650-900 nm), and photons can be detected through living organs. 10 So far, many near-infrared range (NIR) fluorescent probes have been developed (such as quantum dots) for in-vivo imaging studies. [11][12][13][14][15] A single-view two-dimensional (2-D) imaging system is the mainstream of in-vivo fluorescence imaging equipment. [16][17][18][19] However, they cannot meet the needs for accurate positioning during the studies. A three-dimensional (3-D) positioning can provide more accurate target location information, 20 which shows a great significance for biological and medical applications. Researchers can observe the target spatial position, analyze the target metastasis, and determine the relative positional relationship between the target and the organs. In this paper, a NI in-vivo fluorescence imaging system is demonstrated, and a method for computing the 3-D coordinates of targets based on binocular stereovision theory is presented.
Generally, binocular stereo techniques can be divided into the following parts: image acquisition, camera calibration, feature extraction, image matching and 3-D positioning. 21,22 It has been applied into the fields of computer vision, 23 photogrammetry, 24,25 and experimental mechanic. 26,27 In fluorescence imaging in-vivo, the diffusive nature of light in tissue 28,29 and the emission spectra overlap of fluorescent labels pose great challenges for 3-D positioning using binocular stereovision theory. Thus, an algorithm is necessary to extract targets from the multispectral image (MSI) and determine the spot center of biological interest.
Hu et al. 30 proposed a novel technique that combined with binocular stereovision and fluorescent imaging for 3-D surface profilometry and deformation measurement, which enable the noncontact, full-field measurements in biotissue and biomaterial at the microscale. In this paper, the similar technique is applied, and furthermore, a novel system is developed for in-vivo 3-D positioning. A rotary platform is used instead of two cameras to reduce the dimension and cost of the system. The novel system can detect fluorescent signals from deep sources instead of surface information. In addition, the novel system presented is powerful that can perform in-vivo imaging for NI applications. The fluorescence sensing system includes a high-intensity, narrow-bandwidth excitation source, and high-sensitive photon detection. The relatively low quantum yield is the key limitation for penetration and use of the in-vivo optical sensing technique. In this paper, a large power diode pump solid state laser (671 nm, 2 W) with short pulse duration (≤ 10 nm) is used as the excitation source. A deep cooling charge-coupled device (CCD) with large pixel size (minimum −100°C, pixel size 13 μm) is used to detect the weak fluorescent signals at NI spectral range (quantum efficiency (QE) is >90% around NIR). This system is elaborately optimized and is capable of detecting weak fluorescent signals of deep targets with high sensitivity.
In this paper, a NI in-vivo fluorescence imaging system and 3-D positioning algorithm based on binocular stereovision theory are presented. Experiments are designed to demonstrate the validity of the algorithm. The results and further data analysis prove that this method could be used in 3-D positioning of the target in the small animal. This system also shows the ability to detect fluorescent signals from deep sources.

Experimental Setup
The in-vivo fluorescence sensing system consists of a darkroom, a high-energy diode pump solid-state laser, a liquid-core fiber that can divide one beam of light to four, emission filter wheel, rotary platform module, a deep cooling high-sensitive CCD, and a computer in which 3-D positioning software runs on (see Fig. 1). Figure 1 illustrates the schematics and key components of the in-vivo fluorescence sensing system. The darkroom provides light-confined environment, and external light is shielded. High-energy laser works as an excitation source to illuminate the small animal uniformly through liquid-core fiber. A series of LEDs are utilized for bright field illumination. Deep cooling CCD, focusing lens, and emission filter wheel are used to acquire high-sensitive multispectral image data at NIR.
The block diagram of the in-vivo fluorescence sensing system is shown in Fig. 2. Liquid-core fiber is coupled with laser output. Four outputs of the liquid-core fiber are located above the small animal to illuminate it uniformly from four directions. The axis of the optical imaging path intersects the center of the rotating platform with a 45-deg angle. With the assistance of anesthesia apparatus, small animal is very stable while rotating on the accurate rotary platform. High-sensitive deep cooling CCD can reduce the dark current noise greatly, with large pixel size and high QE around 650-900 nm, and it is amenable to acquire very weak fluorescent signal at NIR.
The excitation laser source is a large power diode pump solid-state laser (MRL-N, 2W, New Industries Optoelectronics Technology Co., Ltd. Changchun, China) emitting at 671 nm, with short pulse duration (≤ 10 nm), instead of an xenon lamp, for the xenon lamp will require long warm-up time and a 10%-50% changes of light intensity in working conditions. 31 In the designed system, a uniform excitation light for small animals is necessary, and a solid state laser source only has 0.1% light intensity changes. 32 The laser beam is transmitted via a liquid-core fiber (numerical aperture: 0.52). Four outputs of the liquid-core fiber are located above the small animal to illuminate the small animal uniformly from four directions. Liquid-core optical fiber has high numerical aperture, 33 which is an important parameter for optical fiber for large numerical aperture that represents high-coupling efficiency. 34,35 Liquidcore optical fiber also has good flexibility and reliability. 36 The optical detection unit consists of a deep cooling CCD, lens, and filter wheel. The fluorescence light transmits through lens, filters sequentially, and is detected by CCD. The central element of the detection unit is a 1024 × 1024 deep cooling 16-bit CCD camera (DU-934NBRD, Andor Technology, Belfast, Ireland). The camera is designed for NI applications. 37 It has very high QE at NIR (>90%). QE is a measurement of a device's electrical sensitivity to light at each photon energy level. 38 High QE means more incident photons to converted electrons. 39 This camera can detect extremely weak NI fluorescence signals. With deep cooling features, the read out noise is extremely low [2.5e (50 kHz)]. Dark current is one of the main sources for noise in image sensors such as CCDs; these can be greatly reduced by deep cooling [0.008e∕pixel∕s (−100°C)]. 40 In the system, the filter is placed between the lens and CCD. A customized focusing lens is fabricated to meet the demands, and the focal length and field of view can be adjusted. The filter wheel is fully enclosed, light-tight. A total of 12 filter holes, 11 band-pass filters, and an all-pass piece (K9 glass) are installed. The K9 glass is used in the bright field illumination. The center wavelengths of the filters are distributed evenly in the range of "NI windows." The filters were purchased from China-Quantum Co., Ltd., Changchun, China. The full-width half maximum of the filters is around 10 nm (9.2 nm minimum to 11.6 maximum). Filters currently in use are 681, 702, 711, 759, 780, 794, 808, 850, 880, 903, and 938 nm. The CCD is connected to a common personal computer via USB 2.0, and image acquisition was done with the software Andor SOLIS 4.7.3 provided by the manufacturer of the CCD.
The darkroom provides a light-tight experiment environment and fixes components of the imaging system. The inner surface is coated with black extinction paint. The reflectivity is <0.5%. The possibility of light leakages from the outside such as fiber, anesthetic gas transmission pipes, wires, etc., is greatly reduced by double sealed design. When capturing images using this system, the first step is to open the LEDs, adjust the field of view, and calibrate the CCD camera. The fluorescent-labeled small animal is placed on the rotary platform. Under the LED bright field illumination, the all-pass filter will capture an image of the Fig. 1 A schematic of the in vivo near-infrared fluorescence three-dimensional positioning system. small animal. After that, the LEDs are closed, and the laser is opened; the filters and rotary platform are changed in order to obtain multispectral fluorescence images from different perspectives. Images are subsequently processed and analyzed with customized software running on the computer.

Target Extraction Based on Multispectral Imaging
In NI fluorescence in-vivo imaging, the target is captured as a large light spot. Therefore, the center of the target spot needs to be determined for 3-D positioning.

Multispectral imaging
By switching different filters, the spectroscopic information is obtained in image form. 41 With different wavelengths, the entire emission for every pixel of the entire image in the field of view can be recorded. The MSI provides a "data cube" of spectral information of the entire image at each wavelength of interest. 42 It has the capability to acquire spatial and spectral information of the entire small animal at NI range. 43 Comparing the spectral differences between different fluorescent probes and background signals, target spots can be extracted. Besides 2-D information, MSI also provides the spectral dimension as z-axis. 44 As emission information at each specific wavelength are plotted on one coordination space, a multispectral data cube is formed. In the data cube, fðx; y; zÞ represents the gray value of the pixel corresponding to the coordinates ðx; y; zÞ. The normalized value of the pixel's grayscale is Fðx; y; zÞ ¼ fðx; y; zÞ − f min ðx; y; zÞ f max ðx; y; zÞ − f min ðx; y; zÞ . (1) Spectral curve is reconstructed by plotting f ðx; y; zÞ at each wavelength. Each image contains both spatial (x, y coordinate data) and spectral (z-intensity data) information. f min ðx; y; zÞ and f max ðx; y; zÞ represent the minimum and maximum of the sample data, respectively. Getting spectral curve is the most intuitive way of expressing spectral features. The cubic spline interpolation method is introduced to reconstruct the curve.

Multispectral unmixing
When capturing in-vivo fluorescent images, the sensitivity is restricted by autofluorescence from skin and organ. This limitation makes it difficult to locate the fluorophores of interest accurately. At the same time, it is necessary to monitor a variety of biological processes simultaneously. It is necessary to use multiple fluorescent probes to label different molecules. Multispectral unmixing can remove autofluorescence and separate multiple fluorophores of interest. The unmixing algorithm assumes that the fluorescence spectrum measured is a stack up of several pure spectra multiplied by a weighting factor. The weighing factor is determined by the local concentration, excitation efficiency, and relative luminance of the emission fluorescence. The linear model is described as below: where M represents the measured spectra data matrix for each pixel (m columns: m pixels; n rows: n wavelengths). S represents the pure spectra data matrix of each fluorophores of interest and autofluorescence (k columns: k fluorophores; n rows: n wavelengths). A represents the weighing factor matrix (m columns: m pixels; k rows: k fluorophores). M is the measured multispectral data. R represents the matrix error. To solve this linear equation, the following conditions must be satisfied. The spectral detection channels must be greater than the number of fluorophores in the sample. Autofluoresence must also be calculated as a fluorophore. The pure spectra of fluorophores need to be obtained beforehand. This equation can be calculated by least squares method.
The pure spectra of fluorophores can be obtained from the spectral library directly or extracted from multispectral fluorescence images.

Target spot center determination
The mathematical model of fluorescent spot energy distribution is as below: where E max is the maximum light intensity, ðx c ; y c Þ is the coordinates of the fluorescent spot energy center, and a x , a y are the major and minor axes of spot intensity distribution. The image grayscale distribution function of the light spot on the camera imaging plane is where h 1 , h 2 are the row vectors of H −1 , H ¼ MR. M is internal matrix of camera. R is the rotation matrix. P is the homogeneous coordinates of spot center. The pixel at the imaging plane corresponding to the energy center of the spot is the maxima point of the grayscale distribution function. At gray image Iði; jÞ, the gray weighted center ðx 0 ; y 0 Þ is where Wði; jÞ is the weights, Wði; jÞ ¼ Iði; jÞ in general. The method chooses gray gravity method to process the fluorescence spot after multispectral unmixing.

3-D Positioning Based on Binocular Stereovision Theory
Vision measurement system is based on the image information acquired by the camera to calculate the 3-D position of the target. The paper presents a method to obtain 3-D coordinates of the fluorescence targets based on binocular stereovision theory.

Camera calibration
The correspondence between the 3-D spatial position of a point and its 2-D plane position is determined by the geometry of the camera. The parameters of the geometric model are called the camera parameters. Camera calibration is the process of calculating camera parameters by experiments that determines the internal and external parameters of the camera. In this paper, OpenCV functions 45 are used to calculate internal parameters matrix M based on Zhang. 46 In the stereo vision system, it is necessary to access the relative position of the two cameras (or a single camera at different perspectives). The distortion parameter D is solved by Brown's method. 47

Binocular stereovision
The basic principle of stereo vision is used to observe the same target from two (or more) points of view. Images are acquired under different perspectives. The 3-D information about the target is solved via the triangulation principle. In this study, the small animal is placed on the rotary platform, and the fixed CCD camera takes photos on different rotary angles. Since the rotary platform and the CCD camera move simultaneously, the process is also under view as the CCD camera rotating around the small animal, while the small animal is fixed. Thus, the equivalent convergent stereoscopic model can be obtained. It can be converted to parallel stereoscopic models for further computing. 48 The modules of the rotary platform, the equivalent convergent stereoscopic, and the parallel binocular stereovision are shown in Fig. 3.

3-D positioning
Epipolar constraint has an important role in stereo matching algorithm. It specifies the stereo matching process and affects the efficiency greatly. For the corrected images, the calculated coordinates of the target spot center should have the same v coordinates. After solving the reprojection matrix Q, the 3-D coordinates of the targets can be calculated by obtaining the 2-D coordinates ðu; vÞ on imaging plane and the associated disparity d.
The 3-D coordinates of the target is (X∕W, Y∕W, Z∕W) under the left camera coordination system. When the center coordinates of the target spot at left camera image is ðu 1 ; v 1 Þ, and the coordinates at right camera image is ðu 2 ; v 2 Þ, the dis-

Camera Calibration
The proper calibration checkerboard is printed according to the size of the field of view. In the bright field, the checkerboard is placed on the rotary platform at different angles and positions. After fixing, a set of images is captured with an angle of 0 and 4 deg, ensuring that there is no relative movement between checkerboard and the rotary platform. Open CV functions are used to calculate the internal parameters matrix M and the distortion parameter D. Obtain the rotation matrix R and the translation vector T. The reprojection matrix Q for 3-D location also needs to be calculated.

Target Extraction
To test the performance of our target extraction algorithm based on multispectral imaging, three different quantum dots with emission spectrum 720, 770, and 840 nm are chosen. The quantum dots are hypodermically injected into mice at three different positions. All three positions are around the dorsum of the mice. The fluorescent images are captured by designed in-vivo fluorescent sensing system. Image data are shown in Fig. 4 with the methods described above for fluorescent signal unmixing and extraction. The result is shown in Fig. 5. For each different quantum dots, pseudocolor is imposed for result demonstration. Thus, this method can pick out target signals with different spectral properties well. Gray gravity method is used to calculate the center of the fluorescence spot after multispectral unmixing. The result is shown in Fig. 6.

Quantitative Analyses Using Tissue Equivalent Material
In order to simulate the role of light absorption and scattering of small animal's tissue, equivalent material with similar optical  properties is selected for simulation experiment. We select agargel, and the preparation method is as follows: First of all, 2.4 g agarose is added in 110 ml phosphate buffer (phosphate-buffered Saline), stirring constantly while heating, until clear bubble-free solution is obtained. The solution is cooled to 70°C, shaken the beaker gently, and added 10.5 ml 20% fat emulsion into it. The solution is poured out, cooled, and solidified. The solidified agar-gel is cut to the shape shown in Fig. 7(a). A shallow hole is dug on the arc surface, used for quantum dot injection. Agar-gel slices with thickness of 2, 4, and 6 mm are prepared. The slices are used for certain depth simulation by covering on the arc surface [see Fig. 7(b)]. About 3 μl quantum dots (QDs) solution is injected into the shallow hole on the arc surface. The agar-gel without covering is   As can be seen, with the increase of slice's thickness, the accuracy of the data has a certain reduction but very slightly. This experiment simulates the in-vivo fluorescent target at different depths.

3-D Positioning
After obtaining the 2-D center coordinates of the target spot at both the left and right cameras, the 3-D coordinates of it under the left camera coordinate system can be acquired according to Eq. (7) and the calibration results. To obtain a more intuitive position, a standard checkerboard is placed on the rotary platform to create a reference coordinate system. Select feature points and the relationship between the references coordinates system and the left camera coordinates system can be learned. The target coordinates conversion from left camera system to the reference system is achieved. The 3-D coordinates of the three targets shown in Fig. 4

Performance of Detecting Depth
We tested the capability of deep target detecting on our designed in-vivo fluorescent sensing system by comparing commercialized machine results (Maestro™ CRi, USA). We tested the penetration of Au:CdHgTe quantum dots in muscle and adipose tissue. Au:CdHgTe quantum dots are NI gold-doped CdHgTe QDs with higher photoluminescence and lower cytotoxicity. In the middle of a 96-holes plate, 50 μl Au:CdHgTe quantum dots (emission peak at 840 nm) were added and covered by the muscle tissue. The results show that penetration depth is improved both in muscle and adipose tissue by our designed system. Figure 8 shows the penetration of Au:CdHgTe quantum dots in the muscle tissue. The images at the row (a) are captured using Maestro™ CRi instrument. The row (b) shows our system obtained images. The muscle tissue above the quantum dots is 0 mm (I), 20 mm (II), 25 mm (III), 30 mm (IV), respectively. Figure 9 shows a similar experiment that tests the penetration in the adipose tissue. The adipose tissue above the quantum dots is 30 mm (I) and 35 mm (II) thick. As can be seen, the difference Fig. 7 The shape of tissue equivalent material for quantitative analyses (a). The slices cover on the arc surface for certain depth simulation (b). between two systems is insignificant when the target depth is shallow. However, with the increase of the covering tissue's thickness, the proposed system shows better imaging results. The experiment shows that when the target is quite deep, the penetration depth is increased by 20% in muscle tissue and increased by 16% in adipose tissue, using the proposed system compared with a commercialized machine results (Maestro CRi).

Conclusion
In this paper, the development of a NI in-vivo fluorescence imaging system is presented, and an algorithm for the 3-D coordinates computing based on binocular stereovision theory is demonstrated. The system includes a deep cooling CCD, liquidcore fiber, high-energy laser source, rotary platform, and anesthesia apparatus to realize a dependable system and to detect fluorescence signal from deep sources with high sensitivity. This system can provide more accurate target location by calculating 3-D coordinates. The algorithm has several features as image preprocessing, multispectral unmixing, target spot center calculating, camera calibration, stereo calibration, and 3-D positioning. The validity of the algorithm has been verified by the experiments on the mouse. The designed 3-D positioning will provide more accurate target location information to help researchers analyze the target metastasis as well as determine the relative positional relationship between the target and the organs.
Experimental results of the mouse and pork meat with NIR quantum dots demonstrated that the potential of the designed imaging system for in-vivo optical diagnostics from deep sources and accurate position calculation. It is expected that this NI in-vivo fluorescence imaging system will show its advantage in tumor marker detecting, drug tracking, and also in pharmacokinetic studies as well as in the assessment of tissue response to therapy. Fig. 9 The penetration of Au:CdHgTe quantum dots in the adipose tissue. The adipose tissue is 30-mm thick in column (I) and 35-mm thick in column (II). The row (a) is MaestroTM CRi captured photographs. Images at the row (b) are obtained using our designed imaging system.