Surgeons rely on medical imaging to plan surgery and to accurately locate the lesions. Currently, many traditional modalities, such as x-ray radiography, computed tomography (CT), magnetic resonance imaging (MRI), ultrasound, positron emission tomography, and single-photon emission computed tomography, are widely used in daily clinical practice.1,2 These preoperative images greatly assist in disease diagnosis and surgical planning. However, preoperative imaging modalities, such as CT and MRI, suffer from several limitations, including difficulties in adapting these systems to the operating room (OR) environment as well as slow scan time. For abdominal surgeries, where organs typically deform, the practicality of using preoperative images for surgical planning presents unique challenges that require the development of new intraoperative imaging modalities.
Traditionally, fluoroscopy and intraoperative ultrasound (IUS) are deployed in the OR for guiding surgery.3,4 These imaging modalities provide real-time feedback to complement the preoperative imaging information available to the surgeon. Yet, fluoroscopy provides poor contrast for soft tissues, limiting its utility in abdominal surgery. IUS is a contact structural imaging method that suffers from low contrast and small field of view. Additionally, both fluoroscopy and IUS do not provide molecular and functional information that enhances the accuracy of oncologic imaging. Recent studies have shown that fluorescence imaging fills this void.56.7.–8
The development of diverse fluorescent exogenous molecular probes has enabled the reporting of intricate molecular and cellular processes at an unparalleled sensitivity.5,7,9,10 Recent efforts to adapt fluorescence microscopy to organ imaging have expanded fluorescence molecular imaging to the clinical arena.11,12 With near-infrared (NIR) probes, fluorescence imaging can penetrate deeper into tissues than visible light imaging because of the reduced scattering and absorption of NIR light.11,12
Leveraging the strengths of NIR fluorescence imaging, we have previously developed a prototype fluorescence goggle system to guide surgery.13 The prototype goggle system was a compact wearable fluorescence imaging and display system based on the CCD imaging technology and a monocular eyepiece display.13 The line-of-view of the surgeon was synchronized with the system, and hands free imaging was enabled.
In the current study, we report the development of a newer version of the goggle system using an NIR-sensitive fast complementary metal–oxide–semiconductor (CMOS) sensor and a video-optical, dual-modal see-through head-mounted display (HMD). We characterized the new system and tested it in surgical studies in rodents.
Materials and Methods
New Fluorescence Goggle System
Briefly, the new goggle system consists of a detector module, a display module, a control module, and a light source module. The detector module comprises CMOS imaging sensor (MT9V032, Micron Technology, Boise, Idaho) that is integrated onto a custom printed circuit board platform. An emission 830 nm long-pass filter (#8480, Omega Optical, Brattleboro, Vermont) is mounted on the objective lens (U15764, Tamron, Commack, New York). The control module, which interfaces between the detector module and a PC, consists of a field-programmable gate array (FPGA) integration module with USB communication capabilities (XEM 3050, Opal Kelly, Portland, Oregon) and a laptop computer. The data from the imaging sensor are read out in several stages. First, the data from the CMOS imaging array are read out via a state machine implemented on the FPGA, and the data are stored in a first in, first out (FIFO) implemented on the FPGA. The data from the FIFO are transferred to an SDRAM chip, which is housed on the Opal Kelly board. Two SDRAM chips are used to meet the needs of simultaneously acquiring from the imaging sensor and transferring the data to the PC. Specifically, one SDRAM chip stores the pixel data from the sensor, while the second SDRAM chip transfers the data to an output FIFO in the FPGA chip. The pixel data from the second FIFO are transferred to a computer via USB using Opal Kelly’s FrontPanel interface. In addition, pixel binning and temporal averaging methods are implemented as optional features.
The display module is a see-through HMD unit (ST1080, Silicon micro display, Cambridge, Massachusetts). The computer interfaces with the display module via the HDMI link, where C++ on Microsoft Visual Studio Video displays real-time images on the HMD via simple direct media layer application programming interface. The NIR light source module is composed of four 760 nm NIR light-emitting diodes (LEDs) (H2A1-H760; Roithner Lasertechnik, Vienna, Austria), an LED driver (BuckPuck 3021-D-I-1000; LEDdynamics, Randolph, Vermont), an excitation 775 nm short-pass filter (#NT64-615, Edmund Optics, Barrington, New Jersey), and 12 V Ni-MH battery power supply.
Fluorescence Imaging Sensitivity and Signal-to-Noise Ratio Characterization
The sensitivity of the goggle system to detect fluorescence signal was characterized using signal-to-noise ratio (SNR), which compared the level of desired signal relative to noise level. SNR performance evaluation of goggle device consisted of NIR light source, CMOS detector, three samples of indocyanine green (ICG) (Sigma-Aldrich, St. Louis, Missouri) in dimethyl sulfoxide (DMSO) (Sigma-Aldrich), and one sample of pure DMSO as a negative control. The goggle system was situated 50 cm away from the samples. A black background was used to minimize specular reflection.
Fluorescence emission signal at 830 nm from ICG samples and background signal from pure DMSO negative control samples were measured simultaneously using the detector. The experiment was conducted with different concentrations of ICG ranging from 100 pM to 1 μM. Images and videos were recorded and stored for further analysis using MATLAB.
SNR was computed using Eq. (1),14,15 where was the mean intensity value detected from the three ICG samples, was the mean intensity value detected from the control negative sample of pure DMSO, and was the standard deviation of intensity values detected from the ICG samples. The signal detected from the control negative sample represented background signal, which was in part due to specular reflection, nonideal filtering, and detector noises.
NIR Contrast Agents
ICG was purchased from Sigma-Aldrich. LS301 was synthesized by literature method.13 This dye has absorption and emission maxima around 780 and 830 nm, respectively, in biological media. PEG_QD800 are pegylated CdTeSe/ZnS quantum dots (QDs), which were synthesized in three steps. First, CdTeSe QDs were synthesized via a hot-injection approach16 under Ar. The resulting QDs were washed with methanol and centrifuged, and then redispersed in toluene for further shell growth. Second, the CdTeSe/ZnS QDs were then prepared by coating the CdTeSe shell with ZnS, and the product was purified by methanol and centrifugation, as described above.17 Finally, the PEG_QD800 was prepared by capping the CdTeSe/ZnS with polyethylene glycol-dihydrolipoic acid (PEG-DHLA) in methanol.18 The precipitates were dispersed in water, providing a clear homogenous dispersion of PEGylated QDs.
Sentinel Lymph Node Mapping
All animal procedures were conducted in compliance with Washington University Animal Studies Committee’s requirements for the care and use of animals in research. Female Sprague Dawley rats were purchased from Harlan Laboratories (Indianapolis, Indiana). For animal studies, the rodents were anesthetized with ketamine and xylazine cocktail. The hair overlaying the region of interest was removed by gentle clipping and cream depilatory (Sally Hansen, Morris Plains, New Jersey). For sentinel lymph node (SLN) mapping, rats () were anesthetized, and 10 μL QDs in PBS suspension (particle concentration 40 μM) were administered intradermally into both front paws (). The subcutaneous axillary region was exposed by incising and retracting the overlaying skin. SLNs were identified and resected under image guidance provided. After resection, surgical sites were inspected thoroughly with the goggle under different signal amplification settings. For comparison, ischiatic lymph nodes were also excised and imaged as nonfluorescent controls. SLNs and control lymph nodes were examined with the system ex vivo. Excised tissues were embedded in Tissue-Tek OCT medium (Sakura Finetek, Torrance, California) and stored at .
Fluorescence-Guided Liver Surgery and Intraoperative Imaging
About 100,000 4T1luc cells were implanted in the liver of six-week-old female Balb/c mice. Approximately 10 days after tumor implantation, mice were anesthetized with ketamine and xylazine. LS301 solution (100 μL of 100 μM in 20% DMSO/80% saline) was injected intravenously via lateral tail vein. At 24 h postinjection, fluorescence-guided exploratory laparotomy were performed using the new goggle system. A midline incision was made from sternum to chest, and tissues were reflected laterally to expose the liver. Cancerous tissues were identified and resected under image guidance. Surgical sites were then re-examined to find residual cancerous tissues. Resected tumor and healthy liver tissues were examined ex vivo to compare fluorescence in tumor to healthy liver. Tissues were subsequently embedded with Tissue-Tek OCT (Sakura Finetek), and frozen for histology.
The new goggle system is a compact see-through wearable system that enables real-time fluorescence imaging capability for up to 60 frames per second (fps) (Fig. 1). Compared to the CCD sensor used in the previous prototype (approximately 20% quantum efficiency at 830 nm), the CMOS sensor used in the current system has a high quantum efficiency at 830 nm (). The combination of high frame rate and high NIR quantum efficiency makes the sensor ideal for intraoperative NIR fluorescence imaging. The wearer has the option to choose either automatic or manual gain as well as either automatic or manual exposure time. The automatic gain and exposure time can present the optimized contrast to the surgeons without manually adjusting it during surgical procedures.
We demonstrated the first application of a video-optical dual-modal see-through display in NIR fluorescence imaging. When the video mode was turned off, the wearer could visualize the surroundings with natural vision. When the video mode was turned on, the real time NIR fluorescence video was presented to the wearer with high contrast. The flexibility of a switchable see-through display simplified the surgical workflow and allowed the surgeon to visualize the surgical bed with natural vision or enhanced vision as needed.
We first evaluated the sensitivity of the goggle system by measuring the SNR of different ICG concentrations ranging from 100 pM to 700 nM [Fig. 2(a)]. For this study, the system was operated with an exposure time of 15.25 ms and a frame rate of 28 fps. We demonstrated that the goggle could detect fluorescence signal from as low as 300 pM ICG. Noise analysis model of the CMOS imaging sensor19,20 indicated that photon shot noise dominated reset and readout noise at an SNR of 30, which corresponded to fluorescence signal from approximately 300 nM of ICG detected at 28 fps.
We also measured the SNR of the fluorescence signal using optional features including pixel binning and temporal averaging methods [Fig. 2(b)]. In this study, the system was operated with an exposure time of 45.72 ms and a frame rate of 10 fps. The SNR improved approximately by a factor of 2 when either binning or temporal averaging methods are implemented independently and approximately by a factor of 4 when both methods are implemented together. Pixel binning involved combining signals from a group of pixels in the spatial domain, which was analogous to increasing the number of photons that contribute to the detected signal. Binning improved the SNR by the square root of the number of pixels binned. Thus binning a neighborhood of improved the SNR approximately by a factor of 2. Temporal averaging involved combining signals from a group of pixels in time domain, which was also analogous to increasing the number of photons that contribute to the detected signal. Similarly, temporal averaging increased the SNR by the square root of the number of averaged pixels in time domain. Hence, temporal averaging of four consecutive image frames increased the SNR approximately by a factor of 2. Both temporal averaging and pixel binning were combined together to further improve the SNR of the sensor. Averaging four frames coupled with averaging a pixel neighborhood of improved the SNR approximately by a factor of 4, as expected.
In addition, we also evaluated sensitivity of the goggle system by measuring the SNR versus exposure time for 3 and 10 nM ICG samples. SNR increased linearly with exposure time at a rate that depends on ICG concentration [Fig. 2(c)]. This was due to the fact that for a longer exposure time, the photodiodes accumulated more photons and, hence, the contrast between the target and background increased linearly. As the exposure time increased, the SNR of the florescence signal increased linearly at the cost of a slower frame rate. For example, if the exposure time is set to 20 ms, the SNR is 4, and the frame rate is 50 fps. If the exposure time is 100 ms, the SNR is 20, and the frame rate is 10 fps.
After evaluating the sensitivity of the new goggle imaging system, we deployed it in surgical oncology using rodents. We first investigated the use of new system in SLNs mapping. PEGylated CdTeSe/ZnS QDs with emission at 800 nm (PEG_QD800) were used for this purpose because they do not extravasate rapidly from the lymphatic vessel. Upon injection, the goggle system rapidly detected the QD fluorescence signal at the axillary region in the rats within 30 min (Fig. 3). Under the goggle guidance, the lymph nodes were rapidly located and excised. The putative SLNs were also compared with the control lymph nodes ex vivo [Fig. 4(a)–4(c)]. The SLN-to-control fluorescence ratio is (). The SLNs were examined by histology, which confirmed that all of the six putative SLNs removed were actual SLNs [Fig. 4(d)].
Furthermore, we demonstrated the utility of new goggle system in intraoperative molecular imaging of cancer. For this purpose, a fluorescent molecular probe LS301 was used to detect breast cancer liver metastases.13 At 24 h postinjection, the fluorescence goggle successfully detected the metastases intraoperatively [Fig. 5(a)–5(c)]. In addition, small tumors of approximately 1 mm in diameter were also detected. These tumors were not evident by visual inspection with the naked eye or by palpation. Under the goggle guidance, the metastatic lesions were successfully located and removed. At the time of surgery, the liver also showed relatively higher fluorescence signal than other organs, because it is the major excretion organ for the molecular probes. After resection, the local fluorescence in the surgical bed decreased [Fig. 5(d)]. Ex vivo examination confirmed the in vivo findings [Fig. 6(a)–6(c)]. The histology further confirmed the cancerous nature of tissues excised, as evidenced by their epithelial inclusion [Fig. 6(d)].
In this study, we developed a new fluorescence goggle system with fast temporal resolution and dual-modal see-through display. The current system uses a NIR-sensitive CMOS sensor, unlike the CCD sensor reported in the previous goggle system.13 This CMOS sensor has a quantum efficiency of over 30% at 830 nm spectral window, which is ideal for NIR imaging. The picomolar level fluorescence detection sensitivity provides a powerful platform for imaging fluorescent molecular information intraoperatively. Furthermore, the CMOS technology facilitated imaging at a faster frame rate without the need of binning using the new goggle system. A frame rate of up to 60 fps was realized for in vivo imaging. Compared to the previous generation goggle system, which can operate at up to 30 fps, the frame rate of the new system has further improved to beyond video rate. This is peculiarly desirable for intraoperative imaging, particularly when organs of interests are in constant motion due to many physiological processes such as respiration. For liver surgery, high-temporal resolution is crucial to overcome the large range of liver motion.
The new goggle system features novel display technology. To the best of our knowledge, this is for the first time a dual-modal see-through display technology is used for intraoperative fluorescence imaging. As a wearable imaging and display system, an integral part of fluorescence goggle system is the display module. Leveraging on the existing technology platforms of HMDs, the goggle system is empowered to present the imaging data to the surgeon in a convenient way. HMDs have been extensively used for military purposes in the past.21 They have also been adapted for personal entertainment purposes in recent years. Currently, two types of HMDs have been developed including video see-through HMDs and optical see-through HMDs. In the current system, a switchable optical-video HMD is used. The new goggle system incorporates the capability to switch between the two display modes, a capability which was previously unavailable. This simplifies the surgical workflow for open surgery and enables the surgeon to use their natural vision as needed without taking off the goggle.
From the characterization study, we observed a trend that is in accordance with theoretical understanding of CMOS technologies. The intensity signal measured by a CMOS detector is given by Eq. (2), where is optical power of the incident light, is wavelength of the incident light, is photosensitive area of the pixel, is exposure time, QE is quantum efficiency of detector, is Planck’s constant, and is speed of light. Equation (2) reveals the linear dependence relationship between the intensity signal and quantum efficiency. Quantum efficiency of the current detector greater than 30% at 830 nm is highly desirable for NIR fluorescence imaging purposes. However, the temporal noise limits performance of the imaging system, which is generated during reset, integration, and readout operations. The total noise is governed by Eq. (3), where is reset noise, is readout noise, and is photon noise. Analysis of temporal noise in CMOS image sensors indicates that reset and readout noises dominate at low illumination, while photon noise dominates at high illumination.19,20 The ratio of signal to total noise is SNR, which describes the sensitivity performance of the detector. The number of photons incident on a pixel follows Poisson distribution. Therefore, photon noise has a square root relationship with the number of incident photons. Consequently, SNR of an image sensor that is limited by photon noise has a square root relationship with the number of incident photons. An increase in ICG concentration or exposure time increases the number of incident photons, and thus an improvement in SNR. In addition, pixel binning and temporal averaging are implemented as optional features in this goggle system. These features improve the SNR with trade-off in spatial and temporal resolutions. They may be used in extremely low light imaging conditions, as needed.
In conjunction with QDs, we demonstrated that the new goggle system could be used to map the SLN. QDs have several advantages over ICG that was used in the previous SLN study.13 The higher quantum yield and photostability render the QDs better imaging agent than ICG.10 The new goggle system is versatile, as demonstrated by its compatibility with diverse NIR tracers including small organic molecules and inorganic nanoparticles. Compared to the mouse models used in the previous study,13 the rat model used in this study reflects the scalability of the system and potential translation to human lymph biopsy.
Furthermore, we demonstrated the potential of combining the goggle system with a targeted molecular agent LS301 to guide resection of breast cancer metastases in the liver. Small lesions which could have been otherwise missed were readily detected. The intraoperative fluorescence image guidance complemented the naked eye observation and palpation, which were not sensitive to small lesions. In future study, the fluorescence imaging strategy could be coupled with IUS to further facilitate cancer detection. The high resolution of fluorescence imaging at the liver surface would complement IUS, which has difficulties in resolving surface-located, isoechoic, and small lesions.
Although our current findings are exciting, there is room for improvement. For instance, the current system cannot fully store all imaging data when operating beyond 30 fps, due to the limited bandwidth for data transfer imposed by USB 2.0. In the future, other data transfer standard with larger bandwidth will be utilized to solve this problem. Another way to overcome this challenge is to compress the data with data-compression chip. Another area for improvement is the control module, which currently consists of an FPGA board and a laptop. We plan to develop an embedded system to fully circumvent the need for a PC. Furthermore, the current system uses off-the-shelf dual-modal see-through HMDs primarily designed for entertainment purposes. Although it is useful for demonstrating the concept and feasibility, we will also develop a customized and ergonomical display unit optimized for medical imaging.
We have developed a new fluorescence goggle system based on CMOS sensor and see-through display technologies. The new prototype features picomolar fluorescence detection sensitivity and a fast temporal resolution of up to 60 fps. To the best of our knowledge, this is the first dual-modal see-through display used in intraoperative fluorescence imaging. We showed the feasibility of using this system to guide SLN mapping with NIR QDs. Furthermore, we demonstrated the potential value of the goggle system and tumor-targeted NIR fluorescence molecular probes in detecting and guiding resection of breast cancer metastases in the liver. Further optimization of the new goggle system holds promise to improve surgical planning, in serving various clinical needs in the future.
This study was supported in part by grants from the National Institutes of Health (NCI R01 CA171651 and NIBIB R01 EB008111; Samuel Achilefu) as well as the Air Force Office of Scientific Research (FA9550-10-1-0121 and FA9550-12-1-0321; Viktor Gruev). Yang Liu was supported by the US Department of Defense Breast Cancer Research Program Predoctoral Award (W81XWH-11-1-0059). Walter Akers was supported by Award Number K01RR026095 from the National Center for Research Resources. We thank Mrs. Kexian Liang for preparing LS301. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health and the Department of Defense.