Surgery is the primary curative option for patients with cancer, with the overall objective of complete resection of all cancerous tissue while avoiding iatrogenic damage to healthy tissue. Simultaneous imaging of weak fluorescence signals from multiple targeted molecular markers under bright surgical illumination is an unmet goal with current intra operating instrument. In this talk, I will describe our recent efforts in solving this intraoperative challenge by drawing inspiration from the visual system of the mantis shrimp – a compact biological system optimized for multispectral imaging. We have successfully designed, tested and clinically translated our bio-inspired imagers by monolithically integrating vertically stacked photodetectors with pixelated interference filters. The sensor is capable of recording color and NIR fluorescence from three different molecular markers and display this information using augmented reality goggles. The sensor resolution is 1280 by 720 and operates at 30 frames per second and has been used to simultaneously image tumor targeted dye IR800 and nerve targeted dye, Oxazine-4. Displaying this information in the operating room is a challenging feat. We have used variety of augmented reality displays and will provide overview of both pre-clinical and clinical translation of this technology.
Near infrared fluorescence (NIRF) based image guided surgery aims to provide vital information to the surgeon in the
operating room, such as locations of cancerous tissue that should be resected and healthy tissue that should to be
preserved. Targeted molecular markers, such as tumor or nerve specific probes, are used in conjunctions with NIRF
imaging and display systems to provide key information to the operator in real-time. One of the major hurdles for the
wide adaptation of these imaging systems is the high cost to operate the instruments, large footprint and complexity of
operating the systems. The emergence of wearable NIRF systems has addressed these shortcomings by minimizing the
imaging and display systems’ footprint and reducing the operational cost. However, one of the major shortcomings for
this technology is the replacement of the surgeon’s natural vision with an augmented reality view of the operating room.
In this paper, we have addressed this major shortcoming by exploiting hologram technology from Microsoft HoloLens to
present NIR information on a color image captured by the surgeon’s natural vision. NIR information is captured with a
CMOS sensor with high quantum efficiency in the 800 nm wavelength together with a laser light illumination light
source. The NIR image is converted to a hologram that is displayed on Microsoft HoloLens and is correctly co-registered
with the operator’s natural eyesight.