SignificanceSingle-chip imaging devices featuring vertically stacked photodiodes and pixelated spectral filters are advancing multi-dye imaging methods for cancer surgeries, though this innovation comes with a compromise in spatial resolution. To mitigate this drawback, we developed a deep convolutional neural network (CNN) aimed at demosaicing the color and near-infrared (NIR) channels, with its performance validated on both pre-clinical and clinical datasets.AimWe introduce an optimized deep CNN designed for demosaicing both color and NIR images obtained using a hexachromatic imaging sensor.ApproachA residual CNN was fine-tuned and trained on a dataset of color images and subsequently assessed on a series of dual-channel, color, and NIR images to demonstrate its enhanced performance compared with traditional bilinear interpolation.ResultsOur optimized CNN for demosaicing color and NIR images achieves a reduction in the mean square error by 37% for color and 40% for NIR, respectively, and enhances the structural dissimilarity index by 37% across both imaging modalities in pre-clinical data. In clinical datasets, the network improves the mean square error by 35% in color images and 42% in NIR images while enhancing the structural dissimilarity index by 39% in both imaging modalities.ConclusionsWe showcase enhancements in image resolution for both color and NIR modalities through the use of an optimized CNN tailored for a hexachromatic image sensor. With the ongoing advancements in graphics card computational power, our approach delivers significant improvements in resolution that are feasible for real-time execution in surgical environments.
Lymph node mapping is a routine procedure during numerous cancer surgeries, yet the intraoperative identification of affected lymph nodes remains a clinical challenge. In response to this, we've introduced a multispectral camera that covers the UV, visible, and NIR spectrum. This innovative device facilitates lymph node imaging using both endogenious UV fluorescence and exogenious introduced NIR fluorescence, aiding in pinpointing the affected nodes. This presentation will showcase relevant clinical findings.
SignificanceNear-infrared fluorescence image-guided surgery is often thought of as a spectral imaging problem where the channel count is the critical parameter, but it should also be thought of as a multiscale imaging problem where the field of view and spatial resolution are similarly important.AimConventional imaging systems based on division-of-focal-plane architectures suffer from a strict relationship between the channel count on one hand and the field of view and spatial resolution on the other, but bioinspired imaging systems that combine stacked photodiode image sensors and long-pass/short-pass filter arrays offer a weaker tradeoff.ApproachIn this paper, we explore how the relevant changes to the image sensor and associated image processing routines affect image fidelity during image-guided surgeries for tumor removal in an animal model of breast cancer and nodal mapping in women with breast cancer.ResultsWe demonstrate that a transition from a conventional imaging system to a bioinspired one, along with optimization of the image processing routines, yields improvements in multiple measures of spectral and textural rendition relevant to surgical decision-making.ConclusionsThese results call for a critical examination of the devices and algorithms that underpin image-guided surgery to ensure that surgeons receive high-quality guidance and patients receive high-quality outcomes as these technologies enter clinical practice.
The staging of solid cancers is critical to the planning of both primary treatment with surgery and adjuvant therapies like chemotherapy; however, staging is not always possible with preoperative information and may require intraoperative evaluation of sentinel lymph nodes to confirm or disaffirm the presence of metastasis. Challenges are presented by standard-of-care sentinel lymph node dissection which must be quick and accurate enough to guide the surgical strategy despite a workflow that stretches from the operating room to the pathology lab; however, a solution is posed by fluorescence-assisted sentinel lymph node dissection which uses fluorescent probes to communicate the location and/or status of sentinel lymph nodes, reducing the complexity of the surgery and/or eliminating the need for rapid pathology. In support of this emerging modality, we have constructed a snapshot hyperspectral imaging system with sensitivity from the far-red to the near-infrared that enables sentinel lymph node dissection with multiple near-infrared fluorophores. We have also developed a spectral unmixing routine for in vivo quantification of the readily available fluorophores indocyanine green and methylene blue that can be extended to emerging fluorophores that actively target tumor cells. Both the imaging system and the unmixing routine have been tested in a clinical setting where they have successfully discriminated two dyes exhibiting different distributions.
Access to the requested content is limited to institutions that have purchased or subscribe to SPIE eBooks.
You are receiving this notice because your organization may not have SPIE eBooks access.*
*Shibboleth/Open Athens users─please
sign in
to access your institution's subscriptions.
To obtain this item, you may purchase the complete book in print or electronic format on
SPIE.org.
INSTITUTIONAL Select your institution to access the SPIE Digital Library.
PERSONAL Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.