1.

Introduction

Despite the fact that robot-assisted laparoscopic surgery can provide value in many surgical applications (e.g., gynecologic,¹ liver,² and rectal³ surgery), it is still most commonly used for the surgical management of prostate cancer.⁴ In addition to the removal of the primary prostate cancer (prostatectomy), such procedures are often accompanied by a form of lymph node dissection focused on the removal of lymphatic metastases, e.g., a (extended) lymphadenectomy, a sentinel node (SN) procedure,⁵ or, in the future, a prostate-specific membrane antigen-targeted nodal dissection.⁶

It is widely accepted that image guidance toward the (possible) location of prostate cancer metastases may help improve surgical accuracy and therefore reduce the procedure-associated side effects.⁷ It is for this reason that the da Vinci robotic platform (Intuitive Surgical Inc., Sunnyvale, California) is now routinely equipped with a near-infrared (NIR) fluorescence laparoscope (Firefly, Intuitive Surgical Inc.). The concept here is that fully integrated visual identification of the target, using fluorescence guidance, helps the urologist to interpret the surgical field and, as such, improves surgical guidance. Unfortunately, fluorescence guidance has a very limited in-depth signal penetration due to absorption and scatter of the fluorescence excitation and emission light by tissue (roughly $<1\mathrm{cm}$).^8,9 This makes the technology lose value when lesions are located at unknown locations hidden deep within the patient’s anatomy. We underlined this shortcoming during robot-assisted SN biopsy procedures that made use of a tracer that is both radio- and fluorescence-labeled [indocyanine green $(\mathrm{ICG})\text{-}{}^{99\mathrm{m}}\mathrm{Tc}\text{-}\text{nanocolloid}$].^10,11 These studies illustrated that, dependent on the camera system, up to 19.6% of the SNs were missed during the urologist’s fluorescence-based inspection of the surgical field. Here, insight with regard to the tracer location, provided by the preoperative single-photon emission computed tomography/X-ray computed tomography (SPECT/CT) images, enabled the urologist to also resect the SNs not directly visible using fluorescence imaging. Ex vivo fluorescence imaging then confirmed the presence of the fluorescence signal in the resected SNs.

Based on the preoperatively acquired imaging information, surgeons can cognitively plan their route toward target structures [e.g., tumorous tissue or (tumor positive) lymph nodes] and around delicate structures (e.g., blood vessels, nerves, and/or ureters). Unfortunately, applying this information in the operating room (OR) remains challenging. The link between using pre- and intraoperative imaging methods may be strengthened using surgical navigation (via e.g., optical patient tracking), allowing a technological integration of pre- and intraoperative findings. Applying this technology to soft tissue environments is challenging due to tissue shift and deformation. Here, movement after acquisition of the patient scan(s) poses the biggest problem.^12–15 Such deformations can only be corrected using intraoperative imaging findings and/or reference marker points placed at the tissue of interest. For example, a radioactive readout can be used to confirm the navigation accuracy or even provide an intraoperative freehand SPECT (fhSPECT) scan that allows for navigation in a three-dimensional (3-D) “snapshot” map generated in the OR setting.^16–19 Alternative to (or in conjunction with) using a radioactive signature for intraoperative confirmation of the navigation accuracy, validation in the form of fluorescence imaging can be used.^20,21

In this study, we describe the integration of the above features in a robotic setup using (human-like) phantoms. For this, we used both SPECT/CT and fhSPECT findings as the basis for the navigation process. Additionally, we have integrated these data sets into the laparoscopic video feed using an augmented reality overlay. Such a “hybrid navigator” concept also means that in the future, different pre- and intraoperative imaging sources can be integrated in an interactive augmented reality view that can be fed into the console of the operating urologist.

2.

Methods and Materials

2.1.

Navigated Fluorescence Laparoscope Setup

The complete navigation setup consisted of a combination of the Firefly laparoscope, integrated with the da Vinci^® Si surgical robot, and the declipseSPECT navigation system (SurgicEye GmbH, Munich, Germany) [Fig. 1(a)] with integrated NIR optical tracking system (OTS; Polaris Vicra, Northern Digital Inc., Waterloo, Canada), which provides a tracking accuracy $<0.5\mathrm{mm}$ [95% confidence interval; 0.25 mm root mean square (RMS)²²]. Different reference targets were used to optically track the 3-D position and orientation (3-D pose) of objects, such as the laparoscope, gamma probes, and the different phantom setups. All reference targets contained a unique asymmetric geometry of at least three fiducials visible to the OTS.

Fig. 1

Overview navigation setup and underlying coordinate transformations. (a) Schematic overview of the navigation setup showing the navigation system with the overhead camera and OTS positioned above the patient. Both patient and fluorescence laparoscope are tracked with a unique reference target. Transformations from both the PRT and LRT coordinate systems to the OTS coordinate system are displayed with the red arrows. (b) Zoom-in of the PRT and LRT fixed to the patient and laparoscope, respectively. The green and blue arrows in the lower left corner are figurative and describe the transformation from the SPECT data coordinate system to that of the CT data and from the CT data to the PRT, respectively. The blue ${}^{\mathrm{LRT}}{T}_{\mathrm{CAMERA}}$ transformation arrow is calculated in the calibration shown in (c) and the blue ${}^{\mathrm{LRT}}{T}_{\mathrm{LTIP}}$ transformation arrow is calculated in the calibration shown in (d), with LTIP indicating the laparoscope’s tip. (c) Schematic overview of the coordinate transformations in the checkerboard calibration step. Both of the blue transformation matrices shown here are found in this calibration step. The green transformation is given to the navigation system (here, CB stands for checkerboard and CBRT stands for checkerboard reference target). (d) Schematic overview of the coordinate transformations in the tip calibration step. This calibration step determines the blue transformation matrix ${}^{\mathrm{LRT}}{T}_{\mathrm{LTIP}}$. Similar as stated above, the green transformation is given to the navigation (here, CRT stands for calibrator reference target). 3-D models of dummy human male (by user Areshero) and hospital bed (by user Felipe Ospina Ochoa) come from the community library of grabcad.com.

To allow for the navigation device to describe all the different objects (e.g., surgical target and laparoscope) in the surgical workflow, all objects had to be connected by placing them in a common coordinate system provided by the OTS.²³ Transforming a 3-D object pose from one coordinate system to another was achieved using $4\times 4$ transformation matrices. An overview of the different transformations needed in this setup is illustrated in Fig. 1. Estimation of these transformations is described in more detail throughout the text below.

2.5.

Soft Tissue Phantom

The soft tissue phantom setup [Fig. 2(a)] consisted of a tray with the PRT placed on a stand and a silicone half sphere with four Eppendorf tubes (Eppendorf AG, Hamburg, Germany) placed inside. The silicone half sphere structure, with a diameter of $\sim 15\mathrm{cm}$ at its base and a height of $\sim 5\mathrm{cm}$, was cast from Dragon Skin^® FX-Pro silicone rubber (Smooth-On Inc., Macungie, Pennsylvania), with addition of Thi-Vex^® Silicone Thickener (Smooth-On Inc.) for the skin part and Slacker^® Tactile Mutator (Smooth-On Inc.) for the softer fat part. SilTone pigments (FormX, Amsterdam, The Netherlands) were used to color the rubber.

Fig. 2

Navigation of the fluorescence laparoscope based on preoperative SPECT/CT imaging and intraoperative fluorescence guidance as performed on the tissue-mimicking phantom setup. (a) Volume rendering of the fused SPECT/CT scan of the tissue-mimicking phantom setup. The SPECT hotspots are shown in purple/blue. (b–f) Augmented reality video feed from the perspective of the fluorescence laparoscope as seen on the navigation device, when navigating to the navigation targets. These targets are indicated as gray and green circles and are defined by the SPECT signal (displayed purple/blue). The distance shown is the distance from the tip of the camera to the selected (green) target. The CT overlay is shown in white/gray. (b) The phantom shown by white-light modus, without augmented SPECT/CT overlay. (An error states that the mobile camera is not tracked due to the LRT being just at the border of the tracking volume.) (c) White-light modus with augmented SPECT/CT overlay. (d and e) Fluorescence light modus with augmented SPECT/CT overlay. In this mode, the background video is shown in gray. (f) Fluorescence light modus without augmented SPECT/CT overlay. Fluorescence is displayed in green.

Two $260\text{-}\mu \mathrm{L}$ Eppendorf tubes served as the navigation targets, to be scanned with (fh)SPECT imaging, and were placed at roughly 5 cm apart. With a volume of $260\mu \mathrm{L}$, the Eppendorf tubes should resemble a typical lymph node size found in lymph node dissection for prostate cancer; KleinJan et al.²⁵ reported a range of 80 to $860\mu \mathrm{L}$, with a median of $170\mu \mathrm{L}$. These tubes were filled with a mixture of ICG and ${}^{99\mathrm{m}}\mathrm{Tc}:253\mu \mathrm{L}$ of an ICG solution and $7\mu \mathrm{L}$ of a ${}^{99\mathrm{m}}\mathrm{Tc}$ solution ($\sim 5\mathrm{MBq}$). The ICG solutions used, in this study, consisted of $31.25\mu \mathrm{g}/\mathrm{mL}$ ICG (ICG-Pulsion, Pulsion Medical Systems, Munich, Germany) dissolved in human serum albumin (Albuman $200\mathrm{g}/\mathrm{l}$, Sanquin, Amsterdam, The Netherlands). With this as stock solution, the effective ICG solutions used in the phantom experiments range from 9.62 to $31.03\mu \mathrm{g}/\mathrm{mL}$. This quantity was chosen based on both clinical relevance (0 to $18.10\mu \mathrm{g}/\mathrm{mL}$ typically found in lymph nodes during prostatectomy²⁵) and best visibility during fluorescence guidance (best visibility found between 2.44 and $78.27\mu \mathrm{g}/\mathrm{mL}$²⁶). The ${}^{99\mathrm{m}}\mathrm{Tc}$ solutions used, in this study, were obtained from a ${}^{99\mathrm{m}}\mathrm{Tc}\text{-}\text{sodium}$ pertechnetate solution ($750\mathrm{MBq}/\mathrm{mL}$ saline, Technekow, Mallinckrodt Medical BV, Petten, The Netherlands). Two additional $500\text{-}\mu \mathrm{L}$ Eppendorf tubes were hidden on the sides of the tissue mimicking structure to serve as “anatomical reference” in the CT images. These tubes were filled with CT contrast agent [Ultravist $300\mathrm{mg}/\mathrm{mL}$ (Bayer AG, Leverkusen, Germany) 50% diluted with demineralized water].

To constrain movement of all the objects in the phantom setup, and therefore constrain the navigation error, the silicone structure and the PRT stand were glued to a silicone bottom layer in the tray. The PRT itself was taped to the stand. Note however, to allow easy access to the target sources, the precut phantoms were opened with a wound spreader during navigation, a manipulation that could have possibly introduced a (minor) deformation error to the navigation setup.

3.

Results

Figure 2 shows an example of the navigation process performed using preoperative SPECT/CT; Figs. 2(b) to 2(f) show snapshots of the Firefly video feed combining the laparoscopic view (in white-light and fluorescence modus) and an augmented overlay of the SPECT/CT data. The navigation targets, displayed as dots, are defined by the signal hotspots found in the SPECT imaging data and the two white Eppendorf-shaped signals in the CT view [Figs. 2(a), 2(c), and 2(d)] function as anatomical markers. In a similar manner, we could also use “intraoperative” fhSPECT data sets for navigation (see Fig. 3).

Fig. 3

Navigation of the fluorescence laparoscope based on intraoperative fhSPECT imaging and intraoperative fluorescence guidance as performed on the tissue-mimicking phantom setup. (a) Acquisition of the fhSPECT scan with a tracked gamma probe from the perspective of the overhead navigation system camera. (b) Acquired fhSPECT scan (shown in purple/blue) displayed as an augmented reality projection over the phantom setup. The distance shown is the distance from the tip of the laparoscope to the selected (green) navigation target, defined by the fhSPECT signal hotspots. (A red sign in the upper left corner indicates that the gamma probe is no longer in the tracking volume, since it is no longer in use after the scan acquisition.) (c) Video feed from the perspective of the fluorescence laparoscope in fluorescence light modus, with augmented fhSPECT overlay. (d) Video feed from the perspective of the fluorescence laparoscope in fluorescence light modus, without augmented fhSPECT overlay.

In the laparoscopic navigation view, the CT and/or (fh)SPECT overlay could be turned on and off independent of each other and the size of the SPECT hotspots (threshold on the SPECT signal) could be scaled according to preference. Qualitatively, navigation to the different targets (depicted as green colored dots) in the soft tissue phantoms appeared accurate using both scan modalities; during the navigation process, the augmented hotspots in the video feed and the distance to the targets seemed well registered to the fluorescent findings. As mentioned in Sec. 2.4, optical wavelengths of the OTS and fluorescence emission overlapped. However, other than in previous studies,²¹ the NIR light of the OTS did not interfere with the fluorescence detection of the Firefly.

The navigation accuracy for the SPECT/CT- and fhSPECT-based navigation procedures was quantified (see Table 1). Average errors for the SPECT/CT-based navigation were 2.25 mm (median 2 mm) and 2.08 mm (median 2 mm) in the coronal and sagittal plane, respectively. For the fhSPECT-based navigation, these were quite similar: 1.92 mm (median 2 mm) and 2.83 mm (median 3 mm). These results underline that the navigation accuracy, in the setup studied, stays well below the 1-cm tissue limit needed for successful fluorescence detection.^8,9 Comparison of the errors found for both scan modalities did not result in a significant difference ($p$ values of 0.764 and 0.282 for the coronal and sagittal plane, respectively).

Table 1

Overview of the accuracy measurements in the tissue-mimicking phantom setups for navigation based on both SPECT/CT and fhSPECT.

Evaluation of the navigation setup using a torso phantom (Fig. 4) illustrates the clinical setup to which the navigation should be translated. In addition, it demonstrates the setup remained effective when the robot is fully docked.

Fig. 4

Evaluation of the combined fluorescence navigation system based on preoperative SPECT/CT. (a) Operation room setup with the navigation system, fluorescence laparoscope, the surgical robot, and the torso phantom. On both the torso phantom and the fluorescence laparoscope, a reference target is attached. (b) Volume rendering of the fused SPECT/CT scan of the torso phantom setup. The SPECT hotspots are shown in pink/blue. The arrows indicate the prostate (P) and 3 lymph nodes (LN). (c–f) Augmented reality video feed from the perspective of the fluorescence laparoscope as seen on the navigation device when navigating to the navigation targets. These targets are indicated as gray and green circles and are defined by the SPECT signal (displayed purple/blue). The distance shown is the distance from the tip of the laparoscope to the selected (green) target. (c) White-light modus; letters indicating the specific lymph nodes (LN 1 and LN 2) and the prostate (P) in the anatomy of the phantom (d) White-light modus with augmented SPECT overlay. (e) Fluorescence light modus with augmented SPECT overlay. (f) Fluorescence light modus without the augmented SPECT overlay. Lymph node LN 1 is located too close to the highly active prostate P ($1:100$ ratio in radioactivity) to resolve it as a separate navigation target by the used rendering.

4.

Discussion

We have integrated surgical navigation, made possible via optical tracking, with (robot-assisted) laparoscopic fluorescence-guided surgery. Hereby, a robot-docked fluorescence laparoscope could be successfully navigated to target structures located in two different phantom setups with an accuracy of $\sim 2\mathrm{mm}$. All different imaging modalities used (fluorescence, SPECT/CT, and fhSPECT) could be available during navigation, thereby allowing compensation for the weaknesses of the individual modalities (e.g., the limited tissue penetration of fluorescent signals or the inaccuracies of surgical navigation due to soft tissue deformations). The presented soft tissue navigation setup suggests a next step in providing surgeons with more precise orientation and localization during robot-assisted procedures.

Theoretically, a preoperatively acquired SPECT/CT scan can be considered more precise than an intraoperatively acquired fhSPECT; with SPECT/CT, a lot more information is collected for the 3-D image reconstruction due to larger gamma cameras and longer scan times.¹⁸ Therefore, for a rigid phantom setup, one would expect the navigation errors found when navigation was based on SPECT/CT to be smaller than when navigation was based on fhSPECT. In our measurements, performed on the semirigid soft tissue phantoms, navigational accuracy was similar for both the two scan methods. This suggests that navigation based on fhSPECT may serve as a valuable and cost-effective alternative for navigation based on SPECT/CT. Moreover, due to the intraoperative nature of fhSPECT imaging (e.g., using a drop-in gamma probe^31,32), the technique should suffer much less from tissue deformations that result from patient movement. In fact, due to the short acquisition time (roughly between 2 and 3 min), one could easily acquire a new scan after any expected tissue deformation or displacement (e.g., removal of the primary tumor site). In addition, intraoperative real-time visualization of the fluorescence can be used to compensate for navigational errors below $\sim 1\mathrm{cm}$.

Translation of the proposed navigation setup to clinical use should be straightforward, as the robotic setup remains identical. We do see two questions that require attention, depending on the specific clinical application chosen: (1) Will NIR optical tracking be sufficient for a dynamic robotic intervention? (2) What will be the range of tissue deformation found in patients and how does this relate to the fluorescence detection limit?

Regarding the first point, in this study, optical tracking was used to define the 3-D pose of the phantom setup, laparoscope, and gamma probe using three different reference targets containing at least three fiducials each. As shown, this was feasible, but a general limiting factor was that the OTS has to maintain a direct line-of-sight to at least three of the reference target fiducials to allow for 3-D pose determination of the tracked object.²³ A feature that limits the OR layout and logistics. This may be overcome by increasing the number of fiducials per reference target or by using multiple OTS cameras on different sides of the OR.³³ A possible drawback of using NIR optical tracking in combination with NIR fluorescence imaging is the overlap in spectra used for object tracking and fluorescence imaging. In previous work, we found that this could be a considerable issue when using an open surgery fluorescence camera.²¹ To our surprise, we did not detect such an extensive interference when the Firefly laparoscope was applied outside of the phantom body. This finding could provide a basis for solving the issues previously reported. This said, in a laparoscopic setting, the tracking light is not likely to interfere with the fluorescence detection, since fluorescence imaging is conducted inside the patient’s body.

Alternative to NIR optical tracking, electromagnetic (EM) tracking or mechanical tracking could also provide outcome. With an intrinsic accuracy up to $\sim 0.5\mathrm{mm}$ (RMS) for EM tracking versus $\sim 0.25\mathrm{mm}$ (RMS) for optical tracking,^22,34 EM tracking still seems quite promising. However, a major disadvantage with this technique is the susceptibility to distortions of the EM tracking field by nearby metal objects or EM interferences, rendering it much less accurate.³⁵ Although inaccurate, mechanical tracking of the robotic arms is possible by using forward kinematics to calculate the 3-D pose of the end effector such as the laparoscope or the surgical instrument.³⁶ Fuerst et al.³² greatly improve this accuracy to an approximate error of 0.2 mm by combining the mechanical tracking with visual-based tracking via the stereoscopic laparoscope video feed.

In the presented navigation workflow, the navigation targets are displayed as a two-dimensional augmented reality overlay superimposed in the laparoscopic video feed while the distance from the laparoscope tip to target is shown in numbers. When implemented in clinical use, it might be hard to get a good idea of how deep the target structures really lie. Therefore, the augmented reality visualization may in the future be enhanced by dynamic augmented cues (e.g., small arrows indicating both direction and distance) or more depth perception cues (e.g., as shown by Bichlmeier et al.³⁷ and Kutter et al.³⁸).

5.

Conclusion

In a phantom setup, both pre- and intraoperative SPECT data sets can be used to accurately navigate the da Vinci robot-integrated fluorescence laparoscope toward surgical targets. Hereby, a basis is generated for future robot-assisted navigation toward fluorescent lesions.