17 August 2016 Navigation of a robot-integrated fluorescence laparoscope in preoperative SPECT/CT and intraoperative freehand SPECT imaging data: a phantom study
Author Affiliations +
Robot-assisted laparoscopic surgery is becoming an established technique for prostatectomy and is increasingly being explored for other types of cancer. Linking intraoperative imaging techniques, such as fluorescence guidance, with the three-dimensional insights provided by preoperative imaging remains a challenge. Navigation technologies may provide a solution, especially when directly linked to both the robotic setup and the fluorescence laparoscope. We evaluated the feasibility of such a setup. Preoperative single-photon emission computed tomography/X-ray computed tomography (SPECT/CT) or intraoperative freehand SPECT (fhSPECT) scans were used to navigate an optically tracked robot-integrated fluorescence laparoscope via an augmented reality overlay in the laparoscopic video feed. The navigation accuracy was evaluated in soft tissue phantoms, followed by studies in a human-like torso phantom. Navigation accuracies found for SPECT/CT-based navigation were 2.25 mm (coronal) and 2.08 mm (sagittal). For fhSPECT-based navigation, these were 1.92 mm (coronal) and 2.83 mm (sagittal). All errors remained below the <1-cm detection limit for fluorescence imaging, allowing refinement of the navigation process using fluorescence findings. The phantom experiments performed suggest that SPECT-based navigation of the robot-integrated fluorescence laparoscope is feasible and may aid fluorescence-guided surgery procedures.



Despite the fact that robot-assisted laparoscopic surgery can provide value in many surgical applications (e.g., gynecologic,1 liver,2 and rectal3 surgery), it is still most commonly used for the surgical management of prostate cancer.4 In addition to the removal of the primary prostate cancer (prostatectomy), such procedures are often accompanied by a form of lymph node dissection focused on the removal of lymphatic metastases, e.g., a (extended) lymphadenectomy, a sentinel node (SN) procedure,5 or, in the future, a prostate-specific membrane antigen-targeted nodal dissection.6

It is widely accepted that image guidance toward the (possible) location of prostate cancer metastases may help improve surgical accuracy and therefore reduce the procedure-associated side effects.7 It is for this reason that the da Vinci robotic platform (Intuitive Surgical Inc., Sunnyvale, California) is now routinely equipped with a near-infrared (NIR) fluorescence laparoscope (Firefly, Intuitive Surgical Inc.). The concept here is that fully integrated visual identification of the target, using fluorescence guidance, helps the urologist to interpret the surgical field and, as such, improves surgical guidance. Unfortunately, fluorescence guidance has a very limited in-depth signal penetration due to absorption and scatter of the fluorescence excitation and emission light by tissue (roughly <1  cm).8,9 This makes the technology lose value when lesions are located at unknown locations hidden deep within the patient’s anatomy. We underlined this shortcoming during robot-assisted SN biopsy procedures that made use of a tracer that is both radio- and fluorescence-labeled [indocyanine green (ICG)-Tc99m-nanocolloid].10,11 These studies illustrated that, dependent on the camera system, up to 19.6% of the SNs were missed during the urologist’s fluorescence-based inspection of the surgical field. Here, insight with regard to the tracer location, provided by the preoperative single-photon emission computed tomography/X-ray computed tomography (SPECT/CT) images, enabled the urologist to also resect the SNs not directly visible using fluorescence imaging. Ex vivo fluorescence imaging then confirmed the presence of the fluorescence signal in the resected SNs.

Based on the preoperatively acquired imaging information, surgeons can cognitively plan their route toward target structures [e.g., tumorous tissue or (tumor positive) lymph nodes] and around delicate structures (e.g., blood vessels, nerves, and/or ureters). Unfortunately, applying this information in the operating room (OR) remains challenging. The link between using pre- and intraoperative imaging methods may be strengthened using surgical navigation (via e.g., optical patient tracking), allowing a technological integration of pre- and intraoperative findings. Applying this technology to soft tissue environments is challenging due to tissue shift and deformation. Here, movement after acquisition of the patient scan(s) poses the biggest problem.1213.14.15 Such deformations can only be corrected using intraoperative imaging findings and/or reference marker points placed at the tissue of interest. For example, a radioactive readout can be used to confirm the navigation accuracy or even provide an intraoperative freehand SPECT (fhSPECT) scan that allows for navigation in a three-dimensional (3-D) “snapshot” map generated in the OR setting.1617.18.19 Alternative to (or in conjunction with) using a radioactive signature for intraoperative confirmation of the navigation accuracy, validation in the form of fluorescence imaging can be used.20,21

In this study, we describe the integration of the above features in a robotic setup using (human-like) phantoms. For this, we used both SPECT/CT and fhSPECT findings as the basis for the navigation process. Additionally, we have integrated these data sets into the laparoscopic video feed using an augmented reality overlay. Such a “hybrid navigator” concept also means that in the future, different pre- and intraoperative imaging sources can be integrated in an interactive augmented reality view that can be fed into the console of the operating urologist.


Methods and Materials


Navigated Fluorescence Laparoscope Setup

The complete navigation setup consisted of a combination of the Firefly laparoscope, integrated with the da Vinci® Si surgical robot, and the declipseSPECT navigation system (SurgicEye GmbH, Munich, Germany) [Fig. 1(a)] with integrated NIR optical tracking system (OTS; Polaris Vicra, Northern Digital Inc., Waterloo, Canada), which provides a tracking accuracy <0.5  mm [95% confidence interval; 0.25 mm root mean square (RMS)22]. Different reference targets were used to optically track the 3-D position and orientation (3-D pose) of objects, such as the laparoscope, gamma probes, and the different phantom setups. All reference targets contained a unique asymmetric geometry of at least three fiducials visible to the OTS.

Fig. 1

Overview navigation setup and underlying coordinate transformations. (a) Schematic overview of the navigation setup showing the navigation system with the overhead camera and OTS positioned above the patient. Both patient and fluorescence laparoscope are tracked with a unique reference target. Transformations from both the PRT and LRT coordinate systems to the OTS coordinate system are displayed with the red arrows. (b) Zoom-in of the PRT and LRT fixed to the patient and laparoscope, respectively. The green and blue arrows in the lower left corner are figurative and describe the transformation from the SPECT data coordinate system to that of the CT data and from the CT data to the PRT, respectively. The blue TCAMERALRT transformation arrow is calculated in the calibration shown in (c) and the blue TLTIPLRT transformation arrow is calculated in the calibration shown in (d), with LTIP indicating the laparoscope’s tip. (c) Schematic overview of the coordinate transformations in the checkerboard calibration step. Both of the blue transformation matrices shown here are found in this calibration step. The green transformation is given to the navigation system (here, CB stands for checkerboard and CBRT stands for checkerboard reference target). (d) Schematic overview of the coordinate transformations in the tip calibration step. This calibration step determines the blue transformation matrix TLTIPLRT. Similar as stated above, the green transformation is given to the navigation (here, CRT stands for calibrator reference target). 3-D models of dummy human male (by user Areshero) and hospital bed (by user Felipe Ospina Ochoa) come from the community library of grabcad.com.


To allow for the navigation device to describe all the different objects (e.g., surgical target and laparoscope) in the surgical workflow, all objects had to be connected by placing them in a common coordinate system provided by the OTS.23 Transforming a 3-D object pose from one coordinate system to another was achieved using 4×4 transformation matrices. An overview of the different transformations needed in this setup is illustrated in Fig. 1. Estimation of these transformations is described in more detail throughout the text below.


Single-Photon Emission Computed Tomography/X-Ray Computed Tomography

Before acquisition of the SPECT/CT scan, a three-fiducial phantom reference target (PRT) was fitted to the exterior of the phantom setups, allowing for optical tracking by the OTS. It remained at a fixed position with respect to the phantom throughout the experiments. Next to its visibility to the OTS, this PRT is also clearly distinguishable in the CT imaging data, allowing the navigation system to segment it from the CT imaging data and calculate the transformation TCTPRT from the CT coordinate system to the PRT coordinate system20 [Fig. 1(b)]. Since the SPECT/CT imaging system (Dual Head Symbia TruePoint SPECT/CT, Siemens Healthcare GmbH, Erlangen, Germany) was a combined apparatus, the registration between the CT and SPECT imaging data itself is provided in the digital imaging and communications in medicine tags of the 3-D images. This allows for the connection between the SPECT imaging data and the CT coordinate system provided by the transformation matrix TSPECTCT.


Freehand Single-Photon Emission Computed Tomography

Multiple fhSPECT scans were acquired using either an SOE 311 gamma probe with a Europrobe 3 control unit (Eurorad S.A., Eckbolsheim, France) or a HiSens gamma probe with a, SG03 control unit (Crystal Photonics GmbH, Berlin, Germany). A four-fiducial gamma probe reference target (GPRT) was fitted to allow tracking of these modalities by the navigation system. Scanning times varied between 2 and 3 min. Placement of the PRT was the same as for SPECT/CT-based navigation. Using both the PRT and the GPRT during the fhSPECT acquisition, the 3-D fhSPECT scanning data can be placed in the navigation workflow by linking it to the PRT via the 4×4 transformation matrix TfhSPECTPRT, provided by the navigation system itself.


Fluorescence Laparoscope

During the navigation experiments, both the standard laparoscope white-light setting and the fluorescence light setting were used. To connect the laparoscope video feed to the navigation system, an Epiphan frame grabber (DVI2PCIe, Epiphan Systems Inc., Ottawa, Ontario, Canada) was integrated into the navigation cart. Either the laparoscope processed video feed was connected directly from the digital visual interface (DVI) output at the back of the surgical robot vision cart or the laparoscope raw video feed was connected from the component output on the back of the camera console. To track the 3-D pose of the laparoscope with the navigation system, a three-fiducial reference target was attached to the camera housing [laparoscope reference target (LRT)] [see Fig. 1(b)]. The optical wavelengths used for object tracking and fluorescence emission partly overlapped, being both in the 800- to 900-nm range. Nevertheless, interference issues were not expected during intra-abdominal use of the laparoscope.

An adapted version of the declipseSPECT 6.0 (SurgicEye GmbH) software was used to incorporate the laparoscope video feed in both calibration and navigation workflows. Two steps of calibrations were performed with the fluorescence laparoscope to allow proper use in the navigation workflow:

  • 1. Calibration of intrinsic and extrinsic camera-system parameters using a “checkerboard” calibration and the Open Source Computer Vision (OpenCV) camera calibration library.24 The intrinsic camera parameters were the focal length and principal point while the extrinsic camera parameters described the translation and rotation with respect to the LRT coordinate system represented by the transformation matrix TCAMERALRT [see Figs. 1(b) and 1(c)]. The calibration of these camera parameters was performed to enable an augmented reality projection of the processed pre- or intraoperative imaging data on top of the fluorescence laparoscope video feed; thus providing the augmented projection of navigation targets (with SPECT) and anatomical reference (with CT) in both the white-light and fluorescence light camera options.

  • 2. Calibration of the 3-D pose transformation from the coordinate system of the laparoscope tip to that of the LRT was defined as transformation matrix TLTIPLRT [see Fig. 1(d)]. The laparoscope tip calibration was performed with a cylindrical construction (a calibrator), tracked with a four-fiducial calibrator reference target (CRT). Because of the known geometry of the calibrator, it was possible to determine the relation between the CRT and the center of the tip of the laparoscope, represented by the transformation matrix TLTIPCRT. During the navigated procedure, this tip calibration allows the navigation system to display the relative distance from the laparoscope tip to one of the selected navigation targets.


Soft Tissue Phantom

The soft tissue phantom setup [Fig. 2(a)] consisted of a tray with the PRT placed on a stand and a silicone half sphere with four Eppendorf tubes (Eppendorf AG, Hamburg, Germany) placed inside. The silicone half sphere structure, with a diameter of 15  cm at its base and a height of 5  cm, was cast from Dragon Skin® FX-Pro silicone rubber (Smooth-On Inc., Macungie, Pennsylvania), with addition of Thi-Vex® Silicone Thickener (Smooth-On Inc.) for the skin part and Slacker® Tactile Mutator (Smooth-On Inc.) for the softer fat part. SilTone pigments (FormX, Amsterdam, The Netherlands) were used to color the rubber.

Fig. 2

Navigation of the fluorescence laparoscope based on preoperative SPECT/CT imaging and intraoperative fluorescence guidance as performed on the tissue-mimicking phantom setup. (a) Volume rendering of the fused SPECT/CT scan of the tissue-mimicking phantom setup. The SPECT hotspots are shown in purple/blue. (b–f) Augmented reality video feed from the perspective of the fluorescence laparoscope as seen on the navigation device, when navigating to the navigation targets. These targets are indicated as gray and green circles and are defined by the SPECT signal (displayed purple/blue). The distance shown is the distance from the tip of the camera to the selected (green) target. The CT overlay is shown in white/gray. (b) The phantom shown by white-light modus, without augmented SPECT/CT overlay. (An error states that the mobile camera is not tracked due to the LRT being just at the border of the tracking volume.) (c) White-light modus with augmented SPECT/CT overlay. (d and e) Fluorescence light modus with augmented SPECT/CT overlay. In this mode, the background video is shown in gray. (f) Fluorescence light modus without augmented SPECT/CT overlay. Fluorescence is displayed in green.


Two 260-μL Eppendorf tubes served as the navigation targets, to be scanned with (fh)SPECT imaging, and were placed at roughly 5 cm apart. With a volume of 260  μL, the Eppendorf tubes should resemble a typical lymph node size found in lymph node dissection for prostate cancer; KleinJan et al.25 reported a range of 80 to 860  μL, with a median of 170  μL. These tubes were filled with a mixture of ICG and Tc99m:253  μL of an ICG solution and 7  μL of a Tc99m solution (5  MBq). The ICG solutions used, in this study, consisted of 31.25  μg/mL ICG (ICG-Pulsion, Pulsion Medical Systems, Munich, Germany) dissolved in human serum albumin (Albuman 200  g/l, Sanquin, Amsterdam, The Netherlands). With this as stock solution, the effective ICG solutions used in the phantom experiments range from 9.62 to 31.03  μg/mL. This quantity was chosen based on both clinical relevance (0 to 18.10  μg/mL typically found in lymph nodes during prostatectomy25) and best visibility during fluorescence guidance (best visibility found between 2.44 and 78.27  μg/mL26). The Tc99m solutions used, in this study, were obtained from a Tc99m-sodium pertechnetate solution (750  MBq/mL saline, Technekow, Mallinckrodt Medical BV, Petten, The Netherlands). Two additional 500-μL Eppendorf tubes were hidden on the sides of the tissue mimicking structure to serve as “anatomical reference” in the CT images. These tubes were filled with CT contrast agent [Ultravist 300  mg/mL (Bayer AG, Leverkusen, Germany) 50% diluted with demineralized water].

To constrain movement of all the objects in the phantom setup, and therefore constrain the navigation error, the silicone structure and the PRT stand were glued to a silicone bottom layer in the tray. The PRT itself was taped to the stand. Note however, to allow easy access to the target sources, the precut phantoms were opened with a wound spreader during navigation, a manipulation that could have possibly introduced a (minor) deformation error to the navigation setup.


Laparoscopic Torso Phantom

The human-like laparoscopic torso phantom was made out of a standard life-sized anatomical model of the human skeleton, generally used for educational purposes. A transparent plastic mannequin was split in two parts over the coronal plane and the skeleton was cut to fit inside. The skeleton was fixed after which a silicone model of the prostate and three lymph nodes were incorporated, thereby simulating the SN biopsy procedure during robot-assisted laparoscopic prostatectomy.10,11,27 Both the prostate model and the lymph node models were cast from Dragon Skin® FX-Pro silicone rubber and Slacker® Tactile Mutator using the color pigments for different colors. Mixtures of ICG and Tc99m were then inserted into the prostate and SN models. For the prostate model, a 1.5-mL Eppendorf tube was incorporated in the cast and was used to store a 260-μL tube filled with 80  μL of the ICG solution and 180  μL of the Tc99m solution (200  MBq). The lymph node phantoms contained a 260-μL tube with a mixture of 258  μL of the ICG solution and 1.8  μL of the Tc99m solution (2  MBq). This resulted in a 1001 ratio in radioactivity for the prostate model with respect to each of the individual lymph nodes, which should be a reasonable ratio in SN mapping.2829.30.31

Four 12-mm trocars were placed in the transparent shell of the torso phantom to allow docking of the da Vinci robot, including the fluorescence laparoscope. A PRT was fixed at the sternum location of the phantom.


Evaluation Navigation Accuracy

The total navigational accuracy was evaluated in the soft tissue phantom setups and divided in both coronal and sagittal errors. The laparoscope was placed in a stand vertically with respect to the navigation target at a distance of roughly 5 cm. The navigation accuracy determination was then performed manually by comparing the target distance as given by the navigation system to the distance found with a ruler (measurement precision of 1  mm), in a manner similar to that described by Brouwer et al.16 For the sagittal plane, the distance from laparoscope tip to target was indicated by the navigation system in numbers (precision of 1 mm). The coronal distance was indicated with the augmented reality overlay over the laparoscopic video feed, consisting of a green target point and a scalable purple/blue cloud of activity around it. The green target point provided a clear reference for the accuracy determination and was compared to the actual target as seen on the video. Since the navigation targets consisted of small Eppendorf tubes, the midpoint of the fluid was used for the measurements. For each navigation setup (SPECT/CT and fhSPECT), three different soft tissue phantoms were used with each their own scans, each containing two distinct targets (n=6 over three phantoms). Each individual measurement was performed by two different observers, resulting in 24 measurements in total. IBM SPSS statistics 22 software (International Business Machines Corp., New York, USA) was used to compare if the accuracy of SPECT/CT- and fhSPECT-based navigation was significantly different using an unpaired t-test (95% confidence interval).


Overview of Coordinate Frames and Calibration Steps

An overview of the different coordinate transformations applied for the navigation workflow is provided in Fig. 1. All transformations are divided in three different colors (red, blue, and green) to distinguish how they are found. The transformations determined by the OTS in real time are shown in red, the ones calculated during registration or calibration procedures are shown in blue, and all transformations deduced from known geometries are shown in green.



Figure 2 shows an example of the navigation process performed using preoperative SPECT/CT; Figs. 2(b) to 2(f) show snapshots of the Firefly video feed combining the laparoscopic view (in white-light and fluorescence modus) and an augmented overlay of the SPECT/CT data. The navigation targets, displayed as dots, are defined by the signal hotspots found in the SPECT imaging data and the two white Eppendorf-shaped signals in the CT view [Figs. 2(a), 2(c), and 2(d)] function as anatomical markers. In a similar manner, we could also use “intraoperative” fhSPECT data sets for navigation (see Fig. 3).

Fig. 3

Navigation of the fluorescence laparoscope based on intraoperative fhSPECT imaging and intraoperative fluorescence guidance as performed on the tissue-mimicking phantom setup. (a) Acquisition of the fhSPECT scan with a tracked gamma probe from the perspective of the overhead navigation system camera. (b) Acquired fhSPECT scan (shown in purple/blue) displayed as an augmented reality projection over the phantom setup. The distance shown is the distance from the tip of the laparoscope to the selected (green) navigation target, defined by the fhSPECT signal hotspots. (A red sign in the upper left corner indicates that the gamma probe is no longer in the tracking volume, since it is no longer in use after the scan acquisition.) (c) Video feed from the perspective of the fluorescence laparoscope in fluorescence light modus, with augmented fhSPECT overlay. (d) Video feed from the perspective of the fluorescence laparoscope in fluorescence light modus, without augmented fhSPECT overlay.


In the laparoscopic navigation view, the CT and/or (fh)SPECT overlay could be turned on and off independent of each other and the size of the SPECT hotspots (threshold on the SPECT signal) could be scaled according to preference. Qualitatively, navigation to the different targets (depicted as green colored dots) in the soft tissue phantoms appeared accurate using both scan modalities; during the navigation process, the augmented hotspots in the video feed and the distance to the targets seemed well registered to the fluorescent findings. As mentioned in Sec. 2.4, optical wavelengths of the OTS and fluorescence emission overlapped. However, other than in previous studies,21 the NIR light of the OTS did not interfere with the fluorescence detection of the Firefly.

The navigation accuracy for the SPECT/CT- and fhSPECT-based navigation procedures was quantified (see Table 1). Average errors for the SPECT/CT-based navigation were 2.25 mm (median 2 mm) and 2.08 mm (median 2 mm) in the coronal and sagittal plane, respectively. For the fhSPECT-based navigation, these were quite similar: 1.92 mm (median 2 mm) and 2.83 mm (median 3 mm). These results underline that the navigation accuracy, in the setup studied, stays well below the 1-cm tissue limit needed for successful fluorescence detection.8,9 Comparison of the errors found for both scan modalities did not result in a significant difference (p values of 0.764 and 0.282 for the coronal and sagittal plane, respectively).

Table 1

Overview of the accuracy measurements in the tissue-mimicking phantom setups for navigation based on both SPECT/CT and fhSPECT.

Navigation based onAnatomical planeMean (mm)Standard deviation (mm)Range (mm)
SPECT/CTCoronal2.251.860 to 5
Sagittal2.081.020.5 to 3.5
fhSPECTCoronal1.921.880 to 4.5
Sagittal2.831.251 to 4

Evaluation of the navigation setup using a torso phantom (Fig. 4) illustrates the clinical setup to which the navigation should be translated. In addition, it demonstrates the setup remained effective when the robot is fully docked.

Fig. 4

Evaluation of the combined fluorescence navigation system based on preoperative SPECT/CT. (a) Operation room setup with the navigation system, fluorescence laparoscope, the surgical robot, and the torso phantom. On both the torso phantom and the fluorescence laparoscope, a reference target is attached. (b) Volume rendering of the fused SPECT/CT scan of the torso phantom setup. The SPECT hotspots are shown in pink/blue. The arrows indicate the prostate (P) and 3 lymph nodes (LN). (c–f) Augmented reality video feed from the perspective of the fluorescence laparoscope as seen on the navigation device when navigating to the navigation targets. These targets are indicated as gray and green circles and are defined by the SPECT signal (displayed purple/blue). The distance shown is the distance from the tip of the laparoscope to the selected (green) target. (c) White-light modus; letters indicating the specific lymph nodes (LN 1 and LN 2) and the prostate (P) in the anatomy of the phantom (d) White-light modus with augmented SPECT overlay. (e) Fluorescence light modus with augmented SPECT overlay. (f) Fluorescence light modus without the augmented SPECT overlay. Lymph node LN 1 is located too close to the highly active prostate P (1100 ratio in radioactivity) to resolve it as a separate navigation target by the used rendering.




We have integrated surgical navigation, made possible via optical tracking, with (robot-assisted) laparoscopic fluorescence-guided surgery. Hereby, a robot-docked fluorescence laparoscope could be successfully navigated to target structures located in two different phantom setups with an accuracy of 2  mm. All different imaging modalities used (fluorescence, SPECT/CT, and fhSPECT) could be available during navigation, thereby allowing compensation for the weaknesses of the individual modalities (e.g., the limited tissue penetration of fluorescent signals or the inaccuracies of surgical navigation due to soft tissue deformations). The presented soft tissue navigation setup suggests a next step in providing surgeons with more precise orientation and localization during robot-assisted procedures.

Theoretically, a preoperatively acquired SPECT/CT scan can be considered more precise than an intraoperatively acquired fhSPECT; with SPECT/CT, a lot more information is collected for the 3-D image reconstruction due to larger gamma cameras and longer scan times.18 Therefore, for a rigid phantom setup, one would expect the navigation errors found when navigation was based on SPECT/CT to be smaller than when navigation was based on fhSPECT. In our measurements, performed on the semirigid soft tissue phantoms, navigational accuracy was similar for both the two scan methods. This suggests that navigation based on fhSPECT may serve as a valuable and cost-effective alternative for navigation based on SPECT/CT. Moreover, due to the intraoperative nature of fhSPECT imaging (e.g., using a drop-in gamma probe31,32), the technique should suffer much less from tissue deformations that result from patient movement. In fact, due to the short acquisition time (roughly between 2 and 3 min), one could easily acquire a new scan after any expected tissue deformation or displacement (e.g., removal of the primary tumor site). In addition, intraoperative real-time visualization of the fluorescence can be used to compensate for navigational errors below 1  cm.

Translation of the proposed navigation setup to clinical use should be straightforward, as the robotic setup remains identical. We do see two questions that require attention, depending on the specific clinical application chosen: (1) Will NIR optical tracking be sufficient for a dynamic robotic intervention? (2) What will be the range of tissue deformation found in patients and how does this relate to the fluorescence detection limit?

Regarding the first point, in this study, optical tracking was used to define the 3-D pose of the phantom setup, laparoscope, and gamma probe using three different reference targets containing at least three fiducials each. As shown, this was feasible, but a general limiting factor was that the OTS has to maintain a direct line-of-sight to at least three of the reference target fiducials to allow for 3-D pose determination of the tracked object.23 A feature that limits the OR layout and logistics. This may be overcome by increasing the number of fiducials per reference target or by using multiple OTS cameras on different sides of the OR.33 A possible drawback of using NIR optical tracking in combination with NIR fluorescence imaging is the overlap in spectra used for object tracking and fluorescence imaging. In previous work, we found that this could be a considerable issue when using an open surgery fluorescence camera.21 To our surprise, we did not detect such an extensive interference when the Firefly laparoscope was applied outside of the phantom body. This finding could provide a basis for solving the issues previously reported. This said, in a laparoscopic setting, the tracking light is not likely to interfere with the fluorescence detection, since fluorescence imaging is conducted inside the patient’s body.

Alternative to NIR optical tracking, electromagnetic (EM) tracking or mechanical tracking could also provide outcome. With an intrinsic accuracy up to 0.5  mm (RMS) for EM tracking versus 0.25  mm (RMS) for optical tracking,22,34 EM tracking still seems quite promising. However, a major disadvantage with this technique is the susceptibility to distortions of the EM tracking field by nearby metal objects or EM interferences, rendering it much less accurate.35 Although inaccurate, mechanical tracking of the robotic arms is possible by using forward kinematics to calculate the 3-D pose of the end effector such as the laparoscope or the surgical instrument.36 Fuerst et al.32 greatly improve this accuracy to an approximate error of 0.2 mm by combining the mechanical tracking with visual-based tracking via the stereoscopic laparoscope video feed.

In the presented navigation workflow, the navigation targets are displayed as a two-dimensional augmented reality overlay superimposed in the laparoscopic video feed while the distance from the laparoscope tip to target is shown in numbers. When implemented in clinical use, it might be hard to get a good idea of how deep the target structures really lie. Therefore, the augmented reality visualization may in the future be enhanced by dynamic augmented cues (e.g., small arrows indicating both direction and distance) or more depth perception cues (e.g., as shown by Bichlmeier et al.37 and Kutter et al.38).



In a phantom setup, both pre- and intraoperative SPECT data sets can be used to accurately navigate the da Vinci robot-integrated fluorescence laparoscope toward surgical targets. Hereby, a basis is generated for future robot-assisted navigation toward fluorescent lesions.


This work was partially supported by two grants from the European Research Council under the European Union’s Seventh Framework Program (FP7/2007-2013, Grant Nos. 2012-306890 and 2012-323105), a Eurostars grant (Hybrid Navigator; Grant No. E! 7555), and an NWO-STW-VIDI grant (Grant No. STW BGT11272). Material support for this work was provided by SurgicEye (SurgicEye GmbH, Munich, Germany), allowing the authors to run their prototype software on the declipseSPECT navigation system, and by Intuitive Surgical (Intuitive Surgical Inc., Sunnyvale, California), supplying the authors with a stand-alone Firefly fluorescence laparoscope system for initial laboratory experiments. The authors would also like to thank Petra Dibbets-Schneider from the section Nuclear Medicine at the Department of Radiology, Leiden University Medical Center (Leiden, the Netherlands), for assisting in acquisition of the SPECT/CT scans.


1. E. E. Medlin, D. M. Kushner and L. Barroilhet, “Robotic surgery for early stage cervical cancer: evolution and current trends,” J. Surg. Oncol. 112(7), 772–781 (2015).JSONAU0022-4790 http://dx.doi.org/10.1002/jso.24008 Google Scholar

2. R. Montalti et al., “Outcomes of robotic vs laparoscopic hepatectomy: a systematic review and meta-analysis,” World J. Gastroenterol. 21(27), 8441–8451 (2015). http://dx.doi.org/10.3748/wjg.v21.i27.8441 Google Scholar

3. R. Biffi et al., “Dealing with robot-assisted surgery for rectal cancer: current status and perspectives,” World J. Gastroenterol. 22(2), 546–556 (2016). http://dx.doi.org/10.3748/wjg.v22.i2.546 Google Scholar

4. G. Novara et al., “Systematic review and meta-analysis of studies reporting oncologic outcome after robot-assisted radical prostatectomy,” Eur. Urol. 62(3), 382–404 (2012).EUURAV0302-2838 http://dx.doi.org/10.1016/j.eururo.2012.05.047 Google Scholar

5. N. S. van den Berg et al., “Sentinel lymph node biopsy for prostate cancer: a hybrid approach,” J. Nucl. Med. 54(4), 493–496 (2013).JNMEAQ0161-5505 http://dx.doi.org/10.2967/jnumed.112.113746 Google Scholar

6. T. Maurer et al., “Prostate-specific membrane antigen-radioguided surgery for metastatic lymph nodes in prostate cancer,” Eur. Urol. 68, 530–534 (2015).EUURAV0302-2838 http://dx.doi.org/10.1016/j.eururo.2015.04.034 Google Scholar

7. F. Greco et al., “Current perspectives in the use of molecular imaging to target surgical treatments for genitourinary cancers,” Eur. Urol. 65(5), 947–964 (2014).EUURAV0302-2838 http://dx.doi.org/10.1016/j.eururo.2013.07.033 Google Scholar

8. P. T. Chin et al., “Multispectral visualization of surgical safety-margins using fluorescent marker seeds,” Am. J. Nucl. Med. Mol. Imaging 2(2), 151–162 (2012). Google Scholar

9. F. W. van Leeuwen, J. C. Hardwick and A. R. van Erkel, “Luminescence-based imaging approaches in the field of interventional molecular imaging,” Radiology 276(1), 12–29 (2015).RADLAX0033-8419 http://dx.doi.org/10.1148/radiol.2015132698 Google Scholar

10. G. H. KleinJan et al., “Optimisation of fluorescence guidance during robot-assisted laparoscopic sentinel node biopsy for prostate cancer,” Eur. Urol. 66(6), 991–998 (2014).EUURAV0302-2838 http://dx.doi.org/10.1016/j.eururo.2014.07.014 Google Scholar

11. G. H. KleinJan et al., “Multimodal hybrid imaging agents for sentinel node mapping as a means to (re)connect nuclear medicine to advances made in robot-assisted surgery,” Eur. J. Nucl. Med. Mol. Imaging 43(7), 1278–1287 (2016). http://dx.doi.org/10.1007/s00259-015-3292-2 Google Scholar

12. M. Baumhauer et al., “Navigation in endoscopic soft tissue surgery: perspectives and limitations,” J. Endourol. 22(4), 751–766 (2008). http://dx.doi.org/10.1089/end.2007.9827 Google Scholar

13. J. Rassweiler et al., “Surgical navigation in urology: European perspective,” Curr. Opin. Urol. 24(1), 81–97 (2014).CUOUEQ0963-0643 http://dx.doi.org/10.1097/MOU.0000000000000014 Google Scholar

14. F. Nickel et al., “Navigation system for minimally invasive esophagectomy: experimental study in a porcine model,” Surg. Endosc. 27(10), 3663–3670 (2013). http://dx.doi.org/10.1007/s00464-013-2941-4 Google Scholar

15. N. C. Buchs et al., “Augmented environments for the targeting of hepatic lesions during image-guided robotic liver surgery,” J. Surg. Res. 184(2), 825–831 (2013).JSGRA20022-4804 http://dx.doi.org/10.1016/j.jss.2013.04.032 Google Scholar

16. O. R. Brouwer et al., “Feasibility of intraoperative navigation to the sentinel node in the groin using preoperatively acquired single photon emission computerized tomography data: transferring functional imaging to the operating room,” J. Urol. 192(6), 1810–1816 (2014). http://dx.doi.org/10.1016/j.juro.2014.03.127 Google Scholar

17. T. Wendler et al., “First demonstration of 3-D lymphatic mapping in breast cancer using freehand SPECT,” Eur. J. Nucl. Med. Mol. Imaging 37(8), 1452–1461 (2010). http://dx.doi.org/10.1007/s00259-010-1430-4 Google Scholar

18. T. Engelen et al., “The next evolution in radioguided surgery: breast cancer related sentinel node localization using a freehandSPECT-mobile gamma camera combination,” Am. J. Nucl. Med. Mol. Imaging 5(3), 233–245 (2015). Google Scholar

19. C. Bluemel et al., “Intraoperative 3-D imaging improves sentinel lymph node biopsy in oral cancer,” Eur. J. Nucl. Med. Mol. Imaging 41(12), 2257–2264 (2014). http://dx.doi.org/10.1007/s00259-014-2870-z Google Scholar

20. O. R. Brouwer et al., “Image navigation as a means to expand the boundaries of fluorescence-guided surgery,” Phys. Med. Biol. 57(10), 3123–3136 (2012).PHMBA70031-9155 http://dx.doi.org/10.1088/0031-9155/57/10/3123 Google Scholar

21. G. H. KleinJan, “Towards (hybrid) navigation of a fluorescence camera in an open surgery setting,” J. Nucl. Med. (2016). http://dx.doi.org/10.2967/jnumed.115.171645 Google Scholar

22. Northern Digital Inc., “Northern digital, polaris optical tracking systems,” 2016,  http://www.ndigital.com/medical/products/polaris-family/ (5 August 2016). Google Scholar

23. P. Waelkens et al., “Surgical navigation: an overview of the state-of-the-art clinical applications,” in Radioguided Surgery, , K. Herrmann, O. E. Nieweg and S. P. Povoski, Eds., pp. 57–73, Springer (2016). Google Scholar

24. Z. Zhang, “A flexible new technique for camera calibration,” IEEE Trans. Pattern Anal. Mach. Intell. 22(11), 1330–1334 (2000).ITPIDJ0162-8828 http://dx.doi.org/10.1109/34.888718 Google Scholar

25. G. H. KleinJan et al., “Fluorescence guided surgery and tracer-dose, fact or fiction?” Eur. J. Nucl. Med. Mol. Imaging 43(10), 1857–1867 (2016). http://dx.doi.org/10.1007/s00259-016-3372-y Google Scholar

26. N. S. van den Berg et al., “(Near-infrared) fluorescence-guided surgery under ambient light conditions: a next step to embedment of the technology in clinical routine,” Ann. Surg. Oncol. 23(8), 2586–2595 (2016). http://dx.doi.org/10.1245/s10434-016-5186-3 Google Scholar

27. H. G. van der Poel et al., “Intraoperative laparoscopic fluorescence guidance to the sentinel lymph node in prostate cancer patients: clinical proof of concept of an integrated functional imaging approach using a multimodal tracer,” Eur. Urol. 60(4), 826–833 (2011).EUURAV0302-2838 http://dx.doi.org/10.1016/j.eururo.2011.03.024 Google Scholar

28. L. Bergqvist, S. E. Strand and B. R. Persson, “Particle sizing and biokinetics of interstitial lymphoscintigraphic agents,” Sem. Nucl. Med. 13(1), 9–19 (1983).SMNMAB0001-2998 http://dx.doi.org/10.1016/S0001-2998(83)80031-2 Google Scholar

29. B. A. Kapteijn et al., “Validation of gamma probe detection of the sentinel node in melanoma,” J. Nucl. Med. 38(3), 362–366 (1997).JNMEAQ0161-5505 Google Scholar

30. L. Jansen et al., “Improved staging of breast cancer through lymphatic mapping and sentinel node biopsy,” Eur. J. Surg. Oncol. 24(5), 445–446 (1998). http://dx.doi.org/10.1016/S0748-7983(98)92496-9 Google Scholar

31. M. N. van Oosterom et al., “Revolutionizing (robot-assisted) laparoscopic gamma tracing using a drop-in gamma probe technology,” Am. J. Nucl. Med. Mol. Imaging 6(1), 1 (2016). Google Scholar

32. B. Fuerst et al., “First robotic SPECT for minimally invasive sentinel lymph node mapping,” IEEE Trans. Med. Imaging 35(3), 830–838 (2015). http://dx.doi.org/10.1109/TMI.2015.2498125 Google Scholar

33. T. Sielhorst et al., “Online estimation of the target registration error for n-ocular optical tracking systems,” in Medical Image Computing and Computer-Assisted Intervention (MICCAI 2007), pp. 652–659, Springer (2007). Google Scholar

34. Northern Digital Inc.,, “Northern digital, aurora electromagnetic tracking systems,”  http://www.ndigital.com/medical/products/aurora/ (5 August 2016). Google Scholar

35. H. Kenngott et al., “Magnetic tracking in the operation room using the da Vinci® telemanipulator is feasible,” J. Rob. Surg. 7(1), 59–64 (2013). http://dx.doi.org/10.1007/s11701-012-0347-2 Google Scholar

36. A. Reiter, P. K. Allen and T. Zhao, “Appearance learning for 3D tracking of robotic surgical tools,” Int. J. Rob. Res. 33(2), 342–356 (2014).IJRREL0278-3649 http://dx.doi.org/10.1177/0278364913507796 Google Scholar

37. C. Bichlmeier et al., “Contextual anatomic mimesis hybrid in-situ visualization method for improving multi-sensory depth perception in medical augmented reality,” in 6th IEEE and ACM Int. Symp. on Mixed and Augmented Reality (ISMAR 2007), pp. 129–138 (2007). http://dx.doi.org/10.1109/ISMAR.2007.4538837 Google Scholar

38. O. Kutter et al., “Real-time volume rendering for high quality visualization in augmented reality,” in Int. Workshop on Augmented Environments for Medical Imaging Including Augmented Reality in Computer-Aided Surgery (AMI-ARCS) (2008). Google Scholar

Biographies for the authors are not available.

© The Authors. Published by SPIE under a Creative Commons Attribution 3.0 Unported License. Distribution or reproduction of this work in whole or in part requires full attribution of the original publication, including its DOI.
Matthias Nathanaël van Oosterom, Matthias Nathanaël van Oosterom, Myrthe Adriana Engelen, Myrthe Adriana Engelen, Nynke Sjoerdtje van den Berg, Nynke Sjoerdtje van den Berg, Gijs Hendrik KleinJan, Gijs Hendrik KleinJan, Henk Gerrit van der Poel, Henk Gerrit van der Poel, Thomas Wendler, Thomas Wendler, Cornelis Jan Hadde van de Velde, Cornelis Jan Hadde van de Velde, Nassir Navab, Nassir Navab, Fijs Willem Bernhard van Leeuwen, Fijs Willem Bernhard van Leeuwen, } "Navigation of a robot-integrated fluorescence laparoscope in preoperative SPECT/CT and intraoperative freehand SPECT imaging data: a phantom study," Journal of Biomedical Optics 21(8), 086008 (17 August 2016). https://doi.org/10.1117/1.JBO.21.8.086008 . Submission:

Back to Top