2.1.

Materials

The key component of the proposed projective telementoring and surgical navigation system is a CPI-1 coaxial projective imaging module, which is capable of acquiring the surgical scene images and displaying the annotations simultaneously at the site of surgery. The CPI-1 module consists of a MV-GED130C-T color charge-coupled diode (CCD) camera (MindVision, Shenzhen, China), a PRO4500VIS700 projector (Wintech, Shanghai, China), a 532-nm NF-25C05-40-532 notch filter (OptoSigma Corporation, Japan), and a 530 nm band-pass filter (Zhaojiu Photoelectric, Shanghai, China), as well as a beam splitter (Nano Macro Photonics, Shenzhen, China), as shown in Fig. 2(a). In our previous design of the projective imager, the CCD camera and the projector are placed side by side, inducing the mismatched fields of view that need to be calibrated at different working distances.¹⁸ To eliminate such coregistration error between the acquired and the projected images, we propose a coaxial optical path design as shown in Fig. 2(b). In this design, the surgical scene image is reflected by the beam splitter and acquired by the CCD camera, and the wirelessly transferred annotations are transmitted through the beam splitter and projected onto the same surgical scene with accurate coregistration regardless of any changes of the working distance. To avoid interference of the projected annotations with the surgical scene images acquired by the CCD camera, a 530-nm band-pass filter and a 532-nm notch filter are placed on the projector arm and the camera arm of the coaxial optical path, respectively. With this design, the projected annotations at the surgical site will not affect the surgical scene images acquired by the CCD camera. The CCD camera has a chip sensor with the size of 1/3″, a dynamic range of 12 bits, a frame rate of about 30 frames per second, and a resolution of $1280\times 960$. A prime lens with a focal length of 12 mm is mounted on the camera in order to reach an FOV of $220\mathrm{mm}\times 160\mathrm{mm}$ for surgical navigation in a working distance of 300 mm. The 532-nm notch filter in front of the CCD camera is fixed on a flip mount that is freely accessible for a 0 deg to 90 deg range, enabling us to switch the notch filter on or off for different tasks of image acquisition or color correction. The projector perpendicular to the camera projects instructive annotations or other images to the surgical field at a contrast ratio of 1000:1 and a resolution of $1280\times 800$. The 530-nm band-pass filter is mounted in front of the projector. The beam splitter with the transmission and reflectance ratio of 80:20 is mounted at an angle of 45 deg to the optical axis of the camera. This transmission to reflectance ratio is determined as a trade-off for optical intensity requirements between projection and imaging. The position and the orientation of the projector are aligned carefully for coincident projection and imaging. A TeamViewer software package (TeamViewer GmbH, Göppingen, Germany) that is compliant with the Health Insurance Portability and Accountability Act¹⁹ is used for wireless data transmission between the local and the remote sites. At the local site, a computer is used for image acquisition, projection, and processing tasks. At the remote site, a portable display device is used for surgical field display and instructive annotation.

Fig. 2

Telementoring and surgical navigation system at the local site. (a) System setup. (b) Schematic diagram of the CPI-1 module.

2.2.

Color Correction Algorithm

To facilitate effective decision-making and instructive annotation by the surgical specialist at the remote site, the true color display of the surgical scene is preferred. However, due to the use of the 532-nm notch filter on the camera arm of the CPI-1 module, the surgical field images acquired by the CCD camera lack the color component at 532 nm. In other words, the green channel is much weaker than the other two. Therefore, color correction is required in order to recover the true color of the surgical scene.

Our color correction algorithm is applied to an RGB color image using a transformation matrix. The conventional method of color correction is to scale red, green, and blue channels.²⁰ Since this method only considers the effects of a single channel, it induces a certain error in color recovery of chromatic images. To overcome this limitation, we modify the conventional color correction algorithm by inducing multiple color components as

The green channel is compensated by introducing a color correction card that has 104 different colors and four circular markers on the corners. First, we acquire one color card image that contains four circular markers in its FOV with the notch filter and another image without the filter. The two images are automatically coregistered according to the four markers. Second, the color blocks at the same position in two images are automatically cropped. Finally, the transformation matrix $\mathbf{T}$ is obtained by Eq. (1), where $\mathbf{R},\mathbf{G},\mathbf{B}$ are the mean color values of each color block cropped from the image captured without the notch filter, and ${\mathbf{R}}_{\mathbf{1}}{\mathbf{G}}_{\mathbf{1}}{\mathbf{B}}_{\mathbf{1}}$ are the mean color values of each color block cropped from the image with the notch filter. Python is used for programming the above image processing tasks.

The correction matrix $\mathbf{T}$ obtained from the above color correction process is used to recover colors for the video streams acquired by the CCD camera with the notch filter. The computer at the local site is responsible for acquiring the color images of the surgical scene, correcting the weak green channel, and transferring the corrected images to the remote site. In the meanwhile, the computer at the remote site receives and displays the true-color surgical scene images after color correction, where the experienced surgeons can make surgical annotations. The annotations are transferred back to the local site through TeamViewer software and projected to the surgical site for guided surgical intervention

2.3.

Animal Experiment

Telementored cricothyrotomy in a rabbit model is used as a proof of concept to demonstrate the clinical utility of tele-guided surgical interventions. Cricothyrotomy is a procedure aimed at creating an airway by incision for a certain life-threatening situation with low complications.²¹ Telementored tracheal intubation may add another dimension to airway management by avoiding multiple attempts at intubation and preventing iatrogenic injuries in the emergency setting. In addition, the proposed model of telementored cricothyrotomy also provides a valuable teaching tool for medical students to learn clinical anatomy during surgery.²²

A rabbit of 6- to 7-week-old with a weight of about 200 g is used for the animal validation tests. The test protocol is approved by the Institutional Animal Care and Use Committee of University of Science and Technology of China. The animal is anesthetized by intraperitoneal injection of urethane (20% concentration). After anesthesia, the rabbit is fixed on the operating table, with its four limbs stretched out and secured by the medical tapes. During our experiments, the rabbit is monitored every 5 min in case our experiment may produce extra pain. The CPI-1 module is installed 300 mm above the operating table for image acquisition and annotation projection at the surgical site. At the remote site, an experienced surgical specialist watches the wirelessly transferred surgical scene image and makes the instructive annotations for remote guidance of cricothyrotomy. The annotations are transferred back to the local surgical site and projected onto the surgical field to guide a less experienced surgeon for appropriate incision.

3.1.

Accuracy of the Color Correction Algorithm

The accuracy of the color correction algorithm is evaluated by calculating the Euclidean distance of the color differences in our projective telementoring system. The Euclidean distance is the distance between two colors within a color space such as RGB, Luv, and Lab. The International Commission on Illumination (CIE) chooses the distance metric $\mathrm{\Delta }{E}_{ab}^{*}$ to denote the color difference in the $L^{*}{a}^{*}{b}^{*}$ space and the most recent evaluation standard is $\mathrm{\Delta }E2000$, which resolves the perceptual uniformity issue.^23,24

We use the evaluation standard $\mathrm{\Delta }E2000$ to test the accuracy of our color correction algorithm in both a color card and a hand. Figure 3 shows the color correction results of the color card. Figures 3(a)–3(c) show the images of the color card captured with the notch filter, without the notch filter, and with the notch filter but after color correction, respectively. The color difference $\mathrm{\Delta }E$ between Figs. 3(b) and 3(c) is calculated by the CIDE2000 color difference formulas²⁵ and plotted in Fig. 3(d) as a histogram. According to the evaluation criteria of CIDE2000, a $\mathrm{\Delta }E$ level between 1 and 2 implies that only experienced observer can notice the color difference. The averaged $\mathrm{\Delta }E$ of the color card is 1.58, indicating that the color card image shows no significant difference from the original image after color correction.

Fig. 3

The color correction result of the color card. (a) Image with the 532-nm notch filter. (b) Image without the 532-nm notch filter. (c) Image with the notch filter but after color correction. (d) Histogram of the $\mathrm{\Delta }E$ (CIEDE2000) between the image (b) and the image (c) after color correction. Mean value of the $\mathrm{\Delta }E$ is 1.58.

Similarly, we also test the color correction outcome for the hand. Figures 4(a)–4(c) show the images of the hand captured with the notch filter, without the notch filter, and with the notch filter but after color correction, respectively. Figure 4(d) shows the histogram of the color difference $\mathrm{\Delta }E$ between Figs. 4(b) and 4(c). The averaged $\mathrm{\Delta }E$ of the hand is 1.37, indicating that the hand image shows no significant difference from the original image after color correction.

Fig. 4

The color correction result of a hand. (a) Image with the 532-nm notch filter. (b) Image without the 532-nm notch filter. (c) Image with the notch filter but after color correction. (d) Histogram of the $\mathrm{\Delta }E$ (CIEDE2000) between the image (b) and the image (c) after color correction. Mean value of the $\mathrm{\Delta }E$ is 1.37.

3.2.

Color Correction for the Remote Expert

The color of the image seen by a remote expert through a PC or a tablet is affected by the following factors: (1) color correction accuracy for the images acquired at the local surgical site, (2) compression and decompression of the image data, and (3) color display accuracy by the remote expert through the PC or tablet. Our color correction algorithm described above has calibrated the first factor to ensure true color display at the local surgical scene. To calibrate the second factor, we transfer the image of a 24-color card between two cities in China (Hefei and Changchun) through TeamViewer’s quality priority mode at different bandwidths (i.e., 1.15, 9.98, and 21.84 Mbps).²⁶ The average values of $\mathrm{\Delta }E$ (CIEDE2000) are all less than 1 at three network bandwidths, indicating no observable difference at different network conditions.²⁴ Further study shows that compression and decompression of image data by a commonly used telecommunication software package such as TeamViewer will not induce significant variations in image color as long as the network bandwidth between two places is greater than 1.15 Mbps. In terms of the third factor, the remote display should be color calibrated using an International Color Consortium (ICC) color correction file or a monitor color correction instrument. We choose to present the color calibrated terminal screen to the clinicians in order to ensure that no color distortion is added on the remote display. The color display of the terminal screen (AOC LV253WQP) is first calibrated using a Spyder 4Express display color correction instrument and a DisplayCAL color correction software package. After calibration, the color evaluation is performed using a DisplayCAL color correction software package. The averaged $\mathrm{\Delta }E$ (CIEDE2000) level between the standard and the calibrated 51 colors is 0.46, indicating negligible color difference at the remote terminal site.

3.3.

Accuracy of the Telementoring and Surgical Navigation System

The achievable accuracy of the proposed telementoring system is verified in two benchtop experiments as shown in Fig. 5. In the first experiment, a cardboard of $175\mathrm{mm}\times 135\mathrm{mm}$ with a $9\times 9$ matrix of white circles printed on the black background [Fig. 5(a)] is imaged and projected back onto the target cardboard. Both the original and the projected images of the white circles are acquired subsequently by the same camera. We have tested the projection accuracy of the target cardboard at different working distances (i.e., 300 to 500 mm) and at different angles (i.e., 0 deg, 30 deg, and 60 deg) in order to simulate the biological tissue of different locations and curvatures. According to Fig. 5(b), the maximal projection error (i.e., the distance shift between the original and the projected images) is less than 0.7 mm for the entire FOV and for the working distance ranging from 300 to 500 mm. According to Fig. 5(c), the maximal projection error (i.e., the distance shift between the original and the projected images) is less than 2 mm for the entire FOV and for the working angle ranging from 0 deg to 60 deg.

Fig. 5

The evaluation results for projection accuracy. (a) The target cardboard with a matrix of white circles. (b) Results of the central distance shift between the actual circle and the projected circle at various working distances (i.e., 300, 400, and 500 mm). (c) Results of the central distance shift between the actual circle and the projected circle at various working angles (i.e., 0 deg, 30 deg, and 60 deg). (d) Results of four rectangular markers placed by volunteers to record four corner positions of the projected rectangle. (e) Maximal positional offsets for the rectangular markers placed by the volunteers.

In the second experiment, rectangles of different sizes are generated by the computer at the remote site, transferred back, and projected by the CPI-1 module within a surgical field of $220\mathrm{mm}\times 160\mathrm{mm}$. Volunteers are asked to place four rectangular markers at four corners of the projected image by aligning two edges of the marker with the corresponding edges of the projected rectangle, as shown in Fig. 5(d). The image of the projected rectangle and the image of four aligned markers are acquired by the CPI-1 module, and the coordinates of four corner points are detected by the Harries–Stephens algorithm.²⁷ Positional deviations between the projected image and the marker image are calculated at each of the four corners and plotted in Fig. 5(e). According to Fig. 5(e), the image-guided operation by the proposed telementoring and surgical navigation system may induce a positional error of less than 1 mm in both $X$ and $Y$ directions.

3.4.

In Vivo Demonstration of Telementored Surgery

Clinical utility of the proposed telementoring system is demonstrated in vivo using cricothyrotomy in a rabbit model as a proof of concept. Figures 6(a) and 6(d) are pictures captured at the remote site and the local surgical site, respectively. Figures 6(b), 6(e), and 6(f) demonstrate the guidance for the location of the initial incision. Figure 6(b) shows the main annotation of the surgical incision drawn by the surgical specialist on the remote desktop. Figure 6(e) refers to the surgical instructive annotation superimposed into the actual surgical field. Figure 6(f) is the result of the incision. Figures 6(c), 6(g), and 6(h) show the simulated surgical procedure of tracheal intubation during cricothyrotomy. Figure 6(c) refers to the incision annotation of the tracheal intubation position drawn by a surgical specialist on the remote screen. In the meanwhile, this incision annotation is superimposed into the surgical field as shown in Fig. 6(g) to provide localized guidance for cricothyrotomy. Figure 6(h) shows the experimental result of telementored cricothyrotomy. In this demonstration, an operator without any knowledge of clinical anatomy is able to perform the cricothyrotomy procedure correctly following the remote guidance of an experienced surgical specialist.

Fig. 6

Cricothyrotomy surgery on the rabbit. (a) Overall view of the remote guidance of the surgical scene. (d) Overall view of the surgical field. The red arrow in (b) refers to the surgical incision on the remote screen, in (e) refers to the instructive annotation superimposed into the surgical field, and in (f) refers to the result of the surgical incision, respectively. The red arrow in (c) refers to the position of the endotracheal intubation on the remote screen, in (g) refers to the instructive annotation superimposed into the surgical field, and in (h) refers to the result of the cricothyrotomy, respectively.

In addition to tele-guided surgery, the projective imaging system can also be used as a telementoring tool for medical education. As shown in Fig. 7(a), the experienced surgical specialist remotely makes the surgical annotations and draws the primary anatomical structures on the touch screen that displays the real-time surgical scene. Figures 7(b) and 7(c) show the actual projection window (i.e., the surgical field) before and after superimposition of the instructive annotations remotely provided by the surgical specialist. The seamless coregistration between the annotations and the surgical scene enables interactive teaching of clinical anatomy and surgical skills without any distance barrier.

Fig. 7

Interactive instruction of clinical anatomy by telementored surgery in a rabbit model. (a) Touch screen display of the surgical scene and the surgical specialist’s annotations at the remote site. (b) The actual surgical field before projection of annotations. (c) The actual surgical field superimposed with instructive annotations provided remotely by the surgical specialist. Red arrows refer to the instructive annotations.