Optical surgical instrument tracking system based on the principle of stereo vision

Abstract. Optical tracking systems are widely adopted in surgical navigation. An optical tracking system is designed based on the principle of stereo vision with high-precision and low cost. This system uses optical infrared LEDs that are installed on the surgical instrument as markers and a near-infrared filter is added in front of the Bumblebee2 stereo camera lens to eliminate the interference of ambient light. The algorithm based on the region growing method is designed and used for the marker’s pixel coordinates’ extraction. In this algorithm, the singular points are eliminated and the gray centroid method is applied to solve the pixel coordinate of the marker’s center. Then, the marker’s matching algorithm is adopted and three-dimensional coordinates’ reconstruction is applied to derive the coordinates of the surgical instrument tip in the world coordinate system. In the simulation, the stability, accuracy, rotation tests, and the tracking angle and area range were carried out for a typical surgical instrument and the miniature surgical instrument. The simulation results show that the proposed optical tracking system has high accuracy and stability. It can meet the requirements of surgical navigation.


Introduction
An image-guided surgical navigation system breaks through the limits of the traditional surgical operation and extends the doctors' limited vision scope. 1,2 Tracking and positioning are the key techniques of an image-guided surgical navigation system. The accuracy of the tracking method directly determines the accuracy of the image-guided surgical navigation system. An optical tracking system is designed, using light-emitting markers, combined with self-developed image processing methods and an easy-to-implement three-dimensional (3-D) space tracking algorithm, which achieves the high tracking accuracy of surgical instruments.
The tracking methods most widely adopted in image-guided surgical navigation systems include articulated arms, active or passive optical tracking systems, sonic digitizers, and electromagnetic sensors. [2][3][4][5] Nowadays, the most promising tracking methods are optical and electromagnetic tracking. However, compared with optical tracking, electromagnetic navigation has less precision and its magnetic field is prone to distortion by large metal objects which bring measurement errors to the system. 3,6 The commercial optical tracking systems currently used in surgical navigation products mainly include micron tracker (Claron Technology Inc., Canada), 7 polaris optical tracking systems (Northern Digital Inc., Canada), 8 StealthStation system (Medtronic Inc., United States), 9 etc. Unfortunately, these products are always expensive and they are not convenient for further development and customization due to proprietary techniques.
Based upon stereotactic surgery principles, this paper proposes an optical tracking system with low cost and high precision. In the proposed optical tracking system, three optical LEDs are used as the markers that are installed on the surgical instrument. This system uses the Bumblebee2 binocular camera to capture the images of markers. To eliminate the interference of ambient light, a near-infrared filter is added in front of the camera lens. To extract the pixel coordinate of a marker's center, the region growing (RG) method is used to eliminate the singular point and then the gray centroid (GC) method is adopted to obtain the pixel coordinate of the marker's center. This image processing method greatly enhances the antijamming capability and improves the accuracy of the system. Next, the algorithm matches the markers in the left and right camera images based on the sequence of its spatial positions. As a last step, it derives the coordinates of the surgical instrument tip by 3-D coordinates reconstruction of markers and realizes the tracking system. Through simulation, the accuracy and stability of system has been verified and can meet the needs of surgical navigation.
surgical instruments' calibration that are denoted within the dotted line. This helps the main process obtain the parameters for calculating the coordinates of the markers and the surgical instrument's tip. For the main process, the stereo camera first captures the binocular images of the surgical instrument.
Next, the markers' extraction, markers' matching, distortion correction, and 3-D coordinates' reconstruction procedures are applied to calculate the world coordinates of markers. Finally, the world coordinates of the surgical instrument's tip can be deduced by the world coordinates of the markers.  1 Framework of optical tracking system. The solid line represents the main process and the dotted line represents auxiliary processes that provide several parameters before the main process.

Camera Calibration
Zhang's calibration method 10 is widely used in camera calibration in recent years and is mature. It is a compromise method for traditional camera calibration and self-calibration. This method is robust and it is not necessary to develop a high-precision positioning reference. More importantly, as the calibration accuracy can meet the requirements, Zhang's method is used to calibrate cameras. The specific calibration tool is the camera calibration toolbox. 11 A 5 × 8 chessboard is designed as the calibration board; as shown in Fig. 2, the size of each square is 30 × 30 mm 2 .

Binocular Image Capture
A Bumblebee2 stereo camera 12 is the binocular vision acquisition device adopted in this system. As shown in Fig. 3, two near-infrared filters are installed in front of the camera lens to remove the interference of ambient light.
According to the practical requirements, 2 two kinds of surgical instruments are designed and employed in the proposed system. They are typical surgical instrument (TSI) and miniature surgical instrument (MSI). TSI is suitable for most cases, while the MSI is suitable for surgeries with limited operation space. In the simulation, these two kinds of surgical instruments are tested separately. For each surgical instrument, three 850-nm near-infrared LEDs are installed and used as the markers, as shown in Fig. 4.
The luminance of the three markers is captured by the Bumblebee2 camera and the background is completely filtered out by the near-infrared filter. The visual effects of the captured image with and without the near-infrared filter are shown in Fig. 5. From Fig. 5, it can be seen that the ambient light has been effectively filtered out.

Markers Extraction
In an optical tracking system, the pixel coordinate of the marker's center affects the accuracy of the 3-D space coordinates and the coordinate of the surgical instrument tip directly. In the captured images, the markers are the highlighted regions. The pixel coordinate of the marker's center is the center of the highlighted region and they are used to calculate the world coordinate of surgical instrument tip. So, an effective algorithm is proposed to extract the pixel coordinate of marker's center from the highlighted region. This algorithm is based on RG and GC methods as shown in Fig. 6.
First, the RG method is used to seek all highlighted connected regions in the image. In this step, a threshold is used to determine the seed and region growth criterion. Its value is set to half of the maximum pixel in the proposed experience. In order to avoid duplication of the seed search, a flag matrix is adopted. Next, the highlighted connected region is compared with the range of the marker's area (MAR) to determine whether it is noise or the marker. The noise of the CCD sensor is inevitable. 13 Noise is usually scattered and is shown as small highlighted regions whose areas are much smaller than the marker in the captured image. Through the pre-experiment, the MAR is set as: [11,74] if the area of a highlighted region is less than 11, it is considered to be a noise. Finally, the GC method is applied to calculate the pixel coordinate of the marker's center. The equation for the GC method is E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 0 1 ; 3 2 6 ; 1 3 0 (1)   where ðu i ; v i Þ is the i'th pixel coordinate in the markers' area, whose gray value is fðu i ; v i Þ, and ðu; vÞ is the derived pixel coordinate of the marker's center. The procedures of the markers' extraction algorithm are as follows: 1. Read the captured marker image.
2. Initialize the flag matrix to 0, whose size is the same as the marker image.
3. Select a pixel p ij from the marker image in raster scan order. If the flag of p ij is 0 and the value of p ij is greater than the threshold, p ij is set as the seed s k . Otherwise, go to step (3) and select the next pixel.
4. Use the RG method to find the highlighted connected region around the seed s k . If the values of pixels are larger than the threshold which is the same as that in step (3), the flags of those pixels are set to 1 and those pixels are defined as the highlighted connected region. (4), if the number of pixels in the highlighted connected region is within the MAR, this region is considered to be a marker. The GC method is then adopted to calculate the pixel coordinate of the marker's center. The pixel coordinate of the marker's center is denoted as ðu; vÞ. Otherwise, go to step (3) until all of the pixels are selected.

Markers Matching
The proposed optical tracking system is based on a binocular camera. According to the model for 3-D coordinates reconstruction of markers (Sec.

Three-Dimensional Coordinates Reconstruction of Markers
The mathematical model 14,15 of the camera is as follows, which establishes the relationship between the pixel coordinate of the marker's center and its 3-D spatial world coordinates (2)  where ðu; vÞ is the pixel coordinate of the marker's center, Z C are the Z coordinates of markers in the camera coordinate system, and ðX W ; Y W ; Z W Þ are the coordinates of markers in the world coordinate system (WCS), respectively. I and E are the internal and the external parameters of the camera that can be obtained by camera calibration. Two equations of Eq. (2) can be established for the binocular stereo camera, as shown in Eq. (3). The world coordinates of markers can be derived from E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 0 3 ; 6 3 ; 5 1 7

Distortion Correction
The pinhole imaging mathematical model used in this paper is an ideal linear mathematical model. In the process of manufacturing and installation, camera lens distortion is inevitable. 16 In order to eliminate the influence of lens distortion on the positioning accuracy, the lens distortion must be corrected.
According to the camera model based on Heikkilä and Silvén, 17 the distortion model adopted in this paper is as E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 0 4 ; 6 3 ; 2 8 9 where ðu 0 ; v 0 Þ, ðu; vÞ are the actual and the ideal pixel coordinates of a marker's center, r 2 ¼ u 2 þ v 2 . k 1 , k 2 , and k 3 are the radial distortion coefficients, and p 1 , p 2 are the centrifugal distortion coefficients.
To verify the effect of the camera lens distortion correction module in the proposed optical tracking system, an experiment is carried out. Three kinds of distances, which are short, medium, and long, are measured by the proposed tracing system. For each kind of distance, the endpoints' coordinates are measured using the distortion correction model and the nondistortion correction model. Each distance is measured 50 times and the average value is calculated. The actual distance is measured by using a Vernier caliper. The distances measured by the tracking system are compared with the actual distances. The error is the difference between them as shown in Table 1.
From Table 1, it can be seen that the distortion correction errors are smaller than the nondistortion correction error. With the increasing of distance, the distortion correction error becomes smaller.

Surgical Instrument Calibration
The purpose of optical tracking is to obtain the world coordinates of the surgical instrument's tip. Through 3-D reconstruction, the world coordinates of the markers can be obtained. According to the shape of a surgical instrument, a surgical instrument coordinate system (SICS) can be established, as shown in Fig. 7.
The coordinates of the surgical instrument tip in SICS are P tip s ðX tip s ; Y tip s ; Z tip s Þ, and P tip w ðX tip w ; Y tip w ; Z tip w Þ in WCS. The rotation matrix and the translation matrix of SICS with respect to WCS are R and T, then E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 0 5 ; 3 2 6 ; 4 3 2 R and T can be calculated by markers' coordinates m 1 , m 2 , and m 3 (Ref. 18) E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 0 6 ; 3 2 6 ; 3 7 8 E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 0 7 ; 3 2 6 ; 3 3 1 where the coordinates of o are numerically equal to T. The purpose of the surgical instrument calibration is to get the coordinates of P tip s . Equations (5)-(7) are used to calculate the world coordinates of the surgical instrument tip by markers' world coordinates.
The rotating surgical instrument method 16 is used to get P tip s . The idea is to fix the position of the surgical instrument tip and rotate the surgical instrument. Thus, the same marker on The world coordinates of the tip are denoted as ðX tip w ; Y tip w ; Z tip w Þ. Under the rigid body constraint, the distance between P tip w and P ij is ρ E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 0 8 ; 6 3 ; Eq. (8) Eq. (9) has i × ðj − 1Þ equations, which can be expressed in a matrix form E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 1 0 ; 6 3 ; 4 8 3 WP tip w ¼ F: Among them, W is an i × ðj − 1Þ row by three columns of the matrix and F is the i × ðj − 1Þ vectors. When j ≥ 2, Eq. (10) is an over determined equation, and the least square method can be used to solve P tip . In this paper, the calibration is carried out by 20 rotating surgical instruments. The i'th rotation matrix and the translation matrix of the SISC relative to the WCS are R i and T i , and the tip coordinate can be calculate by E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 1 1 ; 6 3 ; 3 6 3 P tip s ¼ R −1 i ðP tip w − T i Þ:

Simulation Results
In order to verify and determine the stability and accuracy of the system, the stability, space distance, and rotation tests were carried out for TSI and MSI. The stability experiment is designed to measure the system stability under the condition of repeated measurement. The accuracy of the system is measured by a space distance experiment. The absolute error is compared by the results of calculated distance between two points and a Vernier caliper. Furthermore, the stability and accuracy of the tip coordinates are tested by a rotation experiment at different angles.

Calibration Settings
Twenty pairs of the left and right images of the calibration board are captured at different locations as shown in Fig. 8. The left camera's optical center is used as the origin of the WCS. In this system, E left is a fixed value, as shown

Stability Test
When the optical tracking system is working, the system must have sufficient stability. If the surgical instrument remains still, the calculated tip coordinates may change due to the changes in the capture environment and the stability of the algorithm. In order to test the stability of the system, 10 different position points are selected randomly and the surgical instrument is kept still in each position. For each position, 100 sets of images are used for repeating. Then, the standard deviations (SD) of X-, Y-, and Z-directions are calculated, respectively, which are used to evaluate the stability of the system. Two kinds of surgical instruments are tested in the experiment. The results are shown in Table 2. Figure 9 shows another representation of Table 2 for MSI. From the above results, it can be seen that the average SDs of the X-, Y-, and Z-directions in 10 positions are 0.01799, 0.00783, 0.08112 mm for TSI and 0.01289, 0.00942, 0.06751 mm for MSI, respectively. In Table 2, the worst results are 0.09708 and 0.09452 mm for two different instruments.
From Fig. 8, it also can be seen that the range of the X coordinate is about (45, 75 mm), the range of the Y coordinate is about (150, 153 mm), and the range of the Z coordinate is about (330, 390 mm). As a result, it looks like that the SD of the Z-direction is bigger than the other two directions, but the coefficient of variations (CVs) of the three directions from Fig. 9 are almost the same range.

Space Distance Test
Since the absolute coordinates obtained by the optical tracking system are in the WCS, it is difficult to directly evaluate the accuracy of the world coordinates. Therefore, in order to test the position accuracy of the simulated surgical instruments, this system adopts the method of indirect measurement. Within the scope of the effective field of the optical tracking system, the tip coordinates in the two places are randomly chosen and the distances are measured. Then, the tracking accuracy of the system is evaluated by comparing the distance with the actual measured distance using a Vernier caliper and grating ruler.  . 9 The 3-D coordinate stability results for MSI. (a) Results of the X -direction measurement, the range of X -direction SD is from 0.01003 to 0.01677, and the range of X -direction CV is from 0.012% to 0.068%; (b) results of the Y -direction measurement, the range of Y -direction SD is from 0.00787 to 0.01148, and the range of Y -direction CV is from 0.011% to 0.016%; and (c) results of the Z -direction measurement, the range of Z -direction SD is from 0.05100 to 0.09452, and the range of Z -direction CV is from 0.014% to 0.027%.
Journal of Biomedical Optics 065005-7 June 2017 • Vol. 22 (6) Ten groups of different distances were measured, respectively, by Vernier caliper and grating ruler that includes short distance, medium distance, and long distance, and each tip coordinate is measured 50 times. The average absolute errors using the Vernier caliper are shown in Figs. 10 and 11. The average absolute errors of two surgical instruments measured by Vernier caliper are 0.23 and 0.13 mm.
However, the accuracy results of the proposed optical tracking system have more than four decimal places. To compare with the results of the Vernier caliper, the last several decimal places have been rounded. In order to evaluate the accuracy of the proposed optical tracking system more accurately, we use a more accurate measurement device, a grating ruler. The brand of grating ruler is SION, and the model is KA-300/5U/ 1020 mm.
The "grating ruler" is rigidly fixed on the "precision optical table" to ensure the stability, and then the TSI or MSI is also fixed to the grating ruler, as shown in Figs. 12 and 13. The results measured by Vernier caliper are used as a reference, and the new distance measurement values are determined by using the grating ruler, as shown in Tables 3 and 4, and Fig. 14. From  Tables 3 and 4, it can be seen that higher measurement accuracy is obtained by using the grating ruler. The fluctuation range of the distance root mean square error (RMSE) becomes smaller as shown in Fig. 14.
The average absolute distance error is 0.23 mm for TSI and 0.13 mm for MSI by Vernier caliper, 0.041 mm for TSI and 0.024 mm for MSI by grating ruler, respectively. The distance RMSE is 0.32 mm for TSI and 0.09 mm for MSI by Vernier caliper, 0.049 mm for TSI and 0.032 mm for MSI by grating ruler, respectively. The results show that the proposed optical tracking system has a higher accuracy and it can meet the accuracy requirement of space tracking equipment in surgical navigation.

Rotation Test
In the rotational experiment, the stability of the rotation of the surgical instrument and the angular accuracy of the rotation angle are tested.
When the camera captures the marker images from different angles, the size of the marker in the image is constantly changing. In order to verify that the calculated coordinates are accurate for any capturing angle, a rotating stability experiment is designed. Because the size of the MSI is too small to do the experiment, only the TSI is done for this test. As shown in   Table 5. The SDs of the coordinates of the TSI tip in different rotation angles are 0.08074, 0.28266, and 0.33407 mm in the X-, Y-, and      Z-directions, respectively. From the above data, it can be seen that the coordinates of the tip do not change significantly, which shows that the proposed tracking algorithm is accurate enough to guarantee the coordinates of the TSI tip to be unchanged under different capturing angle conditions. The proposed tracking system has a high rotational stability.
In order to get a more accurate rotation angle, TSI or MSI is fixed on the 6-DOF mechanical arm. The brand of the 6-DOF mechanical arm is LOBOT, and the model is MK12352. Twelve angle groups are measured from 5 deg to 60 deg. Every angle is measured 100 times by TSI or MSI, and then averaged. The rotation angle is measured as indicated by the red arrow in Fig. 16 and the results are shown in Table 6. From Table 6, it can be seen that the absolute error of the average angle of 12 groups is 0.36 deg for TSI and 0.28 deg for MSI, respectively.

Tracking Angle and Area Range Test
The tracking angle view and area range of the proposed optical tracking system are shown in Figs. 17 and 18, respectively.
The angle view range is limited by the instrument on which there are only three markers. Rotation around the Z-axis ranges from −78.3383 deg to 74.6440 deg for TSI, and from −67.9872 deg to 53.9503 deg for MSI, which is the red area in Fig. 17. Rotation around the X-axis ranges from −86.2429 deg to 88.1640 deg for TSI, and from −87.9051 deg to 89.0361 deg for MSI, which is the green area in Fig. 17. Rotation around the Y-axis ranges from −74.2184 deg to 75.04111 deg for TSI, and from −57.4553 deg to 73.4661 deg for MSI, which is the blue area in Fig. 17.
The measurement of height and width depends on the distance in the Z-direction. In the Z-direction (from near to far), the common distance ranges from 219.0297 to 2577.6776 mm for TSI, from 198.5920 to 612.5765 mm for MSI, respectively. The occasionally farthest distance that can be measured is 3022.9473 mm for TSI and 2954.3603 mm for MSI, which is shown as the blue point in Fig. 18. The dashed line space is more precise than the solid line space, which can be measured with more instrument poses.  Through the system experiment, the measurable range of the proposed optical tracking system is sufficient to meet the scope of surgical navigation.

Cost Estimation
Polaris Spectra and Polaris Vicra system are popular commercial surgical tracking system, from Northern Digital Inc. (NDI) company. The prices are different for different types of the system, e. g., systems with active or passive markers. The cheapest is more than $31,340, the most expensive one is $52,496. The optical positioning system proposed in this paper mainly includes a 1394 card, binocular camera, LED marker, infrared filter, wires, proposed positioning software system, and so on. The cost estimation of the proposed optical tracking system is $3,366 as shown in Table 7.
Taking into account the ratio between cost and commercialization, if the ratio is 3, the total price is $10,000. It substantially reduces the price compared with the existing commercial tracking systems.   The note * in Table means it is an estimation of the wires used in our system. It depends in practical applications.
Journal of Biomedical Optics 065005-12 June 2017 • Vol. 22 (6) In this paper, an optical tracking system based on a Bumblebee2 binocular camera is designed. An 850-nm near-infrared optical LED is used as the marker of surgical instruments. Meanwhile, an infrared filter is installed in front of the camera lens, which can effectively eliminate the interference of ambient light. In the space location algorithm, the RG method is designed to find the highlighted connected region of markers in the image. In order to eliminate the singular point, the pixel coordinate of the marker's center is calculated by the GC method. With an optical tracking system, it is difficult to match the markers in a traditional way. A matching method for the 3-D reconstruction based on the position order of the markers is proposed, and the results are satisfying. In the end, the rotating surgical instrument method is used to calibrate the surgical instrument to obtain the position relationship between the tip and the SCS, resulting in tracking of the surgical instrument tip.
In the stability experiment, 10 different positions are measured and calculated for the SDs of tip coordinates for TSI and MSI in the X-, Y-, and Z-directions. The worst results of the two kinds of surgical instruments are 0.09708 and 0.09452 mm, respectively. The average value of SDs are 0.01799, 0.00783, 0.08112 mm in the X-, Y-, Z-directions for TSI, and 0.01289, 0.00942, 0.06751 mm for MSI. For the X-and Y-directions, it can be seen that the SD is very small. The SD is slightly larger in the Z-direction because the coordinate is also much larger than in the other two directions. From the CV results of the three directions, the same conclusion can be made. In the space distance experiment, the distance RMSE is 0.32 mm for TSI and 0.09 mm for MSI by Vernier caliper, 0.049 mm for TSI and 0.032 mm for MSI by grating ruler, respectively. The accuracy of Polaris Spectra and Polaris Vicra system of NDI company can achieve 0.25 mm RMS in pyramid. 8 In the proposed optical tracking system, the measurable distance range is from 219.0297 to 2577.6776 mm for TSI, from 198.5920 to 612.5765 mm for MSI in Z-direction (from near to far). The occasionally farthest distance that can be measured is 3022.9473 mm for TSI and 2954.3603 mm for MSI, respectively. In the existing systems, the measurable distance ranges from 950 to 3000 mm for Polaris Spectra system, from 557 to 1336 mm for Polaris Vicra system in Z-direction. Because the surgical instruments are welded by us and are subject to a limitation because the luminescent marker is oriented in a single direction, the measurable area range of TSI and MSI is not as large as that of the Polaris Spectra system and Polaris Vicra system. However, the nearest measurable locations for TSI and MSI are better than Polaris Spectra and Polaris Vicra systems. It is advantageous for surgical operations that commonly have a small operating view. Through the system experiment, it can be seen that the proposed optical tracking system and the corresponding space tracking algorithm have high stability and accuracy. Moreover, comparing with the cost of the proposed optical tracking system and the existing systems, the price has been greatly reduced.

Conclusions
In this paper, a low cost, open optical tracking system based on binocular vision theory is proposed. The system uses the stereo camera and combines with the self-developed image processing methods and easy-to-implement 3-D space tracking algorithm, which accomplishes the high accuracy of surgical instruments tracking. Experimental results show that the system has high stability and accuracy, and it can provide accurate guidance for surgical navigation.
The system and the related algorithm are a meaningful exploration in the field of surgical navigation. Considering the practical application, it has room for improvement. For example, a battery can be used to supply power for the marker, so as to avoid the wire connected to the surgical instruments. The surgical instruments without wire can move more freely. In the future, we will design the power supply circuit for the surgical instruments to install the button batteries. The fast calibration algorithm needs further study. The algorithm of pixel coordinates extraction and the marker matching can also be optimized at the same time, so that the tracking accuracy of the system can be improved. The real-time synchronization needs to be further optimized to achieve higher system accuracy. At present, with more and more accurate operation requirements, the accuracy and stability of the optical tracking system needs continuous improvement.

Disclosures
No conflicts of interest, financial or otherwise are declared by the authors.