## 1.

## Introduction

Endoscopy has been widely used as a minimally invasive diagnostic medical procedure.^{1, 2} Conventional endoscopes with a wide-angle lens can provide physicians with a wide view in the long distance and a magnified view in the short distance. However, since the appearance of an object changes by its distance from the lens, the exact size or location is not obvious only from an endoscopic image. When physicians are required to measure or record the size of the object, they have to thrust a small ruler through a forceps channel in the endoscope and take an internal picture while keeping the ruler within view. This procedure, however, is time consuming even for an experienced endoscopist, and the resulting data may lack precision. Since they can only obtain a rough approximation of the distance, it is difficult for them to judge, for example, whether the size of a tumor is increasing or decreasing. Thus, those methods that can produce an objective 3-D measurement have long been expected.

Recently, novel technologies such as virtual reality and augmented reality are applied to endoscopy and produce 3-D or stereoscopic images using 3-D data from MRI or CT scans.^{3, 4, 5} Regular endoscopes, however, still play a main role in visual inspection and minimally invasive surgery, and 3-D display or measurement that uses only endoscopes is also widely studied. For example, a binocular stereoscopic approach is proposed for increasing the depth and spatial perception of a monocular video endoscope.^{6} 3-D images are also obtained by analyzing the reflection of multicolored light that is projected onto tissue.^{7} There have been proposals to project a laser beam to generate a grid on an object and measure the three dimensions by tracing manually the outline of an object and beam spots,^{8} or to project a striped pattern of light for the 3-D measurements on the stripes.^{9} In this work, we propose an endoscope system that projects multiple laser beams, detects the laser spots automatically, and can display rulers on the endoscopic images, so that physicians can measure the size and position of an object in real time.

## 2.

## Methods for Three-Dimensional Measurement

## 2.1.

### System Outline

Our endoscope contains four laser beams, which are directed via optical fibers and set parallel to the optical axis of the camera (Fig. 1 ). The laser beams are projected from the tip of the endoscope onto an object under inspection, and their spots are automatically detected by image processing techniques. Using the 2-D coordinates of those spots, we can calculate the corresponding 3-D coordinate by the triangulation, and can display virtual rulers on the endoscope monitor. At the same time, we can correct barrel distortion associated by the wide angle lens of the endoscope. The procedural steps are as follows.

## 2.2.

### Barrel Distortion Correction

The wide-angle lenses used in regular endoscopes are susceptible to barrel distortions, which can disturb the physicians’ visual perception of shape and size of lesions or suspicious regions.

To remove the distortion from observed images we have to relate the distance of a pixel from the optical axis in an observed image to the corresponding distance in an undistorted image. Once the distortion function is determined, we can use it for all the source images, since the distortion function that gives the correspondences is unique to the particular lens. We observe a test chart shown in Fig. 2a with the endoscope and determine the distortion function by measuring the distances of several points on the distorted image and those on the test chart. Figure 3 shows the resulting distortion function, Fig. 2a shows the source image, and 2b the corrected image.

## 2.3.

### Laser Spot Detection

To measure the 3-D coordinates of laser spots, the system locates those points in an image using a template matching method.^{10} Since the physical configurations of the camera and the laser beams are *a priori* given, the search areas for the laser spots are limited, and it is not necessary to search the whole image. Since the laser beams of our system are set to be parallel to the optical axis of the camera lens, the search area for the leaser spots are restricted on those lines that are depicted as white lines in Fig. 4
.

To determine the location of the laser spots, we compare the density values of each pixel, which are transformed from $8\text{-}\text{bit}$ RGB values. To judge the similarity to the laser spot, we measure the density distribution and its deviation in the neighborhood of each pixel. Those pixels that have maximum similarity and larger deviation than a predetermined threshold are identified as the laser spots. The similarity is defined by the correlation between an input image and a Gaussian distribution, since we regard the density profile of the laser spot as Gaussian.

As is shown in our experiments, endoscopic images may have reflection highlights. To prevent the highlights from disturbing the laser spot detection, we regard such pixels that have high values of G or B elements of RGB as highlights and neglect them from the laser spot detection.

## 2.4.

### Calculation of Three-Dimensional Coordinates

Figure 5 illustrates the calculation of the 3-D coordinates of the laser spots. Let the lens optical center $O$ be the origin of the 3-D space, the camera optical axis be the $z$ axis, $L({X}_{L},{Y}_{L},{Z}_{L})$ denote the laser beam source, $P({X}_{p},{Y}_{p},{Z}_{p})$ denote the laser spot on an object, and $p({x}_{p},{y}_{p},f)$ denote the laser spot projected on the image plan, where $f$ is the focal length of the lens. Here, for ease of explanation, the image plane is drawn in-between the lens and the object, though it actually is located on the opposite side of the object from the lens.

The 2-D coordinates $({x}_{p},{y}_{p})$ of the projection on the image plane of the laser spot $({X}_{p},{Y}_{p},{Z}_{p})$ can be calculated after the spot is detected by the method mentioned in Sec. 2.3. Here, the line connecting the origin $O$ and the laser spot $P({X}_{p},{Y}_{p},{Z}_{p})$ is given by

and the line connecting the laser beam source $L({X}_{L},{Y}_{L},{Z}_{L})$ and the laser spot $P({X}_{p},{Y}_{p},{Z}_{p})$ is given bywhere $({B}_{X},{B}_{Y},{B}_{Z})$ is the direction vector of the laser beam, which is determined when the system is designed. Even if we do not know the value of the direction vector, we can obtain it empirically by measuring the beam spot on a screen that is set at a known distance. In this case, the direction vector can be determined from the position of the spot $({X}_{s},{Y}_{s},{Z}_{s})$ :Then, the laser spot $P({X}_{p},{Y}_{p},{Z}_{p})$ is determined as the intersection of Eqs. 1, 2:## 4.

Then the 3-D coordinates $({X}_{p},{Y}_{p},{Z}_{p})$ can be determined from Eq. 4 using Eqs. 5, 6:

## 7.

## 2.5.

### Display of Rulers

Once the 3-D position of the laser spot is measured by the previous method, we can estimate the size of objects near the spot. The system can display rulers virtually on the plane parallel to the image plane, and whose center is at the laser spot (Fig. 5). To depict the divisions of the scale according to its 3-D position, we should calculate the length of the interval of those divisions. Let $u$ be the physical length of the interval, and $u$ is set to $0.5\phantom{\rule{0.3em}{0ex}}\mathrm{mm}$ in the following examples. Then, we should calculate the length in pixels on the image plane that corresponds to $u$ .

The length of a line segment on the plane $z={Z}_{p}$ that passes the laser spot and is parallel to the image plane is reduced by the ratio of $f\u2215{Z}_{p}$ . The proper interval of the scale divisions is $uf\u2215{Z}_{p}$ . Even if the focal length $f$ is unknown, the interval of the divisions can be determined from

## Eq. 8

$$\frac{f}{{Z}_{p}}=\frac{{x}_{p}}{{X}_{p}}=\frac{{y}_{p}}{{Y}_{p}}={\left(\frac{{x}_{p}^{2}+{y}_{p}^{2}}{{X}_{p}^{2}+{Y}_{p}^{2}}\right)}^{1\u22152},$$In the same way, if the physical length on the image plane is unknown, the intervals of the divisions can be represented in pixels. In this case, the reduction ratio changes by the direction of the ruler to display. Suppose we should display it in the direction of $\theta $ in the real space. The interval of the divisions should be $uf\phantom{\rule{0.2em}{0ex}}\mathrm{cos}\phantom{\rule{0.2em}{0ex}}\theta \u2215{Z}_{p}{d}_{x}$ pixels in the $x$ direction and be $uf\phantom{\rule{0.2em}{0ex}}\mathrm{sin}\phantom{\rule{0.2em}{0ex}}\theta \u2215{Z}_{p}{d}_{y}$ pixels in the $y$ direction. Even if $f\u2215{d}_{x}$ or $f\u2215{d}_{y}$ is unknown, we only have to display the divisions by the ratio of the number of pixels to the laser spot on the image to the number of pixels to the spot from the lens optical axis.

In 1Video 1 , four laser beams are projected on the gastric wall of a rat, and the distances to the four spots are measured. Four rulers are also displayed with those divisions whose intervals are calculated according to their distances. The rulers can be displayed in one, two, or four directions, or can be concentric circles.

10.1117/1.2800758.1## 2.6.

### Rulers in-between the Laser Beam Spots

When the system places a ruler parallel to the image plane, it marks the divisions repeatedly only with the same interval, since all the divisions of the scale should be equally placed both in the real space and on the image plane. However, a ruler that connects two laser spots is not always parallel to the image plane (Fig. 6 ). Though the divisions are equally placed in the real space, it does not hold for the virtual rulers on the image plane. We have to calculate all the positions of each division.

Suppose that we mark the divisions ${M}_{k}$ $(k=0,1,2,\dots )$ with the interval of $u$ , starting at a laser spot ${P}_{s}({X}_{s},{Y}_{s},{Z}_{s})$ and ending at another laser spot ${P}_{e}({X}_{e},{Y}_{e},{Z}_{e})$ . Then, $({X}_{k},{Y}_{k},{Z}_{k})$ , the 3-D position of the $k$ ’th division ${M}_{k}$ can be calculated as:

## 9.

In Eq. 10, the physical length is used in determining the positions of the divisions of the scale. We can also determine those positions if we have to use pixels as the unit. By observing an object with known displacement, as mentioned before, Eq. 5 gives

## 12.

## 13.

Thus we can depict oblique rulers between the spots. Figure 7 shows an example of displaying the rulers in-between the four laser spots.

## 3.

## Experiments

We have modified a commercially available endoscope Fujinon EC-450WM5, which is used for the lower gastro-intestinal (GI) tract, with a $140\text{-}\mathrm{deg}$ view field and $12.8\text{-}\mathrm{mm}$ distal end diameter. The diameter of the optical fibers, via which the four laser beams are directed, is $120\phantom{\rule{0.3em}{0ex}}\mu \mathrm{m}$ . The maximum power of the laser beam is $6\phantom{\rule{0.3em}{0ex}}\mathrm{mW}$ , and the wavelength is set to $635\phantom{\rule{0.3em}{0ex}}\mathrm{nm}$ to lessen the beam absorption into tissues.

To evaluate the practicability, we used the proposed system to observe the gastric wall of a rat. The figures shown in Sec. 2 were taken during one of those experiments. To estimate the detection rate of the laser spot, we checked 1000 consecutive frames and found that the rate of correct detection was 85%. Most failures occurred when the system mistook reflection highlights for laser spots. It does not matter, however, when it comes to clinical applications, because physicians do not take the trouble to see highlighted areas to observe all the spot locations in each and every frame, but they only need their locations of interest. In addition, our system is made interactive, so that physicians press the foot switch only when they need 3-D measurements while seeing the motion of the laser spots. The detection rate increases to 97% for those areas without highlights.

The processing time for each frame with an Intel Xeon 5160, $3.00\phantom{\rule{0.3em}{0ex}}\mathrm{GHz}\times 2$ is as follows: $9.6\times {10}^{-3}\phantom{\rule{0.3em}{0ex}}\mathrm{sec}$ for the distortion correction, $6.3\times {10}^{-3}\phantom{\rule{0.3em}{0ex}}\mathrm{sec}$ for the laser spot detection, and $1.0\times {10}^{-6}\phantom{\rule{0.3em}{0ex}}\mathrm{sec}$ for the 3-D coordinate calculation and ruler display. Thus, the total processing time for a frame is $1.6\times {10}^{-2}\phantom{\rule{0.3em}{0ex}}\mathrm{sec}$ , so that all the procedures can be performed in real time.

To estimate the accuracy of the distance measurements, we observed a screen at a distance varying between 5 and $40\phantom{\rule{0.3em}{0ex}}\mathrm{mm}$ . We used the parameters obtained at $20\phantom{\rule{0.3em}{0ex}}\mathrm{mm}$ to measure the other distances in such a way as mentioned in Sec. 2.4, and found the differences between the distances that the system calculated and the actual ones are within the range of $0.2\phantom{\rule{0.3em}{0ex}}\mathrm{mm}$ . We also estimated the accuracy of the virtual rulers by observing graph paper at a distance varying between 5 and $40\phantom{\rule{0.3em}{0ex}}\mathrm{mm}$ . We compared the actual lengths on the graph paper and the lengths measured with the virtual ruler, and found that the average error in the measurement is 5.1%.

## 4.

## Conclusions

We present a laser beam endoscope system for 3-D measurements in real time. The system detects the laser spots automatically and calculates their 3-D coordinates by the triangulation method. Physicians can see the rulers that fit the 3-D position, and can measure the size and position of objects on undistorted images, similar to placing rulers on the surface of an organ.

Though we cannot use the proposed system for bedside diagnosis because it needs official medical device evaluation, the experiments using rats show the effectiveness of the system for clinical use.