1.1.

Endoscope Distortion

Optical aberration includes two main types: chromatic aberrations and monochromatic aberrations. The former arises from the fact that the refractive index is actually a function of wavelength. The latter occurs even with quasimonochromatic light and falls into two subgroupings: monochromatic aberrations that deteriorate the image, making it unclear (e.g., spherical aberration, coma, and astigmatism), and monochromatic aberrations that deform the image (e.g., Petzval field curvature and distortion).¹

In this paper, we focus on the monochromatic aberrations that deform the image. We call such aberrations geometric distortions. A geometric distortion is a deviation from the rectilinear projection, a projection in which straight lines in a scene remain straight in their image. While similar distortions can also be seen in display (display distortion, especially in cathode ray tube display), we mainly focus on the geometric distortions caused by geometric optics. Among different types of geometric distortions, radial distortions are the most commonly encountered and most severe. They cause an inward (barrel distortions) or outward (pincushion distortions) displacement of a given image point along the radial direction from its undistorted location (Fig. 4). A radial distortion can also be a combination of both barrel and pincushion distortions, which is called a mustache (or wave) distortion. In an image with a radial distortion, a straight line that runs through the image center (usually also being the center of distortion) remains straight. Since most radial distortions are circularly symmetric (i.e., rotationally symmetric with respect to any angle), or approximately so, arising from the circular symmetry of the optical imaging systems, a circle that is concentric with the image center remains a circle in its image, although its radius may be affected. Some complex distortions include both radial and tangential components, i.e., a given image point displaces along both radial (radial distortion) and tangential (tangential distortion) directions (Fig. 1). Such distortions, called radial-tangential distortions in this paper, include decentering distortions and thin prism distortions.^2,3 Unless otherwise specified, distortions hereafter mentioned in this paper mean radial geometric distortions—the focus of this paper.

Endoscopes usually have severe barrel distortions. An endoscope needs a short focal length and a wide FOV in order to observe a broad area with minimum moving or bending of the endoscope, which is essential for steady and smooth manipulation of the endoscope because of the restricted space and degrees of freedom of movement and the limitation in hand-eye coordination during surgical cases.⁴ However, lenses used in endoscopes usually have a short focal length (a few millimeters only) and a wide FOV (ranging from 100 to 170 deg), which inevitably causes severe distortions.⁵ Typically, endoscopes exhibit barrel distortions. Occasionally, an endoscope exhibits a mustache distortion that varies between barrel and pincushion across the image, mostly because a mathematical algorithm is used to correct distortion at the maximum image height or other parts of the image. Since endoscope distortions can negatively affect size estimation and feature identification related diagnosis,^5–7 quantitative evaluation of endoscope distortions and proper understanding of the evaluation results are essential.

1.2.

Need for an Endoscope Distortion Evaluation Method

The millions of endoscopic procedures conducted monthly in the United States for a wide range of indications are driving the advancement of endoscopic imaging technology. With new diagnostic capabilities and more complex optical designs, technological advances in endoscopes promise significant improvements in both safety and effectiveness. Endoscope optical performance (OP) can be evaluated by OP characteristics (OPCs), including resolution, distortion, FOV, direction of view, depth of field, optimal working distance, image noise, detection uniformity, veiling glare, and so on.

Current consensus standards provide limited information on validated and quantitative test methods for assessing endoscope OP. There is no standardized method to evaluate endoscope distortions. An international standard specifies methods of determining distortion in optical systems.⁸ The methods require the usage of complex devices, such as an autocollimator, or an instrument to measure the object and image pupil field angles and height. While the standard provides complex equations, it does not clarify how the distortion results should be presented and evaluated. Also, the picture height distortion value mentioned in the standard is insufficient for the evaluation of severe barrel or pincushion distortions, and fails for the evaluation of mustache distortions. The definitions of angular magnification and lateral magnification in this standard are only based on a small area near the optical axis of the test specimen, which cannot be extended to endoscopes whose magnification changes significantly within the FOV. The endoscopes working group (WG) of the International Organization for Standardization (ISO), ISO/TC172/SC5/WG6, develops and oversees endoscope standards (the ISO 8600 serial standards) that cover the endoscope OPCs of FOV, direction of view, and optical resolution. However, an endoscope distortion standard has not yet been developed by this WG.

While endoscopic technology is developing fast, the regulatory science for endoscope OP evaluation has been unable to keep pace. Every year, the U.S. Food and Drug Administration receives a large number of endoscope submissions for premarket notification or premarket approval. However, the evaluation of new video endoscopic equipment is difficult because of the lack of objective OP standards. The industry lacks consensus standards on objective test methods to evaluate distortion. The resulting patchwork of tests conducted by different device manufacturers leads to delays in bringing important endoscopic technology to market and may allow the clearance of a less optically robust system that negatively impacts patient care.

In this paper, we tried to establish a quantitative, objective, and simple distortion evaluation method for endoscopes, with the goal of applying the method in an international endoscope standard. We reviewed some common methods described in prior journal articles for distortion evaluation of an optical imaging system and analyzed the relationship between these methods. Based on the review, a quantitative test method for assessing endoscope radial distortion was developed and validated based on the local magnification idea. The method will help facilitate performance characterization and device intercomparison for a wide variety of standards and endoscopic imaging products. The method has the potential to facilitate the product development and regulatory assessment processes in a least burdensome approach by reducing the workload on both the endoscope manufactures and the regulatory agency. As a result, novel, high-quality endoscopic systems can be swiftly brought into the market. The method can also be used to facilitate the rapid identification and understanding of the cause for poorly performing endoscopes, and benefit quality control during manufacturing as well as quality assurance during clinical practices.

2.1.

Picture Height Distortion and Related Methods

There are several methods for distortion evaluation. The picture height distortion method ($D_{\mathrm{PH}}$, where $D$ means distortion) is defined by the European Broadcasting Union (EBU)¹⁰ and recommended by the ISO 9039 International Standard.⁸ It quantifies the bending of the image of a horizontal straight line that is tangent to the circumscribed (for barrel distortion) or inscribed (for pincushion distortion) rectangle of the distorted image (Fig. 2). As shown in Fig. 2, it is calculated as

Fig. 1

Definitions of radial-tangential distortion on the 2-D plane.

Fig. 2

Picture height distortion ($D_{\mathrm{PH}}$): (a) barrel distortion and (b) pincushion distortion.

Another distortion evaluation method is presented in Fig. 3.¹³ If we draw a straight line connecting two ends of a curved line—the image of a straight line in the target, its length is $L$ and the largest distance from this drawn line to any point on the curved image line is $l$ [Fig. 3(a)]—then the distortion is defined as

As opposed to $D_{\mathrm{PH}}$, the $D_{\mathrm{PH}2}$ values are positive for barrel distortions and negative for pincushion distortions. Otherwise, the definition of $D_{\mathrm{PH}2}$ is similar to that of $D_{\mathrm{PH}}$. The absolute value of $l$ in $D_{\mathrm{PH}2}$ is the same as that of $\mathrm{\Delta }H$ in the $D_{\mathrm{PH}}$ method for the lower horizontal edge of a distorted image. Comparing Figs. 2 and 3, we can get the relation of $D_{\mathrm{PH}2}=-D_{\mathrm{PH}}\times H/L$. Since the $D_{\mathrm{PH}2}$ method has no significant advantage over the $D_{\mathrm{PH}}$ method, we do not recommend this method for distortion evaluation.

Fig. 3

$D_{\mathrm{PH}2}$: (a) definition of $D_{\mathrm{PH}2}$, (b) the relation between $D_{\mathrm{PH}2}$ and $D_{\mathrm{PH}}$.

The aforementioned distortion evaluation methods calculate the largest positional error of barrel or pincushion distortions over the whole image. They are meaningful only if the optical system has a steadily increasing distortion (barrel or pincushion distortion) from the image center to the edges. For a complex distortion pattern, it is impossible to evaluate the distortion in detail with a single value since the value might be misleading. Taking a mustache distortion as an example, it is possible that the image displays little or virtually zero distortion at the edges as measured by any of these methods, but a maximum distortion at the midfield. These methods are also related to the aspect ratio of the distorted image. The EBU defines the $D_{\mathrm{PH}}$ for the case of aspect ratio being $4:3$, a ratio for the traditional television and computer monitor standard. However, there are other widely used aspect ratios, such as the $3:2$ ratio of the classic 35 mm still camera film and the $16:9$ ratio of HD video. We cannot directly compare the distortion values of two images with different aspect ratios.

2.2.

Radial Distortion Method

Another distortion evaluation method is based on comparing the radii of distorted ($R_d$) and undistorted ($R_u$) images. It is assumed that the distortion close to the optical center is zero. Therefore, an undistorted image can be calculated based on the information at the center of the distorted image. The distorted image is then evaluated with the undistorted image as a reference along the radial direction. Since this method can be applied to any radial distortion, we call it $D_{\mathrm{RAD}}$. As shown in Fig. 4,

Fig. 4

Definition of $D_{\mathrm{RAD}}$: (a) undistorted image, (b) barrel distortion, and (c) pincushion distortion.

$D_{\mathrm{RAD}}$ can be used to evaluate complex distortions (e.g., mustache distortions) with a distortion profile along a radius line. Mustache distortions can be caused by countermeasures in the design or by an image processing algorithm to limit or remove distortion. If we calculate the $D_{\mathrm{RAD}}$ along a diagonal from the image center to a corner, we can obtain a curve of $D_{\mathrm{RAD}}$ versus $R_u$ or $R_d$, as shown in Fig. 5. For a simple barrel or pincushion distortion, we can identify a barrel distortion if the $D_{\mathrm{PH}}$ or $D_{\mathrm{STV}}$ value is negative and a pincushion distortion if the value is positive. However, this criterion will fail for identifying a mustache distortion. The key point is that the identification of a distortion type should not depend on the sign of the distortion value but on the slope of the radial distortion curve. Typical radial distortion curves are shown in Fig. 5. Generally, these distortion curves start at zero, matching the assumption that the distortion close to the optical center is zero. For a barrel distortion [Fig. 5(a)], the curve slope is always negative; therefore, the distortion values are also negative. For a pincushion distortion [Fig. 5(b)], the curve slope is always positive, resulting in the distortion values being positive. For a mustache distortion, the curve has both positive and negative slope values at different regions. From Fig. 5(c), the curve has negative slope when $R_d<11.7\mathrm{mm}$ and positive slope when $R_d>11.7\mathrm{mm}$. This means that the image has a barrel distortion for $R_d<11.7\mathrm{mm}$ and a pincushion distortion for $R_d>11.7\mathrm{mm}$ even though the distortion values are still negative for $R_d>11.7\mathrm{mm}$. A higher absolute slope value means more pronounced distortion at this radius.

Fig. 5

Typical $D_{\mathrm{RAD}}$ curves: (a) barrel distortion, (b) pincushion distortion, and (c) mustache distortion.

For a simple barrel or pincushion distortion, the absolute value of radial distortion calculated from an image corner is usually larger than that of $D_{\mathrm{PH}}/D_{\mathrm{TV}}$ or $D_{\mathrm{STV}}$.¹⁵ This can be theoretically explained with the barrel distortion in Fig. 4 as an example. In Fig. 4(a), points $A$ and $B$ are, respectively, the middle point and right corner of the upper edge of the undistorted image, with their distance to the image center in vertical direction being $R_{uy}$. In Fig. 4(b), points $A^{\prime }$ and $B^{\prime }$ are the image of $A$ and $B$, with their distances to the image center in vertical direction being $R_{A^{\prime }}$ and $R_{dy}$. $R_{A^{\prime }}$ is larger than $R_{dy}$ for barrel distortion and smaller for pincushion distortion. $D_{\mathrm{STV}}$ at corner $B^{\prime }$ can be calculated as $D_{\mathrm{STV}}=(R_{dy}-R_{A^{\prime }})/R_{A^{\prime }}$, and $D_{\mathrm{RAD}}$ for corner $B^{\prime }$ can be calculated as $D_{\mathrm{RAD}}=(R_d-R_u)/R_u$. For a barrel distortion, both $A$ and $B$ are distorted toward the image center in the vertical direction with the amount of ($R_{A^{\prime }}-R_{uy}$) and ($R_{dy}-R_{uy}$), respectively, with the negative value signifying a barrel distortion. At the same time, $B^{\prime }$ moved the amount of ($R_{dy}-R_{A^{\prime }}$) in the vertical direction toward the image center relative to $A^{\prime }$. Therefore, the equation of $D_{\mathrm{STV}}=(R_{dy}-R_{A^{\prime }})/R_{A^{\prime }}$ evaluates only the position of $B^{\prime }$ relative to $A^{\prime }$. On the other hand, the equation of $D_{\mathrm{RAD}}=(R_d-R_u)/R_u$ can be split into horizontal component of $(R_{dx}-R_{ux})/R_{ux}$ and vertical component of $(R_{dy}-R_{uy})/R_{uy}$. The vertical component is the movement of $B$ and can also be expressed as $[(R_{dy}-R_{A^{\prime }})+(R_{A^{\prime }}-R_{uy})]/R_{uy}$, which is the sum of the movement of $B$ relative to $A^{\prime }$ and the movement of $A$. Since the values of $R_{uy}$ and $R_{A^{\prime }}$ are close, $D_{\mathrm{STV}}$ is roughly equal to the movement of $B$ relative to $A^{\prime }$ in the vertical component of $D_{\mathrm{RAD}}$ for corner $B$, without considering the movement of $A$ in the vertical direction and the movement of $B$ in the horizontal direction. That explains why the absolute $D_{\mathrm{STV}}$ value is usually smaller than the absolute $D_{\mathrm{RAD}}$ value. Similar conclusion can also be obtained for a pincushion distortion.

The main problem with the $D_{\mathrm{RAD}}$ method is that the values of $R_u$ are not available, since the undistorted image does not physically exist. The $R_u$ values can be approximated according to the data at the center of the distorted image, with the assumption that there is none or little distortion near the optical axis.¹⁶ Taking the image of a grid target as an example, if we know the distance of one grid (reference grid) at the center of the distorted image is corresponding to $P$ pixels on the image sensor, then the distance of $G$ grids from the image center in the undistorted image should be $P\times G\text{pixels}$, based on which we can calculate the undistorted image of the grid target and then the $D_{\mathrm{RAD}}$ values. However, there is a problem with this approach. The size of the assumed undistorted area in the distorted image will affect the final results for a severe distortion as shown in most endoscopic images. If the reference grid is too big, it might have been distorted.

Figure 6(a) shows a barrel distorted image. From the image center to the right edge in the horizontal direction, we got the distance ($R_d$) of each cross-point from the center. By assuming that there was no distortion between point 0 and point 1 (0–1), point 0 and point 2 (0–2), or point 0 and point 3 (0–3), respectively, we calculated the undistorted distance ($R_u$) of the cross-points with these different assumptions and obtained three $D_{\mathrm{RAD}}$ curves as shown in Fig. 6(b). For a barrel distortion, the distortion values should monotonically decrease with radius, with the maximum value of zero at the center. However, the 0–2 and 0–3 curves in Fig. 6(b) showed positive values at shorter radial distances, which illuminated that the assumptions of no distortion from point 0 to point 2 or point 0 to point 3 are less accurate than the no distortion assumption from point 0 to point 1, and bigger assumed nondistortion area causes bigger error. On the other hand, if the assumed nondistortion area was too small, the reading error for $R_d$ at center would be enlarged by the large number of grids at further radial distance when calculating $R_u$.

Fig. 6

Effect of the size of the assumed-undistorted area on the $D_{\mathrm{RAD}}$ results: (a) a barrel distorted image, and (b) $D_{\mathrm{RAD}}$ curves with the undistorted radius data calculated based on different sizes of the assumed-undistorted area near the image center (the distance from image center to left or right edge is normalized as 1).

3.1.

Experimental Measurements

We established a distortion evaluation method using an endoscopic system (Olympus EVIS EXERA II) that includes a high-intensity xenon light source (CLV-180), a gastrointestinal videoscope (GIF-H180), and a video system center (CV-180). This system has the common barrel distortion as seen in most endoscopes. A series of grid targets with three different grid sizes ($0.5\mathrm{mm}\times 0.5\mathrm{mm}$, $1.5\mathrm{mm}\times 1.5\mathrm{mm}$, and $3.0\mathrm{mm}\times 3.0\mathrm{mm}$) was designed and printed with a laser printer. The total size of each target was large enough to cover the whole FOV area. The target during tests should be planar and be able to move along the optical axis. To securely keep the endoscope in its place, a customized mold with adjustable height was used. The direction of the endoscope distal end was adjusted with a fiber optic positioner (Newport, FPR2-C1A).

The setup was adjusted so that the endoscope optical axis was perpendicular to the test target and aligned with the test target center (Fig. 7). Criteria for acceptable adjustment are as follows: (1) center of the target, the point on the target that locates at the FOV center, was located at the center of the captured image and (2) two pairs of centrally symmetric points (e.g., $A-A^{\prime }$ and $B-B^{\prime }$ in Fig. 7) on the target were also centrally symmetric on the image. It was assumed that the distortion center (i.e., where the optical axis passes through the image plane) overlaps with the image center. The assumption was evaluated in Sec. 4.1. For the first criterion, the target was positioned at a given distance to take its image. The image was then analyzed with software (e.g., MATLAB) to find the target center $(x_t,y_t)$ in the image. The distance from the target center to the image center $(x_i,y_i)$ was calculated using the formula $\sqrt{{(x_t-x_i)}^2+{(y_t-y_i)}^2}$, with the unit of pixels. The distance was controlled within $1\%$ of the picture height [$\sim 10\text{pixels}$ for the image with size of $1008(H)\times 1280(W)\text{pixels}$]. For the second criterion, the same method as for the first criterion was used to make sure that the midpoint between each pair of points was $<1\%$ of the picture height far away from the target center in the image. The two criteria were satisfied through an iterative trial-and-error process.

Fig. 7

Distortion measurement setup: (a) test target and (b) test setup.

3.2.

Local Magnification Method to Quantify Distortion

To avoid the aforementioned problem of calculating $R_u$ in Sec. 2.2, we proposed to evaluate radial distortions with a new approach—the local magnification ($M_L$) method. For a small object (e.g., a small cross) placed at a local point on the test target, the ratio of the object length on the image sensor (or on a display device from the sensor) to its actual length on the test target is called $M_L$. The term “local magnification” is borrowed from the IEC 1262-4 Standard, which addresses determination of image distortion in electro-optical x-ray image intensifiers.¹⁹ In the standard, discrete $M_L$ values are obtained by measuring size changes in small hash marks and their accuracy can be affected by the sizes of the marks. In this paper, $M_L$ is expressed with an equation that can accurately and continuously express distortion at any location in the FOV. $M_L$ can be separated into the local radial magnification ($M_{\mathrm{LR}}$) and the local tangential magnification ($M_{\mathrm{LT}}$). $M_{\mathrm{LR}}$ is the local magnification of a small one-dimensional (1-D) object oriented radially toward the FOV center. $M_{\mathrm{LT}}$ is the local magnification of a small 1-D object tangentially oriented to a radial direction.

Figure 8 shows the $M_{\mathrm{LR}}$ and $M_{\mathrm{LT}}$ of a cross-shape object (ideally, the object should be infinitely small) with the width of $lr$ and height of $lt$ at radius of $R$ on the test target. The object image located at radius of $R^{\prime }$ on the target image with the width of $l{r}^{\prime }$ and height of $l{t}^{\prime }$. Then the $M_{\mathrm{LR}}$ and $M_{\mathrm{LT}}$ at the cross-point can be calculated as $M_{\mathrm{LR}}=l{r}^{\prime }/lr$ and $M_{\mathrm{LT}}=l{t}^{\prime }/lt$.

Fig. 8

$M_{\mathrm{LR}}$ and $M_{\mathrm{LT}}$: (a) a small cross-shape object at radius $R$ on the test target and (b) image of the object at radius $R^{\prime }$.

Assuming the distortion is radial, data from any straight line crossing the image center can be used to evaluate the distortion of the whole image if the target is well aligned with the device. We used the horizontal line from the image center to the right edge [Fig. 6(a)] as an example to explain the $M_L$ method. Since the distance from the image center to the right edge is not the maximum radius in the whole image, the final evaluation results mainly reflect the distortion characteristics in a circle area with the radius equal to the distance from image center to right edge. Other straight lines (e.g., a vertical or diagonal line) crossing the image center can also be used to cover a bigger circle area or obtain more accurate results. The method is described in detail as follows.

After proper alignment, an image of the target at an established distance from the endoscope is taken [Fig. 6(a)]. The horizontal line from the image center to the right edge is then used to analyze the radial distortion. Following this, the coordinates of each cross-point on the line are read with image analysis software, and the distances ($R_d$) of the points from the image center are calculated in terms of pixels. The actual distance ($R_u$) of these cross-points from the center on the target can also be obtained. To be simple, $R_u$ is used as the number of grids from the center to the cross-points (Table 1), instead of measuring the actual distances. A matrix of $R_d$ is then mapped to a matrix of $R_u$.

Table 1

Example of evaluating geometric distortion.

The $R_u$ value on the image edge is needed in order to evaluate the distortion at the edge. In most cases, however, an image near the edge is often blurred due to severe distortion and vignette effects. Additionally, the edge may not exactly lie on a cross point, and the number of cross points from the center to the edge would therefore not be an integer. Based on available $R_d$ and $R_u$ data from the cross-points, a polynomial equation of $R_u=f(R_d)$ is calculated. The maximum pixel number from the image center to the image edge (i.e., half the picture width, 640 in our images) is then used as $R_d$ to calculate $R_u$ at the edge (bold numbers in Table 1). The $R_u$ value of 12.2 is obtained in this example.

Both $R_d$ and $R_u$ are normalized, setting their maximum value as 1 (Table 1). From the curve of normalized $R_d$ versus normalized $R_u$ (Fig. 9) or vice versa, a polynomial fitting equation of $R_d=f_d(R_u)$ or $R_u=f_u(R_d)$ is created to fit and define the relation between $R_u$ and $R_d$.

Fig. 9

Relations of normalized $R_d$ versus normalized $R_u$ based on Fig. 6(a): (a) $R_d=f_d(R_u)$ and (b) $R_u=f_u(R_d)$. The dashed lines are the fitting curves. By default, $y$ represents the variable on the vertical axis and $x$ represents the variable on the horizontal axis in the fitting equations.

The normalized $R_d$ versus normalized $R_u$ polynomial function of $R_d=f_d(R_u)$ can be easily converted to other scales. Assume the function has a polynomial form^9,20,21 of

3.2.1.

Local radial magnification method

If $X$ and $Y$ are the actual lengths of $R_u$ and $R_d$, $M_{\mathrm{LR}}$ is defined as follows from the derivative of Eq. (6):

$$M_{\mathrm{LR}}=\frac{dY}{dX}=\frac{1}{k_y}[n{k}_n{k}_x^n{X}^{n-1}+(n-1)k_{n-1}{k}_x^{n-1}{X}^{n-2}+\dots +2k_2{k}_x^{2}X+k_1{k}_x].$$

Substituting $X=x/k_x$ in the above equation, we get

Eq. (7)

$$M_{\mathrm{LR}}=\frac{k_x}{k_y}[n{k}_n{x}^{n-1}+(n-1)k_{n-1}{x}^{n-2}+\dots +2k_{2}x+k_1].$$

Taking the data in Table 1 as an example, we can get the following fitting equation based on Eq. (5), with the normalized $R_d$ as $y$ and normalized $R_u$ as $x$.

Eq. (8)

$$y=-0.6896x^5+2.5386x^4-2.8434x^3+0.2744x^2+1.7125x.$$

Degree 5 is the lowest degree of the polynomial fitting equation with the $R$-squared value of the fitting equation being $>0.9999$. The target grid size is $3\mathrm{mm}\times 3\mathrm{mm}$. So the maximum $X$ is 36.6 mm ($3\times 12.2$) and $k_x$ is 0.0273 ($1/36.6$). Assuming the image sensor has pixel size of $2.8\mu \mathrm{m}\times 2.8\mu \mathrm{m}$, the maximum $Y$ is 1.792 mm ($2.8\times 640/1000$) and $k_y$ is 0.5580 ($1/1.792$). Then, Eq. (7) for these data will be

Eq. (9)

$$M_{\mathrm{LR}}=-0.1688x^4+0.4972x^3-0.4177x^2+0.0269x+0.0838,$$

from which the $M_{\mathrm{LR}}$ data at each cross-point can be calculated as shown in Table 1. The $M_{\mathrm{LR}}$ data can be normalized so that the maximum magnification at the image center is one. The $M_{\mathrm{LR}}$ and normalized $M_{\mathrm{LR}}$ versus normalized $R_d$ curves are shown in Fig. 10, from which we can see that the two curves can totally overlap by adjusting the scale of $y$ coordinates.

Fig. 10

$M_{\mathrm{LR}}$ and normalized $M_{\mathrm{LR}}$ versus normalized $R_d$.

3.2.2.

Local tangential magnification

$M_{\mathrm{LT}}$ as defined in Fig. 8 is equal to the ratio of two circumferences with $R^{\prime }$ and $R$ as radius, respectively, $M_{\mathrm{LT}}=(2\pi R^{\prime })/(2\pi R)=R^{\prime }/R$. The equation for $M_{\mathrm{LT}}$ can be derived from Eq. (6). If $X$ and $Y$ are the actual lengths of $R_u$ and $R_d$, we define $M_{\mathrm{LT}}$ as

$$M_{\mathrm{LT}}=\frac{Y}{X}=\frac{1}{k_y}(k_n{k}_x^n{X}^{n-1}+k_{n-1}{k}_x^{n-1}{X}^{n-2}+\dots +k_2{k}_x^{2}X+k_1{k}_x).$$

Substituting $X=x/k_x$ in the above equation, we get

Eq. (10)

$$M_{\mathrm{LT}}=\frac{k_x}{k_y}(k_n{x}^{n-1}+k_{n-1}{x}^{n-2}+\dots +k_{2}x+k_1).$$

There is no $M_{\mathrm{LT}}$ value at $x=0$, because the ratio of two circumferences both with a value of zero cannot be calculated. Again, take the data in Table 1 as an example. Assuming $k_x$ is 0.0273 and $k_y$ is 0.5580, we can get the $M_{\mathrm{LT}}$ equation based on Eqs. (8) and (10) as follows:

Eq. (11)

$$M_{\mathrm{LT}}=-0.0337x^4+0.1242x^3-0.1391x^2+0.0134x+0.0838,$$

from which the $M_{\mathrm{LT}}$ and normalized $M_{\mathrm{LT}}$ data can be calculated as shown in Table 1.

Based on Table 1, we can compare $M_{\mathrm{LR}}$ and $M_{\mathrm{LT}}$ as shown in Fig. 11. While we use the normalized $R_d$ as the $x$ axis, the $x$ axis can also be the normalized $R_u$ based on the requirement. From Fig. 11, $M_{\mathrm{LR}}$ and $M_{\mathrm{LT}}$ are the same at the center position, when $R_d$ is less than 0.2. However, $M_{\mathrm{LR}}$ decreases faster than $M_{\mathrm{LT}}$ with increasing $R_d$, which indicates that the image of an object will be compressed further in the radial direction than in the tangential direction, when $R_d$ is greater than 0.2. This property is important for a physician to interpret an endoscopic image.

Fig. 11

$M_{\mathrm{LR}}$ and $M_{\mathrm{LT}}$ versus normalized $R_d$.

3.3.

Deriving $D_{\mathrm{RAD}}$ and $D_{\mathrm{PH}}$ (or $D_{\mathrm{STV}}$) from $M_{\mathrm{LR}}$

The local magnification method could also help improve the traditional way of calculating $D_{\mathrm{RAD}}$ and $D_{\mathrm{PH}}$. Assume that $X$ and $Y$ are the actual sizes of the undistorted and distorted images, respectively. (Please note that $X$ represents the actual size of the target in Sec. 3.2, but the equations are exactly the same.) The radial distortion equation can be obtained from Eq. (6) as

Taking the data in Table 1 as an example, we assume the normalized $R_d$ as $Y$ (then $k_y=1$) and the undistorted images has maximum diameter of 1.71, and $k_x$ is 0.585 ($1/1.71$). (Assuming the center grid of the distorted image is undistorted, the undistorted image has a diameter of $0.14\times 12.2=1.71$.) Therefore, the $D_{\mathrm{RAD}}$ values can be calculated (Table 1) based on Eqs. (8) and (12). While an image can be magnified to different sizes by changing the values of $k_x$ and $k_y$, the ratio of $k_x$ to $k_y$ should be constant, i.e., $D_{\mathrm{RAD}}$ should be constant.

We can also calculate the traditionally used $D_{\mathrm{PH}}$ or $D_{\mathrm{STV}}$. Take the barrel distortion in Fig. 4(b) as an example. Assume that the distorted image is $2W_d$ wide and $2R_{A^{\prime }}$ high and we have obtained two functions of $R_u=f_u(R_d)$ and $R_d=f_d(R_u)$ with $R_d$ and $R_u$ being normalized values. Then the $D_{\mathrm{PH}}$ or $D_{\mathrm{STV}}$ of a barrel distortion can be calculated with the procedure shown in Fig. 12. For pincushion distortion, similar method can be used. The advantage of calculating $D_{\mathrm{PH}}$ or $D_{\mathrm{STV}}$ with this method is that it does not need lines that are tangent to the image boundaries [for barrel distortion, Fig. 13(a)] or whose end points overlap with the image corners (for pincushion distortion). For example, while Fig. 13(a) is an ideal image to analyze $D_{\mathrm{STV}}$ in the traditional way, Fig. 13(b) is not because there are no lines being tangent to the edges. However, the method in Fig. 12 will work for both images. Most importantly, $D_{\mathrm{PH}}$ or $D_{\mathrm{STV}}$ can be directly calculated if the polynomial equations describing the relationship between $R_d$ and $R_u$ are known.

Fig. 12

Procedure used to calculate $D_{\mathrm{PH}}$ or $D_{\mathrm{STV}}$ of a barrel distortion.

Fig. 13

Distorted images of a grid target: (a) four lines tangent to the image edges and (b) no line tangent to the edges.

4.1.

Assumptions of Circular Symmetry and Overlap of the Image Center with the Distortion Center

As mentioned before, we assumed that the endoscope is circularly symmetric, and therefore, the tangential component of the distortion can be ignored. We also assume that the distortion center overlaps with the image center. These assumptions can be verified. In Sec. 3, we used the data from the image center to right edge of Fig. 6(a) to demonstrate the distortion evaluation methods and their results (Table 1). Similar results can also be obtained based on data on any other radius. If the endoscope is circularly symmetric and the distortion center overlaps with the image center, the distortion results based on data from different radii should be close.

Four sets of data on four radii, i.e., the image center to the right, left, top, and bottom edges, were obtained from Fig. 6(a) to derive the normalized $R_d$ versus normalized $R_u$ curves (Fig. 14). The data were normalized with the longest radius as one. From Fig. 14, the four curves overlap, meaning the assumption of a circularly symmetric optical system without tangential distortion is correct and the distortion center overlaps with the image center.

Fig. 14

Normalized $R_d$ versus normalized $R_u$ based on data from radial lines at four different directions on the same distorted image.

4.2.

Accuracy of the Local Magnification Method

To evaluate the accuracy of the obtained $M_{\mathrm{LR}}$ results, we applied the results in a MATLAB routine to correct a distorted image from the same endoscope. The distorted and corrected images are shown in Fig. 15. Since the points selected to establish the distortion function did not cover the four corners of the image, the equation only covers a circle area [Fig. 6(a)] with the radius as the distance from the image center to the furthest point. Therefore, only the image located within the circle was corrected with the four corners outside the circle discarded. Overall, the corrected image removed the vast majority of the distortion originally present. Some errors were still present near the image boundary. The main reason for the errors was that the coordinate reading for a point at further distance from the center was less accurate than at closer distance because of the smaller magnification, lower resolution, and dimmer light intensity at a further distance than at the center position. By adjusting the illuminating light, the accuracy can be improved.

Fig. 15

Original and corrected images: (a) Original distorted image taken with the endoscope and (b) the corrected image according to $M_{\mathrm{LR}}$.

4.3.

Number of Data Needed for Distortion Evaluation and the Formats of Polynomial Equations

Effect of the number of grids imaged from image center to edge (i.e., number of points to derive the polynomial equation of normalized $R_d$ versus normalized $R_u$) on distortion evaluation was studied. A grid target with the grid size of $3\mathrm{mm}\times 3\mathrm{mm}$ was placed 6.4 cm away from the endoscope distal end and perpendicular to the optical axis of the endoscope. The distorted image of the target was used to analyze the effects of data number on distortion curves/equations. From the image center to the right edge, 34 radial distance data were obtained from 34 cross-points. From these 34 data, 18, 12, 8, 6, and 5 data were selected, respectively, with roughly even distribution to obtain the normalized $R_d$ versus normalized $R_u$ curve as shown in Fig. 16. From the figure, all the curves overlap with the curve based on all the 34 sets of data, which means that the minimum data number for distortion evaluation can be the number of unknown constant parameters in the distortion equation [e.g., the five parameters of $k_1,k_2,\dots ,k_5$ in Eq. (5) with $k_0$ set as zero] if the cross-points on the target image can be clearly read, on the premise that the equation format is correct.

Fig. 16

Normalized $R_d$ versus normalized $R_u$ based on different number of cross-point data on the image (the legend shows the number of cross-points used to get the fitting lines).

For all the normalized and non-normalized fittings, a polynomial fitting equation of degree 5 is accurate enough for most severe barrel or pincushion distortions, with the $R$-squared value $>0.9999$. However, the actual degree of a fitting equation can be flexible based on the required $R$-squared value. For example, the degree can be 2, 3, and 4 if the required $R$-squared values are 0.9898, 0.9987, and 0.9998, respectively, for the endoscope we evaluated. The degree can be $>5$ for more complex distortion. The equations can have all the terms from degree 0 to the degree of the equation or only some of the terms (e.g., the constant term is not necessarily zero or the equation can only have terms with degrees of odd numbers²²).

4.4.

Projection Methods of Endoscopes

Distortion is the consequence of the projection method used in an optical design. Theoretically, the distortion pattern can be derived based on the known projection method, which is usually unknown to consumers. On the other hand, if the distortion pattern is known, the projection method can be inversely derived.

Most consumer cameras have rectilinear lenses based on the perspective projection (also called gnomonic projection) that renders a straight line in the object space as a straight line in the image. The perspective projection obeys the mapping function of $r=f·\tan (\theta )$, where $\theta$ is the angle between the optical axis and the line from an object point to the entrance pupil center, $r$ is the distance from the image of the object point to the image center, and $f$ is the focal length of the optical system.¹⁵ For a 2-D object that is perpendicular to the optical axis of the camera, perspective projection can produce an image that faithfully reflects the geometry of the object. However, it is difficult to make a rectilinear lens with more than 100 deg of FOV. Therefore, some other projection methods (Fig. 17), such as stereographic [$r=2f·\tan (\theta /2)$], equidistant ($r=f·\theta$), equisolid angle [$r=2f·\sin (\theta /2)$], and orthographic/orthogonal [$r=f·\sin (\theta )$] projections, are used to design lenses with a wide FOV, such as fisheye lenses and lenses in endoscopes.²³

Fig. 17

Some projection methods for lenses with a wide FOV (assuming $f$ is 1): (a) r versus θ, and (b) normalized r versus θ.

The projection method of an endoscope can be derived based on its distortion evaluation results. The 34 sets of data in Sec. 4.3 were used as an example of deriving the projection method of an endoscope. These data were obtained from 33 cross-points plus the image center in the distorted image of a grid target with $3\mathrm{mm}\times 3\mathrm{mm}$ grid size. The distance ($d$) from each cross-point to the target center can be calculated by multiplying the grid size of 3 mm by the grid number. The distance ($l$) from the target to the distal end of the endoscope is 6.39 cm. Then the angle $\theta$ was calculated for each cross-point with the equation $\theta =\text{arctangent}(d/l)$. Strictly speaking, $l$ should be the distance from the target to the endoscope entrance pupil, which is not necessarily the endoscope distal end. However, if the distance from the endoscope distal end to entrance pupil is much smaller than the distance from the target to the distal end, the distal end location can be used to approximate the entrance pupil location. The distance from the entrance pupil to the distal end should be considered when the distance from the target to the distal end is short or for special design where the distal end is not a lens, such as a capsule endoscope. So we got 34 $\theta$ values, including the 0 deg from the target center, with the maximum angle being 0.9976 (or 57 deg) for the 33rd cross-point from the target center. We also had 34 normalized $R_d$ values [same as normalized $r$ in Fig. 17(b)] corresponding to these angles. We used normalized $r$ because we did not know the image sensor size to calculate the actual $r$ values. So we got a curve of normalized $R_d$ versus $\theta$ for the endoscope.

To determine the projection method of the endoscope, we normalized the $r$ values in Fig. 17(a) for $\theta$ from 0 to 0.9976 and compared these curves with the curve from the endoscope data, as shown in Fig. 17(b). From Fig. 17(b), the endoscope adopted the orthographic/orthogonal projection during the design since the measured curve almost overlaps with the curve of $r=\sin (\theta )$. This curve can achieve bigger FOV with a given image sensor size than other curves in Fig. 17.

We can also get other optical parameters of the endoscope in the process of analyzing its projection method. The endoscope FOV in the horizontal direction is twice of the maximum angle, i.e., 1.98 (114 deg). If we know the size of the image sensor, we can also calculate the actual values of $r$ and in turn calculate the focal length using the equation $r=f·\sin (\theta )$.