## 1.

## Introduction

Noncontact measure techniques, such as speckle photography,^{1} hologram interferometry,^{2} and laser Doppler vibrometry,^{3} have been developed for years and well applied in various fields. Compared to traditional measurement devices, such as accelerometer or linear displacement gauge, devices with noncontact approaches have a more flexible installation and provide intuitionistic exhibitions of the actual movements of the target without affecting its behavior. In some environments where traditional sensors do not have clear access or cannot work effectively, e.g., remote measurement targets or fields with high temperature or strong magnetic, noncontact measurement devices obviously have great advantages over conventional ones. However, most noncontact equipment requires high cost and strict construction structures, thus limiting the wide use of such systems in practical applications.

Technological developments in image sensors and optics lens have contributed to the rapidly increasing use of vision-based measurement methods as noncontact measurement methods in numerous research and industrial areas, such as vibration analysis,^{4}^{,}^{5} condition monitoring,^{6}7.8.9.10.^{–}^{11} human motion,^{12}^{,}^{13} and underwater measurement.^{14} With a relatively lower cost and better flexibility in structure, optical devices and cameras offer effective alternatives to noncontact equipment. Benefiting from the wide availability of affordable high-quality digital image sensors and high-performance computers, cheap high-resolution cameras have been growing used in many areas. Recently, vision-based techniques were successfully used to measure various structures and get satisfactory results.^{15}16.17.18.19.20.21.22.23.^{–}^{24} Quan et al.^{25} achieved three-dimensional displacement measurement based on two-dimensional (2-D) digital image correlation (DIC). Kim et al.^{26} proposed a vision-based monitoring system that uses DIC to evaluate the cable tensile force of a cable-stayed bridge. The same method was also applied in experimental mechanics for noncontact, full-field deformation measurement.^{27}^{,}^{28} Park et al.^{29} realized displacement measurement for high-rise building structures using the partitioning approach. Wahbeh et al.^{30} realized the measurement of displacements and rotations of the Vincent Thomas Bridge in California by using a highly accurate camera in conjunction with a laser tracking reference. Fukuda et al.^{31} proposed a camera-based sensor system in which a robust object search algorithm was used to measure the dynamic displacements of large-scale structures. Feng et al.^{32} developed a vision-based sensor that employed an up-sampled cross-correlation (UCC) algorithm for noncontact structural displacement measurement, which can accurately measure the displacements of bridges.^{33}

The traditional camera system for displacement measurement is composed of commercial digital cameras and video-processing devices (normally computers). However, ordinary digital cameras often have low video-sampling rate, which limits the application range of their vibration frequency.^{34} To overcome this restriction, a high-speed vision system with 1000 frames per second (fps) or even higher has been developed and applied in practice.^{35} In the present paper, a high-speed camera sensor system composed of a zoom optical lens, a high-speed camera body with a CCD receiver, and a notebook computer. A USB 3.0 interface is used to ensure stable data transfer between the camera body and the computer. On the notebook computer, the captured video can be processed by software to realize the tracking of an object and to extract motion information in real time.

Similar to other measurement equipment, a vision-based measurement system is mainly concerned with measurement of speed and accuracy, both of which significantly depend on the performance of the image-processing algorithm. Owing to their high sampling rate, high-speed sensors have strict requirements for motion-tracking algorithms on computing speed to satisfy the demand of real-time signal processing. Conventional motion extraction algorithms based on template matching registration techniques [i.e., sum of absolute difference (SAD) or normalized cross-correlation (NCC)] are mostly complex and have a heavy computation load. Moreover, template matching techniques can only achieve integer-pixel resolution because the minimal unit in a video image is 1 pixel. Such accuracy is far from satisfactory in numerous practical applications, particularly for those where the vibrations of small structures are required. Various methods have been proposed to refine measurement accuracy, including interpolation techniques and subpixel registration,^{36}37.^{–}^{38} most of which improved accuracy indeed but exhibited low computational efficiency to some degree.

Chan et al.^{39} proposed a subpixel motion estimation method that uses a combination of classical block matching and simplified optical flow. The method is tremendously faster than any existing block-matching algorithm because no interpolation is needed. In the first step, a block-matching algorithm, such as three-step search (TSS) or cross-diamond search, is used to determine integer-pixel displacement. The result is then refined through local approximation using a simplified optical flow to subpixel level. In this paper, we simplified Chan’s algorithm by replacing block matching with rounding-iterative Taylor approximation. With no subpixel interpolation needed during frame cutting in each iteration, this modified algorithm runs much faster than conventional iterative DIC optical flow. Given that the improvement brings no additional parameter that requires specification, the modified algorithm naturally executes with a high degree of automation. After several times of optimization, the computation time of one extraction in the modified algorithm is reduced to less than 1 ms. The modified algorithm is utilized in the high-speed camera system for its high efficiency and satisfactory subpixel accuracy. A simulation and two experiments under laboratory and realistic conditions are carried out for performance verification. The positive results demonstrated the accuracy and efficiency of the camera sensor system in measuring dynamic displacement.

The rest of the paper is organized as follows. Section 2 introduces the components and capability parameters of the high-speed vision sensor system. Section 3 presents the theory of motion estimation algorithm without interpolation and the modified Taylor algorithm. Section 4 evaluates the performance of the modified algorithm with a simulation test. Section 5 presents two experiments for performance verification. Section 6 discusses the results and outlook.

## 2.

## High-Speed Vision Sensor System

Traditional camera systems for displacement measurement are commonly composed of commercial digital cameras and personal computers. However, commercial digital cameras usually have low frame rates (i.e., 100 fps), which limit their application in vibration frequencies over 50 Hz. In this paper, a high-speed sensor system composed of a notebook computer (Intel Core processor 2.9 GHz, 2.75 GB RAM) and a video camera with telescopic lens is developed for displacement measurement, as shown in Fig. 1(a). The telescopic lens has a large zooming capability [Fig. 1(b)] that can reach the measurement requirement at different distances. The camera head uses a CCD sensor as the image receiver, which can capture 8-bit gray-scale images at a maximum of 1000 fps when the image resolution is set as $300\text{\hspace{0.17em}\hspace{0.17em}}\text{pixels}\times 300\text{\hspace{0.17em}\hspace{0.17em}}\text{pixels}$. A USB 3.0 interface is used to ensure stable data transfer between the camera and the computer. With its high sampling rate and computing efficiency, the image-processing software on the notebook computer can use the refined Taylor algorithm to track specific fast-moving objects and extract motion information in real time.

A target panel preinstalled on the target is very helpful to ensure extraction accuracy during measurement. If the target panel is unavailable because of the limitation of the measurement environment, the distinct surface patterns of the structure, such as textures or edges, can be used as tracking templates. Then the camera system is ready to capture images from a remote location, and the displacement time history of the structure can be obtained by applying the displacement tracking algorithm to the digital video images.

## 3.

## Motion Extraction Algorithm

## 3.1.

### Subpixel Motion Estimation Without Interpolation

Figure 2 shows the subpixel motion estimation algorithm combined with block-matching algorithm and simplified optical flow. Two consecutive frames, $f(x,y)$ and $g(x,y)$, with real displacement $(\mathrm{\Delta}x,\mathrm{\Delta}y)$ are given. The real displacement can be divided into an integer part $(\mathrm{\Delta}\overline{x},\mathrm{\Delta}\overline{y})$ and a subpixel part $(\delta x,\delta y)$ as

## (1)

$$\mathrm{\Delta}x=\mathrm{\Delta}\overline{x}+\delta x,\phantom{\rule[-0.0ex]{2em}{0.0ex}}\mathrm{\Delta}y=\mathrm{\Delta}\overline{y}+\delta y.$$A block-matching algorithm is first applied to estimate integer-pixel displacements $\mathrm{\Delta}\overline{x}$ and $\mathrm{\Delta}\overline{y}$. When the integer part is determined, the image block $f(x,y)$ is shifted by $\mathrm{\Delta}\overline{x}$ pixel in the $x$-direction and $\mathrm{\Delta}\overline{y}$ pixel in the $y$-direction.

For the subpixel part, the Taylor series approximation is used to refine the search. The shifted image $f(x+\mathrm{\Delta}\overline{x},y+\mathrm{\Delta}\overline{y})$ differs from the accurate location only by $|\delta x|<1$ and $|\delta y|<1$, which can be computed by using one-step Taylor approximation.

Total displacement can be determined by combining the integer part and the subpixel part. Analytical error analysis^{39}^{,}^{40} is deduced in one dimension and can be generalized straightforwardly to a 2-D situation. The results imply that this two-step method can extract more accurate motion vectors than other block-matching algorithms. With no requirement for any interpolation and motion-compensated frames, the algorithm is much faster than the conventional method.

## 3.2.

### Taylor Approximation with Rounding-Iterative Operation

In this part, an analytic model is built to illustrate the proposed modified algorithm used in the sensor system. Figure 3 illustrates the displacement extraction procedure from consecutive frames $k$ and $k+1$. A random subimage $f(x,y)$ in frame $k$ is selected as the matching template. With all its pixels moved by displacement vector $\stackrel{\rightharpoonup}{p}={(\mathrm{\Delta}x,\mathrm{\Delta}y)}^{\mathrm{T}}$, the template image will become a new subimage $g(x,y)$ in the next frame at the same position.

With the assumption of brightness constancy or intensity conservation, the relationship between the template images $f(x,y)$ and $g(x,y)$ at the same position in frame $k+1$ can be written as

Note that the surface radiance remaining fixed from one frame to the next rarely holds exactly. As the scene might be constrained with no specularities, object rotations, and secondary illumination (shadows or intersurface reflection), the brightness constancy assumption works well in practice.^{40}

Given the fact that the displacement vector $\stackrel{\rightharpoonup}{p}={(\mathrm{\Delta}x,\mathrm{\Delta}y)}^{\mathrm{T}}$ is usually a small value (normally several pixels), Eq. (2) can be approximated using first-order Taylor expansion with the higher-order terms ignored as follows:

## (3)

$$g(x,y)=f(x+\mathrm{\Delta}x,y+\mathrm{\Delta}y)\approx f(x,y)+\mathrm{\Delta}x\frac{\partial}{\partial x}f(x,y)+\mathrm{\Delta}y\frac{\partial}{\partial y}f(x,y).$$## (4)

$$E(\stackrel{\rightharpoonup}{p})=\sum _{x,y}{[g(x,y)-f(x,y)-\mathrm{\Delta}x\frac{\partial}{\partial x}f(x,y)-\mathrm{\Delta}y\frac{\partial}{\partial y}f(x,y)]}^{2}.$$## (5)

$$\frac{\partial E(\stackrel{\rightharpoonup}{p})}{\partial \mathrm{\Delta}x}=0,\phantom{\rule[-0.0ex]{2em}{0.0ex}}\frac{\partial E(\stackrel{\rightharpoonup}{p})}{\partial \mathrm{\Delta}y}=0.$$## (7)

$$\mathrm{\Delta}I=\left[\begin{array}{c}f({x}_{1},{y}_{1})-g({x}_{1},{y}_{1})\\ f({x}_{2},{y}_{2})-g({x}_{2},{y}_{2})\\ \vdots \\ f({x}_{n},{y}_{n})-g({x}_{n},{y}_{n})\end{array}\right],\phantom{\rule[-0.0ex]{2em}{0.0ex}}\nabla I=\left[\begin{array}{cc}{f}_{x}({x}_{1},{y}_{1})& {f}_{y}({x}_{1},{y}_{1})\\ {f}_{x}({x}_{2},{y}_{2})& {f}_{y}({x}_{2},{y}_{2})\\ \vdots & \vdots \\ {f}_{x}({x}_{n},{y}_{n})& {f}_{y}({x}_{n},{y}_{n})\end{array}\right],$$In the sense of general LS, if $\nabla {I}^{\mathrm{T}}\xb7\mathrm{\Delta}I$ is invertible (full rank), then the displacement vector can be expressed with an LS estimate as

## (8)

$$\stackrel{\rightharpoonup}{p}={[\nabla {I}^{\mathrm{T}}\xb7\mathrm{\Delta}I]}^{-1}\xb7\nabla {I}^{\mathrm{T}}\xb7\mathrm{\Delta}I.$$Figure 4 shows the architecture of the proposed displacement extraction method with the reformative iteration process involved. The procedure for the proposed modified method can be summarized as follows:

Step 1: Cut $f(x,y)$ and $g(x,y)$ from consecutive frames at the same position;

Step 2: Compute the partial derivatives ${f}_{x}$ and ${f}_{y}$ of $f(x,y)$ by the central difference;

Step 3: Compute the difference matrix $\mathrm{\Delta}I$ and gradient matrix $\nabla I$ according to Eq. (7);

Step 4: Compute the displacement vector ${\stackrel{\rightharpoonup}{p}}_{j}={(\mathrm{\Delta}{x}_{j},\mathrm{\Delta}{y}_{j})}^{\mathrm{T}}$ according to Eq. (8) and make sure ${\stackrel{\rightharpoonup}{p}}_{j}$ is less than 0.5. If ${\stackrel{\rightharpoonup}{p}}_{j}$ is less than 0.5, the algorithm proceeds to Step 5; if not, update $f(x,y)$ to the new $f[x+\text{round}(\mathrm{\Delta}{x}_{j}),y+\text{round}(\mathrm{\Delta}{y}_{j})]$ and return to Step 2 (Symbol round denotes rounding to the nearest integer);

Step 5: Accumulate $\mathrm{\Delta}{x}_{j}$ and $\mathrm{\Delta}{y}_{j}$ to obtain the refined displacement vector $\stackrel{\rightharpoonup}{p}={(\mathrm{\Delta}x,\mathrm{\Delta}y)}^{\mathrm{T}}$.

With this rounding-iterative modification, the integer-level motion estimation is also accomplished with optical flow instead of block matching. This modification of the original method is so simple because it does not introduce any unnecessary computation and pixel interpolation. The rounding-off operation eliminates the subpixel interpolation computation in each frame cutting, which makes the algorithm much faster than conventional iterative DIC optical flow. The algorithm naturally executes with a high degree of automation because the improvement brings no additional parameter that requires specification. Although the rounding-off operation significantly decreases the time consumed for subpixel interpolation, the modified algorithm may, to some extent, sacrifice accuracy for its relatively loose termination condition. Fortunately, the proposed method performs stable satisfactory subpixel results and executes with high efficiency in the following simulation and experiments. Thus, the algorithm can be used in high-speed camera systems to measure displacement in real time. The contrast simulation and experiments for validation are presented in the following sections.

## 4.

## Simulation Test

The performance of the proposed modified Taylor algorithm is first evaluated through a simulation test. The simulation gives a simple case with only one vignetting black circle (a diameter of 160 pixels) on white ground, as shown in Fig. 5. The black circle is programmed to rotate with the following ellipse equation:

## (9)

$$x(t)=10\text{\hspace{0.17em}}\mathrm{cos}(2\pi ft),\phantom{\rule[-0.0ex]{2em}{0.0ex}}y(t)=6\text{\hspace{0.17em}}\mathrm{sin}(2\pi ft).$$The maximum displacements in the $x$- and $y$-directions are 10 and 6 pixels, respectively. The rotation frequency is set to 1 Hz, and the sampling frequency is 50 Hz. Four algorithms, namely, classical NCC, UCC, TSS with optical flow, and the proposed modified Taylor, are applied to extract the motion displacement of the moving circle. All code programming and computing works are accomplished with MATLAB R2015a.

During the testing, an $80\times 80\text{\hspace{0.17em}\hspace{0.17em}}\text{pixels}$ region (within the red box) is selected as the tracking template for the stable tracking error.^{41} UCC algorithm is an advanced subpixel image registration technique that allows resolution adjusting by changing the up-sampling factor.^{34} The up-sampling factors for UCC algorithm are set as 1, 10, and 100 for subpixel levels of one integer pixel, 0.1 pixel, and 0.01 pixel, respectively. Meanwhile, the UCC algorithm cannot give accurate displacement results until the template size is large enough. The contrast test results on the same template condition are summarized in Table 1 with an asterisk.

## Table 1

Errors and time consumption comparisons in the simulation test.

Algorithm | Max error (pixel) | Absolute average error (pixel) | Ttotal (s) | Tavg (ms) | ||
---|---|---|---|---|---|---|

x | y | x | y | |||

Classical NCC | 0.6279 | 0.6231 | 0.2340 | 0.2421 | 4.46 | 17.77 |

UCC ( | 0.5710 | 0.3742 | 0.2309 | 0.2378 | 2.45 | 9.89 |

UCC ( | 0.1897 | 0.1520 | 0.0580 | 0.0491 | 3.51 | 14.03 |

UCC ( | 0.1497 | 0.1220 | 0.0548 | 0.0481 | 4.20 | 16.78 |

TSS + optical flow | 0.3383 | 0.3383 | 0.1151 | 0.1126 | 1.07 | 4.08 |

Modified Taylor | 0.3327 | 0.2553 | 0.0817 | 0.0681 | 0.15 | 0.46 |

Modified Taylor | 0.1590 | 0.1480 | 0.0531 | 0.0463 | 0.32 | 0.97 |

## a

With the same template size as the UCC algorithm.

Motion extraction results of two integer-level methods, namely, NCC and UCC ($\mathrm{usfac}=1$), are shown in Figs. 6(a) and 6(b). These two algorithms only scan the template’s best-matching region per pixel, and such a deficiency certainly leads to a step-type curve shape and reduces extraction accuracy. Results of subpixel level motion extraction are shown in Figs. 6(c)–6(f). The figures show that the motion curves with these four subpixel level algorithms are obviously smoother than the curves with NCC and UCC ($\mathrm{usfac}=1$).

Quantitive contrast results regarding tracking error and computation time are given in Table 1. The table indicates that with the improvement of subpixel resolution level from 1 to 0.01 pixel, the absolute average horizontal error of the UCC algorithm reduces from 0.2309 to 0.0548 pixel, and the absolute average vertical error reduces from 0.2378 to 0.0481 pixel. Meanwhile, the time consumed increases from 14.03 to $16.78\text{\hspace{0.17em}\hspace{0.17em}}\mathrm{ms}/\text{frame}$. When subjected to the requirement of template size, the UCC algorithm does not have the capability of giving dual attention to both accuracy and high efficiency. The error analysis of the combination of TSS and optical flow reveals that the combination method has a clear advantage in time consumed ($4.08\text{\hspace{0.17em}\hspace{0.17em}}\mathrm{ms}/\text{frame}$) with acceptable average errors (0.1151 pixel in the horizontal and 0.1126 pixel in the vertical). The proposed modified Taylor method is also observed to have the highest computation efficiency among all the listed methods. The average elapsed time of handling one frame costs only 0.46 ms with a relatively better error performance than the combined TSS and optical flow. Furthermore, with a large size template, the modified Taylor even gives a similar error performance as UCC ($\mathrm{usfac}=100$) with the elapsed time per frame of just $1/17$ of the latter.

Owing to the satisfactory performances in time efficiency and accuracy during displacement extraction in the simulation test, the modified Taylor algorithm was integrated into the real-time vision sensor system mentioned in Sec. 2. The software contains several modules. The high-speed camera module can control the parameters of the digital camera, such as contrast, brightness, and exposure time. The calibration part has the ability to compute the actual displacement of one pixel based on a target with an already known size. With the image-capturing part, the streaming image data can be acquired in real time and sent to the template tracking module, where the modified Taylor algorithm is operating. The entire sensor system is implemented based on the Qt and OpenCV libraries and is capable of realizing the displacement measurement of actual structures.

## 5.

## Experimental Verification

## 5.1.

### Case 1: Experiment on a Grating Ruler Motion Platform

To evaluate the performance of the developed vision-based sensor system, an experimental verification was carried out on a laboratory platform with a conventional grating ruler, as shown in Fig. 7. Using the Moiré fringe technology of grating and photoelectric conversion, the incremental grating displacement sensors widely act as a high-accuracy displacement measurement tool with numerous advantages, such as stability, reliability, and high accuracy. The experimental installations are shown in Fig. 7(a). The grating ruler displacement sensor was installed on the moving table platform, with its reading moving synchronously with the junction plate in the horizontal direction. With this structure, the displacement of the target can be recorded simultaneously by the high-speed sensor system and the grating ruler for comparison. The sampling frequency of the grating ruler sensor used in the experiment is 20 Hz, and the grating pitch is 0.02 mm with a resolution of $1\text{\hspace{0.17em}\hspace{0.17em}}\mu \mathrm{m}$.

In the experiment, the vision-based high-speed camera system was experimentally evaluated against the grating ruler. As seen in Fig. 7(a), a circle target with a diameter of 20 mm was installed on the junction plate in advance. The target can be programmed to move with arbitrary amplitudes and frequencies in the horizontal direction. The video camera was placed at a stationary position 3 m away from the platform. The camera captured the moving target at a resolution of $160\times 160\text{\hspace{0.17em}\hspace{0.17em}}\text{pixels}$ with 200 fps during the shooting process. To measure the displacement in real-time dimensions, the actual size of the preinstalled target panel in the video images was calculated. The results showed that 20 mm in real life corresponds to 104.7 pixels in the captured images, which means the pixel resolution would be $0.191\text{\hspace{0.17em}\hspace{0.17em}}\mathrm{mm}/\text{pixel}$. A $50\times 50\text{\hspace{0.17em}\hspace{0.17em}}\text{pixels}$ region on the target, as shown in Fig. 7(b), was chosen as the template matching image.

The guide screw was driven by a 10 s manual arbitrary input. As shown in Fig. 8, the horizontal displacement time history measured by the vision-based system was compared with that measured by the grating ruler sensor. The grating ruler data (green dashed line) matched well with the vision-based sensor data (dashed blue line). The integer-level tracking result is marked with a red solid line. The step-type result indicates that the integer-level algorithms can only acquire the integer-pixel motion of the target, which leads to a large measurement error.

Similar to the simulation test, the captured video was analyzed by the different motion extraction algorithms previously mentioned. Quantitive experimental results regarding the tracking error and computation time are given in Table 2. To further evaluate the error performance, the normalized root mean squared error (NRMSE) is introduced as

## (10)

$$\mathrm{NRMSE}=\frac{\sqrt{\frac{1}{n}\sum _{i=1}^{n}{({a}_{i}-{b}_{i})}^{2}}}{{b}_{\mathrm{max}}-{b}_{\mathrm{min}}}\times 100\%,$$## Table 2

Errors and time consumption comparisons in grating ruler motion platform experiment.

Algorithm | Erroravg (mm) | NRMSE (%) | Tavg (ms) |
---|---|---|---|

Classical NCC | 0.051 | — | 5.93 |

UCC ( | 0.058 | — | 3.78 |

UCC ( | 0.028 | 1.08 | 13.09 |

UCC ( | 0.022 | 0.87 | 15.35 |

TSS + optical flow | 0.025 | 1.01 | 2.54 |

Modified Taylor | 0.020 | 0.75 | 0.22 |

Modified Taylor | 0.019 | 0.73 | 0.45 |

## a

With the same template size as the UCC algorithm.

## 5.2.

### Case 2: Vibration Measurement of Steering Wheel System

To validate the effectiveness of the proposed sensor system in a practical environment, a vibration measure experiment on a forklift’s steering wheel system was conducted. The steering wheel system consists of several components, including front panel, install panel, mounting plate, commutator pump, steering wheel, and tubular column. As shown in Fig. 9(a), the steering wheel and tubular column are assembled using an interference fit, and the mounting plate is connected with the commutator pump using a locking device. The mounting plate and front panel are both welded to the install plate. Owing to the design defects, resonance exists on the steering wheel system when the engine is working at idle speed (22.8 to 28.3 Hz). This resonance may lead to physical complaints and industrial accidents if drivers operate the forklift for a long drive.

A finite element model (FEM) of the steering wheel system was built with Pro/Engineer, as shown in Fig. 9(b). The locking device between commutator pump and mounting plate was simplified into a bolted connection in the FEM modeling. Figure 9 also illustrates the grid results using a hexahedral mesh. Modal test results proved that a first-order natural frequency occurs at 22.3262 Hz. From Fig. 10, the vibration mode of this frequency is shown as a horizontal steering wheel bending. The FEM analysis confirmed the resonance speculation because the natural frequency is apparently close to the resonance frequency range.

The vision-based experimental setup on the forklift’s steering wheel system is shown in Fig. 11. The high-speed camera sensor was installed on a special support to avoid additional interference. Measurement targets with a $10\text{\hspace{0.17em}\hspace{0.17em}}\mathrm{mm}\times 10\text{\hspace{0.17em}\hspace{0.17em}}\mathrm{mm}$ size were marked on the upper surface of the steering wheel. The distance between the camera and the steering wheel was about 60 cm. The actual size of one pixel was $0.0375\text{\hspace{0.17em}\hspace{0.17em}}\mathrm{mm}/\text{pixel}$, which was calculated by using the known physical size of the targets. With the efficient modified Taylor algorithm, the vibration can be analyzed and displayed in real time.

The horizontal and vertical vibration displacements with their corresponding Fourier spectra after the modified Taylor algorithm were applied to the vibration video, as shown in Fig. 12. The results show that the center of the steering wheel vibrates with an amplitude under 0.5 mm after excitation. Two obvious spectral peaks can be observed at 22.27 and 44.24 Hz in the Fourier spectrum results; these peaks can be considered as the first-order master frequency and its double frequency of the steering wheel system. During the motion extraction process, the elapsed time for each frame was less than 0.4 ms, and more than 87% extractions were completed within 0.1 ms. The results are very close to the natural frequency obtained with FEM analysis with an acceptable error. Therefore, the same frequency can be obtained accurately from the proposed vision-based displacement sensor.

## 6.

## Conclusions

This study developed a vision-based high-speed sensor system for dynamic displacement measurement. The sensor system is composed of a high-speed camera head with a zoom optical lens and a notebook computer. To meet the requirement of real-time measurement, a motion extraction algorithm with high efficiency is used. With the combination of block-matching algorithm and simplified optical flow, motion vectors between frames can be extracted accurately. The method is proven to be much faster than conventional algorithms because there is no interpolation or motion compensation. However, this combination method still has room for improvement.

In our proposed algorithm, the integer-pixel searching is replaced with a rounding-iterative operation on Taylor approximation. This simple modification does not bring any unnecessary computation or pixel interpolation to the original method. By benefiting from no additional parameter requiring specification, the modified algorithm can execute with high automation and even faster with better accuracy. Based on the assumption of brightness constancy or intensity conservation, the proposed algorithm obtains the displacement vector between frames in the sense of LSs and achieves fast automatic computation by iteratively updating the template’s position. Without the image feature extraction process, the algorithm simplifies the selection of thresholds and is completed through a simple matrix operation. The properties of high efficiency, high precision, and good robustness of the proposed algorithm itself contribute to the applications of the high-speed camera sensor system.

A simulation on the tracking rotation motion of a black circle as well as two experiments on a grating ruler motion platform and vibration analysis of steering wheel system are conducted to verify the effectiveness of the modified algorithm and developed sensor system. The results of displacement extraction using the modified algorithm are compared with the actual values and the results of three other existing extraction algorithms. From the simulation test, a satisfactory agreement is observed between the real motion curve and the curve obtained through the modified algorithm. In the grating ruler motion platform experiment, the motion of the grating ruler platform is accurately measured using the developed sensor system. In a realistic environment, the performance of the vision sensor is further confirmed by the vibration analysis of the forklift’s steering wheel system. Of all the simulation and experiments, the modified algorithm shows its outperformance on computing efficiency. The average elapsed time of handling one frame can be reduced to less than 1 ms with an impressive measurement error.

Although the brightness constancy assumption works well in practice, the large vibration on illumination intensity may still influence the measurement results and lead to large errors. Different from the improvement through multiframe,^{39} the modified algorithm acquires the image basis by handling only one frame. This characteristic makes the method concise and highly effective, but the differential operation may amplify the image noise and cause an undesired error. The developed sensor system can only meet the real-time measurement under a frequency sampling below 500 Hz, which is limited by the camera module we can access. Future work will be focused on improving the algorithm’s robustness under large illumination changes and developing a sensor system for high frequency sampling over 500 Hz.

## Acknowledgments

This work was supported by the Key Project of Natural Science by Education Department of Anhui Province (No. KJ2015A316) and the Outstanding Young Talents at Home Visit the School Training Project (No. gxfxZD2016101).

## References

Y. Zhang et al., “Application of the Fourier transform in electronic speckle photography,” Exp. Mech. 42, 409–416 (2002).EXMCAZ0014-4851http://dx.doi.org/10.1007/BF02412146Google Scholar

J. L. Valin et al., “Methodology for analysis of displacement using digital holography,” Opt. Laser Technol. 43, 99–111 (2005).OLTCAS0030-3992http://dx.doi.org/10.1016/j.optlaseng.2004.05.010Google Scholar

H. H. Nassif et al., “Comparison of laser Doppler vibrometer with contact sensors for monitoring bridge deflection and vibration,” NDT&E Int. 38, 213–218 (2005).http://dx.doi.org/10.1016/j.ndteint.2004.06.012Google Scholar

Y. Ji and C. Chang, “Nontarget stereo vision technique for spatiotemporal response measurement of line-like structure,” J. Eng. Mech. 134, 466–474 (2008).JENMDT0733-9399http://dx.doi.org/10.1061/(ASCE)0733-9399(2008)134:6(466)Google Scholar

D. L. B. R. Jurjo et al., “Experimental methodology for the dynamic analysis of slender structures based on digital image processing techniques,” Mech. Syst. Signal Process. 20, 1112–1133 (2006).http://dx.doi.org/10.1016/j.ymssp.2004.09.008Google Scholar

T. Wu et al., “Full-life dynamic identification of wear state based on on-line wear debris image features,” Mech. Syst. Signal Process. 42, 404–414 (2014).http://dx.doi.org/10.1016/j.ymssp.2013.08.032Google Scholar

Y. V. Filatov et al., “Noncontact measurement of angular position and angular movement by means of laser goniometer,” Opt. Eng. 54(5), 054103 (2015).http://dx.doi.org/10.1117/1.OE.54.5.054103Google Scholar

S. W. Park et al., “3D displacement measurement model for health monitoring of structures using a motion capture system,” Measurement 59, 352–362 (2015).0263-2241http://dx.doi.org/10.1016/j.measurement.2014.09.063Google Scholar

Y. Arai, “Development of in-plane and out-of-plane deformations simultaneous measurement method for the analysis of buckling,” Opt. Eng. 54(2), 024102 (2015).http://dx.doi.org/10.1117/1.OE.54.2.024102Google Scholar

P. J. Figueroa, N. J. Leite and R. M. L. Barros, “Tracking soccer players aiming their kinematical motion analysis,” Comput. Vis. Image Understanding 101, 122–135 (2006).CVIUF41077-3142http://dx.doi.org/10.1016/j.cviu.2005.07.006Google Scholar

R. Aharoni et al., “Real-time stand-off spatial detection and identification of gases and vapor using external-cavity quantum cascade laser open-path spectrometer,” Opt. Eng. 54(6), 067103 (2015).http://dx.doi.org/10.1117/1.OE.54.6.067103Google Scholar

F. Cheli et al., “Vision-based measuring system for rider’s pose estimation during motorcycle riding,” Mech. Syst. Signal Process. 38, 399–410 (2013).http://dx.doi.org/10.1016/j.ymssp.2013.01.009Google Scholar

Y. Shao, Y. Guo and C. Gao, “Human action recognition using motion energy template,” Opt. Eng. 54(6), 063107 (2015).http://dx.doi.org/10.1117/1.OE.54.6.063107Google Scholar

F. C. Trigo et al., “Identification of a scaled-model riser dynamics through a combined computer vision and adaptive Kalman filter approach,” Mech. Syst. Signal Process. 43, 124–140 (2014).http://dx.doi.org/10.1016/j.ymssp.2013.10.005Google Scholar

J. Guo, “Dynamic displacement measurement of large scale structures based on the Lucas–Kanade template tracking algorithm,” Mech. Syst. Signal Process. 66, 425–436 (2015).http://dx.doi.org/10.1016/j.ymssp.2015.06.004Google Scholar

Y. Song et al., “Virtual visual sensors and their application in structural health monitoring,” Struct. Health Monit. An Int. J. 13, 251–264 (2014).http://dx.doi.org/10.1177/1475921714522841Google Scholar

U. Yang et al., “Illumination-invariant color space and its application to skin-color detection,” Opt. Eng. 49(10), 107004 (2010).http://dx.doi.org/10.1117/1.3497058Google Scholar

J. J. Lee, H. N. Ho and J. H. Lee, “A vision-based dynamic rotational angle measurement system for large civil structures,” Sensors 12, 7326–7336 (2012).SNSRES0746-9462http://dx.doi.org/10.3390/s120607326Google Scholar

H. S. Park, “A new position measurement system using a motion-capture camera for wind tunnel tests,” Sensors 13, 12329–12344 (2013).SNSRES0746-9462http://dx.doi.org/10.3390/s130912329Google Scholar

J. Sadek et al., “Development of a vision based deflection measurement system and its accuracy assessment,” Measurement 46, 1237–1249 (2013).0263-2241http://dx.doi.org/10.1016/j.measurement.2012.10.021Google Scholar

J. Morlier and G. Michon, “Virtual vibration measurement using KLT motion tracking algorithm,” J. Dyn. Syst. Meas. Control 132, 011003 (2010).JDSMAA0022-0434http://dx.doi.org/10.1115/1.4000070Google Scholar

B. Ko and S. Kwak, “Survey of computer vision-based natural disaster warning systems machine vision and applications,” Opt. Eng. 51(7), 070901 (2012).http://dx.doi.org/10.1117/1.OE.51.7.070901Google Scholar

H. Wang et al., “Vision-based vehicle detection and tracking algorithm design,” Opt. Eng. 48(2), 127201 (2009).http://dx.doi.org/10.1117/1.3269685Google Scholar

A. Jaume-i-Capo et al., “Automatic human body modeling for vision-based motion capture system using B-spline parameterization of the silhouette,” Opt. Eng. 51(2), 020501 (2012).http://dx.doi.org/10.1117/1.OE.51.2.020501Google Scholar

C. Quan et al., “Determination of three-dimensional displacement using two-dimensional digital image correlation,” Appl. Opt. 47, 583–593 (2008).APOPAI0003-6935http://dx.doi.org/10.1364/AO.47.000583Google Scholar

S. W. Kim et al., “Vision based monitoring system for evaluating cable tensile forces on a cable-stayed bridge,” Struct. Health Monit. An Int. J. 12, 440–456 (2013).http://dx.doi.org/10.1177/1475921713500513Google Scholar

E. S. Bell, J. T. Peddle and A. Goudreau, “Bridge condition assessment using digital image correlation and structural modeling,” in 6th Int. Conf. on Bridge Maintenance, Safety and Management, Stresa, Italy, pp. 330–337 (2012).Google Scholar

W. Tong, “Formulation of Lucas–Kanade digital image correlation algorithms for noncontact deformation measurements: a review,” Strain 49, 313–334 (2013).http://dx.doi.org/10.1111/str.12039Google Scholar

J. W. Park et al., “Vision based displacement measurement method for high-rise building structures using partitioning approach,” NDT&E Int. 43, 642–647 (2010).http://dx.doi.org/10.1016/j.ndteint.2010.06.009Google Scholar

A. M. Wahbeh et al., “A vision-based approach for the direct measurement of displacements in vibrating systems,” Smart Mater. Struct. 12, 785–794 (2003).SMSTER0964-1726http://dx.doi.org/10.1088/0964-1726/12/5/016Google Scholar

Y. Fukuda et al., “Vision-based displacement sensor for monitoring dynamic response using robust object search algorithm,” IEEE Sensors J. 13, 4725–4732 (2013).ISJEAZ1530-437Xhttp://dx.doi.org/10.1109/JSEN.2013.2273309Google Scholar

D. Feng et al., “A vision based sensor for noncontact structural displacement measurement,” Sensors 15, 16557–16575 (2015).SNSRES0746-9462http://dx.doi.org/10.3390/s150716557Google Scholar

M. Guizar-Sicairos, S. T. Thurman and J. R. Fienup, “Efficient subpixel image registration algorithms,” Opt. Lett. 33, 156–158 (2008).OPLEDP0146-9592http://dx.doi.org/10.1364/OL.33.000156Google Scholar

J. J. Lee and M. Shinozuka, “Real-time displacement measurement of a flexible bridge using digital image processing techniques,” Exp. Mech. 46, 105–114 (2006).EXMCAZ0014-4851http://dx.doi.org/10.1007/s11340-006-6124-2Google Scholar

D. You, X. Gao and S. Katayama, “Monitoring of high power laser welding using high-speed photographing and image processing,” Mech. Syst. Signal Process. 49, 39–52 (2014).http://dx.doi.org/10.1016/j.ymssp.2013.10.024Google Scholar

P. Bing et al., “Performance of sub-pixel registration algorithms in digital image correlation,” Meas. Sci. Technol. 17, 1615–1621 (2006).MSTCEP0957-0233http://dx.doi.org/10.1088/0957-0233/17/6/045Google Scholar

Z. Zhang and R. Wang, “Robust image super resolution method to handle localized motion outliers,” Opt. Eng. 48(7), 077005 (2009).http://dx.doi.org/10.1117/1.3159871Google Scholar

L. Li et al., “Subpixel flood inundation mapping from multispectral remotely sensed images based on discrete particle swarm optimization,” J. Photogramm. Remote Sens. 101, 10–21 (2015).http://dx.doi.org/10.1016/j.isprsjprs.2014.11.006Google Scholar

S. Chan et al., “Subpixel motion estimation without interpolation,” in 2010 IEEE Int. Conf. on Acoustics, Speech and Signal Processing, pp. 722–725 (2010).http://dx.doi.org/10.1109/ICASSP.2010.5495054Google Scholar

.D. Fleet and Y. Weiss, Handbook of Mathematical Models in Computer Vision, pp. 239–256, Springer, New York (2006).Google Scholar

X. Lei et al., “Vibration extraction based on fast NCC algorithm and high-speed camera,” Appl. Opt. 54, 8198–8206 (2015).APOPAI0003-6935http://dx.doi.org/10.1364/AO.54.008198Google Scholar

## Biography

**Bingyou Liu** received his BS and MS degrees in detection technology and automation devices from Anhui Polytechnic University in 2003 and 2008, respectively. He is an associate professor at Anhui Polytechnic University. He has authored 15 journal papers. His current research interests include optoelectronic systems, intelligent control, and weak signal detection.

**Dashan Zhang** received his BS degree in mechanical engineering from Guizhou University in 2012. Currently, he is a PhD candidate in the Department of Precision Machinery and Precision Instrumentation at the University of Science and Technology of China. His research interests include image processing and optical measurement.

**Jie Guo** received his BS and PhD degrees in mechanical engineering from the University of Science and Technology of China, Hefei, China, in 2010 and 2015, respectively. Currently, he is a postdoctoral fellow with the Department of Precision Machinery and Precision Instrumentation, University of Science and Technology of China. His current research interests include machine vision, pattern recognition, machine learning, and fault diagnosis.

**Chang’an Zhu** received his BS degree from HeFei University of Technology in 1982, his MS degree from Xidian University in 1985, and his PhD from National University of Defense Technology in 1989. He is a professor at the University of Science and Technology of China. His current research interests include intelligent control, fault diagnosis technology, and advanced manufacturing technology.