2.1.

Deviation Extraction from the Edge Profile

The deviation extraction process is derived as follows. Consider an apparent simple straight-line edge feature, which always exists in captured subimage $I(x,y,t)$ in timeline $t$ and has a subtle deviation signal $f(x,t)$ at every location $x$ along the straight line. The edge profile at location $x$ can be defined as follows:

Assuming the deviation signal $f(x,t)=0$ for an arbitrary $x$ value, the edge profile $E_x(y,t)$ will remain constant and is rewritten as follows:

As the deviation $f(x,t)$ is considered to be subtle in reality, $E(y,t)$ is estimated by using the mean value of all available edge profiles in $I(x,y,t)$. Therefore, subimage $I(x,y,t)$ is then expressed as the sum of the edge profiles and independent image noises $n(x,y,t)$:

Considering that $f(x,t)$ is small, subimage $I(x,y,t)$ can be approximated with a first-order Taylor expansion when higher components are ignored:

The average pixel value over $x$ is as follows:

Accordingly, the average edge profile approximates the common edge profile up to a constant shift. The item $\frac{1}{N}\sum _{x}f(x,t)$ can be regarded as a global translation of $f(x,t)$ and is removed with a follow-up bandpass filter. Therefore, using the translated equation properly approximates the original edge profiles $E(y,t)$, and least square estimation method calculates the deviation signal $f(x,t)$:

An experiment is performed to demonstrate the effectiveness of the deviation extraction algorithm. Figure 1 shows a carbon tool steel saw blade fixed on a table vice captured with a Canon 70D SLR Camera, and the shape of its jagged edge on the right side is difficult to identify with the naked eye in the original image. To lessen the calculation burden, areas within the yellow rectangle is selected as analysis area for further operations. Our ultimate target is to extract the subtle deviation along the jagged edge.

Fig. 1

Experimental setups of the saw blade experiment.

Figure 2 presents the schematic and extraction results of the experiment. Figure 2(a) shows that the RGB image is first transformed into grayscale, and the jagged edge within the red rectangle region of interest (RoI) is fitted ideally through line segment detector algorithm and marked with a blue line. The size of the RoI is $480\times 50\text{pixels}$ and green arrows in Fig. 2(a) show the orthogonal relationship between the blue fitted line and sampling direction. The detected straight-line segments usually require advance rotation for the convenience of sampling edge profiles because edge profiles need to be sampled strictly perpendicular to identified line segments and fitted line segments are not always idealized horizontally or vertically in an actual situation. In this particular case, the sampled rectangle is rotated clockwise at 3.2743 deg to become horizontal through bilinear interpolation method. Figure 2(b) shows the output deviation result without any denoising approach. An unnoticeable subtle jagged shape is extracted successfully from the saw blade. The extracted deviation signal has undergone a cyclical change with a maximum amplitude of no more than three pixels. Benefitting from the extreme short sampling interval of high-speed imaging, the global translation items between adjacent frames are negligibly small in the high-speed video, and the relative vibration signal perpendicular to the fitted line is presented as follows:

Fig. 2

(a) Schematic and (b) extraction results of the saw blade experiment.

Sampling processes will be more complicated than approximate line segments for other simple curved boundaries, such as circular and ellipse edges. These kinds of edges need to be idealized and fitted using methods, such as Hough transformation, to obtain their outlines and center coordinates. With the fitted curve boundaries and their center coordinate information, the RoIs are generated and unrolled to satisfy the orthogonal sampling requirement based on their polar equations:

2.2.

Discussion of Textured Edge Profile

Deviation extraction algorithm displays stable performance when the constant assumption of tested edge profile is satisfied. In the experiment above, the idealized edge profile of the tested saw blade is clear and constant. However, the assumption of a constant edge profile may be invalid with textured contours. The textured edge profile can be determined when the edge of the idealized line-segment is not clear due to pixel variations. In practical measurement, the textured part near structural contour can invalidate the assumption of a constant edge profile and influence the deviation extraction result. Thus, an image colorization optimization method is applied to remove unwanted variations and avoid unpredictable measuring error due to texture.

The colorization optimization algorithm works in YUV color space, where Y refers to the monochromatic luminance channel related to pixel intensity, and U and V are the chrominance channels that encode the color. The transformation between RGB and YUV color spaces is as follows:

For a given Y channel, this algorithm assumes that two neighboring pixels should be similar in chrominance channels if they have similar intensities. As U and V are relatively independent, the values in channel U can be estimated by minimizing

In this section, a simulation test is shown to demonstrate the effect of the denoising process. Figure 3 shows that the sand dune boundaries within the red rectangular region are clear straight-line contours combined with subtle fluctuations. The employed idealized line segment is marked with blue lines in the clean and textured images. A series of color stripes are painted manually along the blue line’s contour to create noise textures.

Fig. 3

Clean and textured sand dune straight-line boundaries.

Figure 4 compares the sampling conditions of the clean and created textured images. The selected subimage is rotated counterclockwise at 28.1093 deg before sampling, and the size of sampling area is $480\times 18\text{pixels}$. The constraint condition of colorization optimization is obtained by marking the black and white colors two pixels away from the fitted blue line. Figure 4(b) shows the ellipse regions with textures along the edge that lead to obvious noises and will finally result in deviation extraction errors. Figure 4(c) shows the sampling area, where the denoising algorithm was conducted and most of the noise textures were removed successfully.

Fig. 4

Sampling results of the (a) clean image, (b) textured image, and (c) textured image with denoising.

Figure 5 presents the comparisons of the edge profiles when $x=300$. The curves of the clean and textured images indicate that textures along the edge will cause intensities on both sides of the fitted line blur into each other and lower their difference, causing the contour to become unclear and the extracted deviation signal is contaminated. Colorization optimization process has the opposite effect and can remove variations due to textures. Consequently, the constant edge profile assumption is satisfied.

Fig. 5

Edge profile results ($x=300$) in the simulation test.

Figure 6 shows the extracted deviation signals, and Table 1 presents their quantitative analysis results. Simulation data are processed using MATLAB R2018a on a machine with a single 4.00 GHz processor (Intel Core i7-6700k) and 16 GB RAM without overclocking. The deviation signal extracted from the textured image after denoising is consistent with the clean image and their correlation coefficient is 0.9986. However, elapsed time also increases because of the colorization optimization process. In practical measurement, avoiding textured areas along the contours is better.

Fig. 6

Extracted deviation signals in the simulation test.

Table 1

Quantitative analysis of the extracted deviation signals in the simulation test.

2.3.

SVD-Based Variation Extraction

Relative deviations of adjacent frames in the video can be calculated using the steps discussed above. Global translations are reconstructed into the following analysis matrix formed with subtle variations of the edge profile:

The SVD is a factorization of a matrix in linear algebra, which is broadly applied in signal processing and statistics. For a given real matrix $\mathbf{G}\in R_{(t-1)\times N}$, the SVD decomposes the matrix in the following form:

SVD theory states that vectors in matrices $\mathbf{U}$ and $\mathbf{V}$ are orthonormal to one another in each vector group and form orthonormal bases of ($t-1$)-dimensional and $N$-dimensional spaces, respectively. Decomposition breaks a matrix into pieces based on the descending order of the singular values. Given the magnitudes of singular values indicate the energy distribution of the decomposed pieces, matrix $\mathbf{G}$ can be approximated by reserving former $k$ singular values ${\sigma }_i(i=\mathrm{1,2},\dots ,k)(k<q)$. The low-rank approximation of matrix $\mathbf{G}$ takes the form of

Given the corresponding singular values of noises are normally small, the summation of former $k$ elements is recreated as final vibrations. In real life, the signal extraction process utilizes the data compression function of SVD. Figure 7 shows the flowchart of the overall variation extraction process.

Fig. 7

Flowchart of the variation extraction process.

3.1.

Sound-Induced Subtle Vibration Analysis

Structural modal parameters, such as resonant frequencies and elasticity (Young’s modulus), are important evaluating indicators in structural safety inspection. For lightweight structures, the added mass introduced by traditional contact sensors will inevitably affect the final modal test result. In the first experiment, the proposed approach was applied to estimate the material properties of the lightweight clamped saw blade from subtle motions in the video.

Figure 8 shows the experimental setup. The lightweight saw blade shown in Fig. 1 was clamped using a table vice and then excited by a linear ramp of frequency airwaves. The dimensions of the clamped saw blade are $0.29\times 0.0126\times 0.00065\mathrm{m}$, and Young’s modulus and density are $2.06\times {10}^{11}\mathrm{N}·{\mathrm{m}}^{-2}$ and $7.85\times {10}^3\mathrm{kg}·{\mathrm{m}}^{-3}$, respectively. An audio file with a frequency band from 10 to 500 Hz was played by a loudspeaker from $\sim 0.1\mathrm{m}$ away from the clamped saw blade. The volume of the sound was set to 80 dB. Figure 9 presents the waveform and short-time Fourier transform spectrogram of the input excitation sound. When air fluctuations hit the saw blade, these fluctuations may cause small forced vibrations on the surface of the vibrating object. Vibrations caused by the fluctuations were found to be in the order of one hundredth to one thousandth of a pixel using a phase-based optical flow. Small vibrations motivated by the excitation sound signal were recorded by the high-speed camera system (Mode-5KF10M, Agile Device, Inc.) at 500 fps with a resolution of $580\times 180\text{pixels}$. A USB 3.0 interface ensured data transmission between high-speed camera and computer, and a LED light source was applied to provide adequate illumination.

Fig. 8

Setups of the beam property estimation experiment.

Fig. 9

(a) Waveform and (b) spectrogram of the input excitation signal.

Two analysis areas were selected within the blue and red boxes in Fig. 8 to calculate the deviations of the contours. The image size for both areas was $350\times 30\text{pixels}$, and the smooth side of the saw blade was located on the right of the blue rectangle. Considering the acceptable clean boundary condition in this case, the analysis areas in the first experiment were not subjected to image colorization optimization. SVD decomposition result indicated signals corresponding to the first two singular values occupy over 97% of energy in the analysis matrix. Thus, the rest of the components were considered noises and ignored in our final data. The average elapsed time per frame was $\sim 0.04\mathrm{s}$.

Figure 10 shows the waveforms and frequency spectrum results. Four peaks were observed at 6.37, 40.16, 113.10, and 221.60 Hz in their spectrograms. These peaks can be considered the first four resonant frequencies of the clamped saw blade in the modal test. Note the first peak 6.37 Hz is smaller than the lowest frequency (10 Hz) of the input excitation sound because of the presence of signal components below 10 Hz and the relatively high sensitivity of the lower modes.

Fig. 10

(a)–(d) Vibration extraction results in the clamped beam experiment.

The clamped saw blade could be considered as a cantilever beam, the theoretical resonant frequencies are estimated according to Euler–Bernoulli beam theory as follows:

Table 2

Comparisons of theoretical and extracted resonant frequencies.

3.2.

Vibration Analysis of Sound Barrier

The sound barrier is a type of platy structure installed on both sides of the railway to protect inhabitants from noise pollution. The sound barrier may suffer from strong suction caused by high-speed trains passing by in practical working conditions. The vibrations induced by the wind may lead to structural damage and decrease the working life of sound barriers along high-speed railways. These structures are usually installed on suspension viaducts and contact sensors are difficult to install. Vision-based noncontact method is a suitable alternative to analyze their dynamic responses when a train passes by.

Figure 11 shows the setup of the second experiment. Field test was conducted near KunShan railway station, which is an important railway hub in East China. The high-speed camera system (Mode-5KF04M, Agile Device, Inc.) was placed below the viaduct in a distance. The height of the railway bridge is around 12 m, and the height of the sound barrier is 2.15 m. The distance between the camera head and the pier is $\sim 28\mathrm{m}$, and the distance between the camera head and sound barrier was estimated at 30 m. Measurement error rate caused by the camera tilt angle (about 26 deg) is $\sim 0.6\%$ to 0.8%,³³ which is considered acceptable. Hence, the errors have limited influence on frequency information analysis. The frame rate was set to 232 fps, and the resolution of the image was $660\times 1880\text{pixels}$ during the experiment. Figure 11(c) presents selected areas for deviation extraction in which a clean straight-line boundary within the blue box and a textured boundary within the red box. The dimension of the subimages was $120\times 500\text{pixels}$. Colorization optimization algorithm before deviation extraction process was conducted on the textured subimages.

Fig. 11

Sound barrier experiment setups on (a) the experiment environment, (b) experimental devices and (c) video image and selected analysis areas.

Figure 12 shows the final extracted waveforms and their frequency spectrum results. SVD decomposition results indicated that signals corresponding to the first three singular values occupy over 95% of energy in the analysis matrix. Thus, the other components were regarded as noises and ignored in our final data. Three obvious peaks—10.42, 21.07, and 45.77 Hz—were found in their Fourier spectrums. The textured edges are closer to the surface of bridge, and the moment of the train arrival is determined in the signal from the clean edge deviations and unclear in the signal from the textured edge deviations. Studies have indicated that the natural frequencies of the railway bridge are relatively low and rarely found from train-induced dynamic responses, wherein these extracted vibrations are considered dominated by excitation frequencies associated with the passing of the high-speed train.^34,35 The three peaks can be regarded as excited by the pulsed wind from the train because the excitation frequency of the wind load caused by the train carriages ranges from $\sim 2$ to 4 Hz, and the natural frequencies of the tested sound barrier (less than 5 Hz) are far from the excitation frequency of the train. These three peaks can be regarded excited by the pulsed wind from train locomotive.³⁶ The three peaks can be considered the characteristic frequencies of the sound barrier.

Fig. 12

(a)–(d) Vibration extraction results in the sound barrier experiment.