2.1.

Architecture of U-Net Network Used in Our Method

Our network architecture, which is based on the U-Net CNN, consists of a decomposition path (encoder) and a reconstruction path (decoder) as shown in Fig. 1. The advantage of this architecture is that the deeper features of an image can be obtained by the decomposition path and the reconstruction path gradually recovers the image. The decomposition path has four convolutional blocks, which increase the number of feature maps from 1 to 512. Every block has two convolutional layers with the filter sizes of $3\times 3\text{pixels}$ and the convolution stride is fixed to 1 pixel. Every convolutional layer is equipped with the rectified linear unit (ReLU), which is used to resolve the vanishing gradient problem. After every block in the decomposition path, a $2\times 2\text{pixels}$ max pooling with stride 2 is used to reduce the dimensionality, so the size of the feature maps decreases from $512\times 512$ to $64\times 64$. In the reconstruction path, every block starts with a $3\times 3$ up-convolution with stride 2. After the up-convolution step, the size of the feature maps is increased, but the number of the feature maps is halved. Then a merge operation is performed by concatenating the feature maps into the correspondingly feature maps that from the decomposition path. In the final layer, a $1\times 1$ convolution is used to reduce the number of the feature maps to 2. Different from the original U-Net architecture, we use zero padding so that the output dimension does not decrease after each convolution. As a result, the final output of the U-Net network produces a pixel-wise probability map. In order to reduce over-fitting, there is a dropout layer after each convolutional layer in the network. The dropout is to randomly drop units (along with their connections) from the neural network during training. The dropout layer makes the network independent of some local features and prevents the parameters over-fitting. Using dropout, the generalization performance and robustness of the network are improved.

Fig. 1

Structure of the proposed deep convolutional network for fringe skeleton detection and extraction.

2.2.

Training Process of Our Method

The training process can be divided into two parts, the forward propagation and the backpropagation. The flowchart of the training process is shown in Fig. 2.

Fig. 2

Flowchart of the training process.

In the forward propagation, an input ESPI image is fed to the U-Net network. In the network, we let symbols $W^l$ and $B^l$ be the weight parameters and deviation parameters of the $l$’th conventional layer. The parameters can reflect the relationship between the input ESPI fringe pattern and the output skeleton image. The training network actually converts into a process of finding the optimal parameters.

The problem of the training process can be formulated as a per-pixel classification problem. For an original input ESPI fringe image $I=\{{\widehat{I}}_i,i=1,\dots ,Z\}$, where ${\widehat{I}}_i$ is the pixel in the image and $Z$ is the number of pixels in the image. Our purpose is to predict the skeleton image $S=\{{\widehat{S}}_i,i=1,\dots ,Z\}$ of the input fringe image by the trained network. The pixel ${\widehat{S}}_i\in \{\mathrm{0,1}\}$ denotes the predicted label for each pixel ${\widehat{I}}_i$, which means that if the pixel ${\widehat{I}}_i$ is predicted as a skeleton pixel, the pixel ${\widehat{S}}_i=1$; otherwise ${\widehat{S}}_i=0$.

In the forward propagation, the input image $I$ will finally convert into a feature vector with two channels after all of the $l$ convolutional layers. The feature vector of the pixel ${\widehat{I}}_i$ can be expressed as

The output feature vectors $I_{fm}$ of $I$ can be expressed as $I_{fm}=(H_i^l,i=1,\dots ,Z)$. Then a softmax function is used to determine whether the pixel ${\widehat{I}}_i$ belongs to the skeleton pixel or the nonskeleton pixel. The softmax function is defined as

Then a loss function is used to quantify the difference between the probability values obtained by the forward propagation and the truth values. Here we use the cross entropy loss function, which can be expressed as

In the backpropagation, by minimization of the cross entropy loss function, the optimal parameters can be finally obtained

To solve Eq. (4), the stochastic gradient descent method²⁹ is used. When the parameters of the network are determined, the training process is finished.

2.3.

Prediction Process of Our Method

In the prediction process, the testing images are fed to the network. Each pixel of the testing images will be determined whether it belongs to the skeleton pixel or not by the trained network. Finally, the skeleton images can be obtained.

3.1.

Simulated Dynamic ESPI Measurement

3.1.1.

Training samples

We simulate a dynamic ESPI experiment and obtain 1000 noisy ESPI fringe patterns. The phase values of the 1000 ESPI fringe patterns are generated by the following equations:

Eq. (5)

$$\begin{gathered} {\psi }_{ij}=\alpha \times \left\{\exp \right[-\frac{{(2i-m)}^2+{(2j-\frac{11n}{8})}^2}{50000}\left]\right\}+\exp [-\frac{{(2i-m)}^2+{(2j-\frac{n}{2})}^2}{35000}] \\ +10\times [{\left(\frac{6i-3m}{2m}\right)}^2+{\left(\frac{3j-n}{n}\right)}^2], \end{gathered}$$

where ${\psi }_{ij}$ is the phase, $m$, $n$ are the image sizes, here $m=512$, $n=512$, and $\alpha$ is a parameter and $\alpha \in [\mathrm{0.01,100}]$.

$\alpha$ is a parameter for the simulated phase. By changing the value of $\alpha$, we can obtain a series of simulated ESPI fringe patterns. We simulate 1000 images with a range of $\alpha$ from 0.01 to 100. The value of $\alpha$ increases by 0.0999 each time.

We pick 21 original ESPI fringe patterns and the corresponding skeleton images as the training samples from the whole dynamic process. The other images are used for the test. The 21 simulated fringe patterns are picked from the 1000 fringe patterns evenly. The values of $\alpha$ in the 21 ESPI fringe patterns, respectively, are 0.01, 5.0095, 10.009, 15.0085, 20.008, 25.0075…. The interval of $\alpha$ is 4.9995.

To obtain the skeleton images of the training samples, we have to apply the GVF method based on VID¹³ 21 times to the 21 ESPI fringe patterns.

The details for the GVF method based on VID can be found in the published literatures in Ref. 13. Here we give a brief review of this method.

A skeleton extraction method is proposed based on the GVFs and VID.¹³ The GVFs of a whole image are the sum of the decomposed GVFs of low-density regions and high-density regions based on VID. Each part of the decomposed GVFs is described by a suitable function space. The skeletons of ESPI fringe patterns with variable density can be obtained based on the topological analysis of the GVFs of a whole image.

The GVF is defined by

Eq. (6)

$$V(x,y)=[u(x,y),v(x,y)],$$

where $u(x,y)$ and $v(x,y)$ are the two components of the GVF.

The GVFs $V(x,y)$ of the ESPI fringe patterns with variable density are decomposed into three components

Eq. (7)

$$V(x,y)=V_l+V_h+V_n,$$

where $V_l(u_l,v_l)$, $V_h(u_h,v_h)$, and $V_n(u_n,v_n)$ are the decomposed GVFs in low-density regions, high-density regions, and noise regions, respectively.

The GVFs in low-density regions $V_l(u_l,v_l)$, high-density regions $V_h(u_h,v_h)$, and noise regions $V_n(u_n,v_n)$ are described in Beppo-Levi space, the high-density regions in Hilbert space, and the noise regions in Curvelet space, respectively.

In the VID method, the GVFs in low-density regions $V_l(u_l,v_l)$, high-density regions $V_h(u_h,v_h)$, and noise regions $V_n(u_n,v_n)$ can be obtained by solving the following minimization problem:

Eq. (8)

$$\begin{gathered} (V_l,V_h,V_n)=\underset{{\tilde{V}}_l,{\tilde{V}}_h,{\tilde{V}}_n}{\arg \min }\frac{1}{2}{\Vert V-{\tilde{V}}_l-{\tilde{V}}_h-{\tilde{V}}_n\Vert }_{L^2}^2 \\ +\lambda {\Vert V_l\Vert }_{BL}^2+\mu {\Vert V_h\Vert }_{\xi }^2+\sigma {\Vert V_n\Vert }_{\text{Curvelet}}^2, \end{gathered}$$

where $\lambda$, $\mu$, and $\sigma$ are three positive reals to balance each term and $\frac{1}{2}{\Vert V-{\tilde{V}}_l-{\tilde{V}}_h-{\tilde{V}}_n\Vert }_{L^2}^2$ is a fidelity term that takes into account the presence of noise in the image.

The GVFs of whole image can be calculated by adding desired results of $V_l$ with $V_h$

Eq. (9)

$$\begin{gathered} u=u_l+u_h, \\ v=v_l+v_h. \end{gathered}$$

After calculating the GVFs, the skeletons can be extracted based on the topological analysis of GVFs. For instance, the GVFs are first normalized, then the skeleton image of black fringes $f_{bfs}(x,y)$ can be obtained by

Eq. (10)

$$f_{bfs}(x,y)=\{\begin{array}{lll}1& \text{skeleton pixel}& \text{if}{S}_{x,y}>T,J{M}_{x,y}>0\\ 0& \text{nonskeleton pixel}& \text{otherwise},\end{array}$$

where $T$ is a preset threshold. $S_{x,y}$ and $J{M}_{x,y}$ can be expressed as

Eq. (11)

$$S_{x,y}=\frac{{(\partial \frac{u_{xy}^N}{\partial x})}^2+{(\partial \frac{v_{xy}^N}{\partial y})}^2}{\max [{(\partial \frac{u_{xy}^N}{\partial x})}^2+{(\partial \frac{v_{xy}^N}{\partial y})}^2]},$$

Eq. (12)

$$\begin{gathered} J{M}_{x,y}=(\partial \frac{u_{xy}^N}{\partial x}+\partial \frac{v_{xy}^N}{\partial y}) \\ -\frac{\text{real}\sqrt{{(\partial \frac{u_{xy}^N}{\partial x})}^2-2(\partial \frac{u_{xy}^N}{\partial x})(\partial \frac{v_{xy}^N}{\partial y})+{(\partial \frac{v_{xy}^N}{\partial y})}^2+4(\partial \frac{u_{xy}^N}{\partial x})(\partial \frac{v_{xy}^N}{\partial y})}}{\tau }. \end{gathered}$$

Figure 3 shows some noisy ESPI fringe patterns and the corresponding skeleton images, which are used as the training samples.

Fig. 3

Parts of the training samples and the corresponding skeleton images used for training the network: (a)–(c) initial images and (d)–(f) skeleton images of (a)–(c), respectively.

As we all know, the precision of the prediction produced by the CNN can be improved by increasing the number of training examples. In our case, it is not easy to obtain annotated ESPI skeleton images. In order to tackle the deficiency of the training samples, we construct the training samples for patch-based pixel-wise segmentation and sampled overlapped patches.

In our training dataset, each noisy ESPI fringe pattern and each skeleton image are both cropped into 4800 patches, respectively. The size of each patch is $32\times 32\text{pixels}$. In order to maintain AC, the patches extracted from the training samples are overlapped and the total number of patches sampled from the training dataset is 100,800. This effectively reduces GPU memory usage and takes advantage of efficient batch processing.

Figure 4 shows an example of the ESPI fringe patterns patches and the corresponding skeleton patches. Then all of the patches are as input fed to the U-Net CNN. After 200 iterations, the network training process is completed.

Fig. 4

Example of the input training sample patches and the corresponding skeleton image patches for training the network: (a) $32\times 32\text{pixel}$ random portions of computer-generated noisy ESPI fringe and (b) the corresponding skeletons of (a).

3.1.2.

Testing results

The other 979 ESPI fringe patterns as the testing images are fed to the trained network. In each test, 20 noisy ESPI fringe patterns are simultaneously fed to the network. The corresponding skeleton images can be obtained simultaneously. Here we provide eight testing images shown in Fig. 5 and the corresponding skeleton extraction results by our method are shown in Fig. 6.

Fig. 5

Parts of the testing samples: (a)–(h) the initial ESPI images.

Fig. 6

The skeleton extraction results by the proposed method: (a)–(h) the skeleton images of Figs. 5(a)–5(h), respectively.

Subsequently, we apply this trained network to a group of difform ESPI fringe patterns. The formulas of phase in these images are different from those of phase in Figs. 3 and 4. The shapes of the testing images are different from those of the training samples. The skeleton extraction results are still reliable. The testing ESPI fringe patterns and the corresponding skeleton images are shown in Fig. 7.

Fig. 7

Three difform ESPI fringe images and the corresponding skeleton results: (a)–(c) initial ESPI images with different density and (d)–(f) skeleton images of (a)–(c), respectively.

AC is an extensively used index for the measure of the ratio of pixels that are precisely classified across all pixels. The AC³⁰ is given by the following equation:

Eq. (13)

$$\mathrm{AC}=\frac{\mathrm{TP}+\mathrm{TN}}{\mathrm{TP}+\mathrm{TN}+\mathrm{FP}+\mathrm{FN}},$$

where TP is the number of classifications where pixels are classified into skeleton pixels correctly and TN is the number of classifications where pixels are identified as nonskeleton pixels correctly in the outputs of the network. FP is the number of classifications where pixels are misclassified into skeleton pixels in the outputs of the network. FN is the number of classifications where pixels are misclassified into nonskeleton pixels in the outputs of the network.

As shown in Figs. 3 and 5, the original ESPI fringe patterns are obviously very noisy and of limited visibility. Using the previous methods, it is necessary to process the 1000 ESPI fringe patterns for one thousand times. However, using our method, we only need to process few numbers of the training samples. Once the network is trained well, the skeleton images can be obtained in batches. In our trained network, we can simultaneously process 20 noisy ESPI fringe patterns and it takes just 40 s to obtain 20 skeleton images. As can be seen in Fig. 6, the skeleton extraction results by the trained network are desired.

In addition, as shown in Fig. 7, we are surprised to find that even though the shapes of the testing images are slightly different from the training samples, our method also gives desired results. In the simulation test, the AC of the trained network is 0.9727.

3.2.

Experimental Dynamic ESPI Measurement

In this part, we apply our network to a real dynamic measurement of a printed circuit board with a chip under normal working conditions by ESPI.

Figures 8(a) and 8(b) show the two experimentally obtained original ESPI fringe images with sizes of $390\times 390\text{pixels}$, which depict the out-of-plane displacements of the board at 1.80 and 2.04 s. As can be seen in Figs. 8(a) and 8(b), the quality of the two experimentally obtained ESPI fringe patterns is very poor because of high noise and low contrast. It takes a lot of effort with specialized technology to obtain the skeleton images. Extracting skeleton from this type of ESPI image is particularly challenging using the previously existing methods.

Fig. 8

Parts of the training samples for the experimental dynamic ESPI measurement: (a) and (b) two simulated ESPI fringe images with low quality; (c) and (d) two real ESPI fringe images with low quality; and (e)–(h) the corresponding skeleton images of (a)–(d), respectively.

For the experimentally obtained ESPI fringe patterns, it is very difficult to extract the perfect skeletons. We laboriously obtain six fringe skeleton images of the original noisy ESPI fringe images using the GVF methods based on VID.¹³ The six training samples (six experimentally obtained ESPI fringe patterns and the corresponding skeleton images) are not enough to train the U-Net CNN. Compared with the experimentally obtained ESPI fringe patterns, it is relatively easy to obtain the skeletons of the computer-simulated ESPI fringe patterns. Therefore, we add 10 simulated ESPI fringe patterns to the training samples. The criteria are that: first, the sizes of the simulated ESPI fringe patterns should be same as those of the experimentally obtained ESPI fringe patterns; second, the contrasts and the shapes of the simulated ESPI fringe patterns should be similar to those of the testing images as much as possible. The corresponding skeleton images of the 10 simulated ESPI fringe patterns are obtained using the GVF methods based on VID. The training samples in the test are composed of six experimentally obtained ESPI fringe patterns and 10 simulated ESPI fringe patterns. This strategy can be extended to other real-dynamic ESPI measurements.

Each image used in the training process is also cropped into 4800 patches with size of $32\times 32\text{pixels}$. The total number of patches sampled from the training dataset is 76,800. Figure 8 shows the parts of the training samples. Then all of the patches are as input fed to the U-Net CNN, and after 150 iterations, the training process is finished. The sizes of the patch will affect the experimental results. In order to ensure the number of positive samples, the sizes of the patch should be larger than the width of the fringes. Figure 9 shows an example of the ESPI fringe patterns patches and the corresponding skeleton patches. We apply our trained network to the other real ESPI fringe patterns. Parts of the testing results of the real ESPI fringe patterns are shown in Fig. 10. The AC of the trained network in the test is 0.9514.

Fig. 9

Example of the input training sample patches and the corresponding skeleton image patches for training the network: (a) $32\times 32\text{pixel}$ random portions of noisy ESPI fringe and (b) the corresponding skeletons of (a).

Fig. 10

Skeleton extraction results on the experimental obtained ESPI fringe patterns: (a)–(c) three real ESPI fringe images with low quality; (d)–(f) the corresponding skeleton images of (a)–(c) by GVFs-VID, respectively; (g)–(i) the corresponding skeleton images of (a)–(c) by our method, respectively; (j)–(l) the superimposition of (g)–(i) onto enhanced (a)–(c).

For better illustration of our method, we give a comparison of our method with the GVFs method based on VID.¹³ Figures 10(a)–10(c) show the experimentally obtained original ESPI fringe patterns; Figs. 10(d)–10(f) show the skeleton images, which are obtained by the GVFs method based on VID; and Figs. 10(g)–10(i) show the skeleton images, which are obtained by our method. In addition, we enhance the contrast of Fig. 10(a) and superimpose Fig. 10(g) on Fig. 10(a), shown in Fig. 10(j). The similar results of Figs. 10(h) and 10(i) are shown in Figs. 10(k) and 10(l), respectively.

From Figs. 8 and 10, we can observe that the qualities of initial ESPI images are considerably low, because of variable density, high noise, and low contrast. Extracting the skeletons in this case is very challenging. As shown in Figs. 10(d)–10(f), the skeletons obtained by the GVFs method based on VID are satisfactory. But the good results of the GVFs method based on VID are relative to the parameters. The parameters need to be adjusted carefully for each image. By employing our trained network, we can obtain the skeleton images directly.