Automatic temporal segment detection via bilateral long short-term memory recurrent neural networks

Constrained by the physiology, the temporal factors associated with human behavior, irrespective of facial movement or body gesture, are described by four phases: neutral, onset, apex, and offset. Although they may benefit related recognition tasks, it is not easy to accurately detect such temporal segments. An automatic temporal segment detection framework using bilateral long short-term memory recurrent neural networks (BLSTMRNN) to learn high-level temporal–spatial features, which synthesizes the local and global temporal–spatial information more efficiently, is presented. The framework is evaluated in detail over the face and body database (FABO). The comparison shows that the proposed framework outperforms state-of-the-art methods for solving the problem of temporal segment detection. © The Authors. Published by SPIE under a Creative Commons Attribution 3.0 Unported License. Distribution or reproduction of this work in whole or in part requires full attribution of the original publication, including its DOI. [DOI: 10 .1117/1.JEI.26.2.020501]


Introduction
Human behavior recognition is an important subject in the field of pattern recognition, which is of great interest for human-computer interaction. Although they are constrained by one's physiology, the temporal factors of human behavior, e.g., facial movements or body gestures, are described by four phases: neutral, onset, apex, and offset. 1 Different phases have different manifestations and intensities. Some researchers have shown that active unit (AU) activation detection 2 and genuine (spontaneous) emotion recognition 3 both benefit from different temporal segments. In addition, they have been proven to be beneficial for emotion recognition, especially for multimodal emotion recognition combining facial expression, body expression, voice, etc. [4][5][6][7][8][9][10][11] Therefore, the temporal segment detection of human behavior warrants further exploration.
The work introduced in this letter offers the following contribution: to date, we are the first to introduce the bilateral long short-term memory recurrent neural networks (BLSTM-RNN) for the automatic detection of the temporal phases of human behavior. A high-level feature that simultaneously contains temporal-spatial information is learned with the BLSTM-RNN method, which has synthesized both local and global temporal information. It shows outstanding performance.
The remainder of this letter is organized as follows. Section 2 introduces related works. Section 3 provides details of the overall methodology. Section 4 describes the experiments and the extensive experimental results. Finally, the conclusion is given in Sec. 5.

Related Work
In this section, we will review some existing methods that are related to temporal segments. A number of studies have detected the temporal segments by temporal rules drawn up by researchers or other classification schemes, such as support vector machines (SVM) 12 and hidden Markov models (HMMs). 13 Pantic and Patras 14,15 used the temporal rules they drew up to detect the temporal segment of facial AUs. Focusing on bimodal affect recognition, Gunes and Piccardi 10 proposed a method to automatically detect temporal segment of facial movements and body gestures, which includes both frame-and sequence-based strategies. HMM was applied as a sequence-based classifier. Several different algorithms provided in the Weka tool, including SVM, AdaBoost, and C4.5, have been utilized as frame-based classifiers. Jiang et al. 2 used HMMs to detect the temporal segment of facial AUs. Chen et al. 16 design two features to describe face and gesture information, then use SVM to segment expression phase. However, the segment accuracy (Acc) of the existing methods is still to be improved.
Recently, deep learning methods have become very popular within the community of computer vision. BLSTM-RNN is one of the state-of-the-art machine-learning techniques.
In this letter, to synthesize the local and global temporalspatial information more efficiently, we present an automatic temporal segment-detection framework that uses BLSTM-RNN 17 to learn high-level temporal-spatial features. The framework is evaluated in detail over the face and body (FABO) database. 18 The result of the experiments in Sec. 4 proves that the proposed framework outperforms other stateof-the-art methods for solving the problem of temporal segment detection.

Methodology
This section presents the details of our method. Figure 1 shows an overview of our proposed method, which consists of two major parts: (1) data preprocessing and (2) feature extraction and representation. We use SVM as a classifier.

Data Preprocessing
Face detection is a very important step in the entire pipeline of facial movement temporal segment detection, which directly affects the effectiveness of the feature extraction. Before considering more accurate feature extraction by using methods, such as discrete cosine transform (DCT) combined, 19 the local binary pattern (LBP), 20 the pyramid histogram of oriented gradient (PHOG), 21 the entropy (E), 22 the motion area (MA), 16 and the neutral divergence (ND), 16 we follow the methods of 23

Low-level features
We extract E and MA based on the motion history image (MHI). An MHI is a static image template which is helpful in understanding the motion location and path as it progresses. 27 As is common for facial movements, we extract six low-level descriptors, including the sum of pyramid LBP (SPLBP), the sum of pyramid two-dimensional DCT combined (SPDCT), the sum of PHOG (SPHOG), E, MA, and ND. For body gestures, only E, MA, and ND are extracted as low-level descriptors. Brief introductions can be found in Refs. 16, 19-22.

High-level features
After the first step, we obtain several low-level features. Additional temporal information is obtained by learning the high-level features in the time domain with the BLSTM-RNN method and feature-level fusion strategy, which was used to synthesize the local and global temporal information.
Because the input of BLSTM-RNN is required to be a sequence, first, we employ a fixed-size temporal window with its center located at the current frame. Figure 2 shows some examples. In our experiments, we set the temporal window size equal to that of Chen et al. 16 Therefore, each low-level feature vector has the dimensionality of the temporal window size. Then these feature vectors can be used as input for BLSTM-RNN.
For BLSTM-RNN, we use the implementation of Theano. 28 Figure 3 shows the network structure of highlevel features. First, we use low-level features as input for BLSTM-RNN and compute the forward hidden sequencẽ h and the backward hidden sequence ⃖ h, respectively. Then we link them together by concatenation.
Then we use a feature-level fusion strategy for all of the extracted high-level features. This fusion can be implemented by concatenating all feature vectors together.
Finally, we employ SVM 29 as the classifier to detect temporal phases.

Experimental Setup
We conduct the experiments on the bimodal face and body database FABO. 18 This database consists of both face and body recordings using two cameras simultaneously. So far, it is the only bimodal database that has both expression annotation and temporal annotation. We choose 245 × 2 videos in which the ground truth expressions from both the face camera and body camera are identical. Among them, 129 videos were used for training and 119 videos were used for testing. In this section, we select the ACC as the measure to evaluate the results. The calculating equation of ACC is given as E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 0 9 ; 3 2 6 ; 2 6 7 ACC ¼   where s denotes the number of temporal phase categories, P i denotes the precision of the i'th temporal phase class, and TP i and FP i denote the number of correct classification and the number of wrong classifications in the i'th temporal phase class, respectively.

Experimental Results
In this section, we compare the ACC in percent under different conditions.

Result of feature-level fusion
In this part, we apply three methods for feature-level fusion, and the best one is used on the experimental results. Table 1 shows the result of feature-level fusion from features of body gestures using BLSTM-RNN with a softmax classifier. In the table, the item BLSTM(1) means using the output of the first layer of BLSTM as features; the item BLSTM(2) means using the output of the second layer of BLSTM as features; the item BLSTM(1+2) means concatenating the output of the first and second layers of BLSTM as final features; the item ALL(before) means first simply concatenating all low-level features together, then combining them for input into a BLSTM-RNN; the item ALL(after) means first use each low-level feature as input into a BLSTM-RNN, then simply concatenate all highlevel features together. The results in this table indicate that the most accurate result is 95.20. It is obtained by first using each single feature as input into a BLSTM-RNN, of which the output of the first and second layers is concatenated together as high-level features. Then apply softmax as the classifier. Table 2 presents the results of feature-level fusion from features of facial movement by using BLSTM-RNN with softmax. This table indicates that the most accurate result is 88.54, which is obtained by first using each low-level feature as input into a BLSTM-RNN, and concatenating the output of the first and second layers, before simply concatenating all high-level features together.

Results of some classifiers on low-level features and high-level features
In this part, we compare different classifiers for temporal segmentation. Table 3(a) contains the classification results of some classifiers [e.g., softmax, SVM, random forest (RF)] for body gestures. In this table, the item ALL(low level) means simply concatenating all low-level feature vectors together. The item ALL(high level) means simply concatenating all high-level features together. The results in this table show that: (1) the best result is 95.30, which is obtained with the SVM, and (2) the results on high-level features are more accurate than those on low-level features. It shows the validity of high-level features. Table 3(b) shows the classification results of some classifiers for facial movement. The results in this table show the following: (1) the best result is 89.52, which is obtained with the SVM and (2) the results on high-level features are more accurate than on low-level features. It shows the validity of high-level features. From Table 3, we can see that SVM performs best, so we apply SVM as our final classifier.

Performance comparison
In this part, we present a comparison of results from featureand decision-level fusion for facial movement and body gesture, respectively, in Table 4, and a comparison of results from our approach and relevant experiments on FABO database in Table 5. Table 4 shows the result of feature-and decision-level fusion for facial movement and body gesture. These results   (2) indicate that the results obtained for feature-level fusion are more accurate than those obtained for decision-level fusion. Thus, we employed feature-level fusion as our final strategy. The performance of the proposed approach and the stateof-the-art approach reported in Refs. 10 and 16 is compared in Table 5. Gunes and Piccardi 10 proposed a method to automatically detect temporal segment of facial movement and body gesture, which includes both frame-and sequencebased strategies. HMM was applied as a sequence-based classifier. Several different algorithms provided in the Weka tool, including SVM, AdaBoost, and C4.5, have been utilized as frame-based classifiers. They obtained an ACC of 57.27% for face and 80.66% for body, respectively. Chen et al. 16 designed two features to describe face and gesture information, then used SVM to segment expression phase. They only used body video to detect the temporal segment of the expression; they did not detect the temporal segment of facial movement and body gesture separately. They obtained an ACC of 83.10% for expression. Results of the proposed approach are obviously more accurate than those of the state-of-the-art methods.

Conclusions
This letter presents a temporal segment detection framework using BLSTM-RNN to learn high-level temporal-spatial features. The framework is evaluated in detail using data obtained from the FABO database. A comparison with other state-of-the-art methods shows that our method outperforms the other approaches in terms of temporal segment detection. In the future, we plan to focus on affect recognition based on temporal selection face and body display.