3.1

Problem definition

In scene graph generation tasks, images are transformed to graph structures by I →G. G, the scene graph structure, is used to describe the target nodes in the scene graph and the predicate relationships between the target nodes, which contain the candidate target box B={b₁, b₂, …, b_n}, b_i ∈ℝ⁴. The category collection corresponding to the target box is O={o₁, o₂, …, o_n}, o_i∈ C¹⁵¹, where C¹⁵¹ is the candidate category collection truth set. The triadic relationship set between nodes is R={r₁, r₂, …, r_m}, where r_k is the subject-predicate-object triplet, with subject node (b_i, o_i), (b_j, o_j)∈B×O; predicate node p_i→j∈P⁵¹, P⁵¹ as the candidate relationship truth set. Therefore, the scene graph generation process containing image I to graph structure G, and target box B, target category O and subject-object relationship R can be expressed by the following factorization model:

where P_r (B|I) is the probability of the entity candidate box obtained by the target detection model (section 3.2); P_r (O|B, I) is the probability distribution of the entity box categories in target detection; P_r (R|B,O,I) is the probabilities of the inference fusion relationship among image I, category O and candidate target box B (sections 3.3 and 3.4).

3.2

Visual information generation module

For a given image, we use the target detection method for the targets and the convolutional neural networks for the image’s advanced feature F^I, F^I ∈ ℝ^n×512×w×h. The alignable fused visual feature F^o ∈ ℝ^n×512 can be obtained by mean-pooling mapping on F^I, and can be calculated by

where AVG takes the mean sampling of the feature F^I with size of w×h and transforms them to 1×1. To match with the size of the feature fusion, the matrix n×512×1×1 is dimensionally compressed and finally F^o is obtained by flattening operation on features mapping.

The proposed target visual features V = {v₁, v₂, …, v_n}, v_i∈ℝ⁴⁰⁹⁶ are obtained through the target detection module, where each target i corresponds to a bounding box b_i, and the candidate bounding box collection B={b₁, b₂, …, b_n}, b_i=(x_i1, y_i1,y_i2,y_i2). Since the obtained entity information is regional, it cannot reflect the size relationship between targets.

We try to recompute the bounding box features and construct the following encoding way:

where “:” and fc are used as splicing and full connection operations. In order to align the input fusion features and the extensive information space, equation (3) has full connection adjustment on the code information and then pos_bi ∈ℝ^n×128 is obtained, and so as the spatial relative size of each target.

Lastly, we take all the above information into consideration and get the overall visual information of the ith node as follows:

where is the visual information of the ith node, and the category embedding feature S_oi is obtained by embedding words into GloVe²⁰.

3.3

Relationship inference module

The relationship inference module in GECF model contains two sub-modules: attention structure based context module and long and short-term memory networks based context module. The two sub-modules map the target information in different ways using the before-and-after information of sequence data to the relationship space in parallel, so that the information transferring among nodes and edges can be sufficiently proceeded and the long-range dependence between the targets can be captured.

3.3.1

Attention Structure Based Context Module.

In the area of context information modeling, Cao et al.²¹ propose a simplified GC (Global Context) module as shown in Figure 3a. The excessive operation volume caused by nonlocal weighting structures drops using the GC module²², which also solves the problem of the insufficient effectiveness of the channel attention mechanism on global context modeling²³.

Figure 3.

Attention-based global context module.

Inspired by the Non-local²² and SENet²³ methods, this paper also introduces a global context method into relationship inference and constructs a global weighting module as shown in Figure 3b, in which the long-range dependency relationship between targets is captured after using the multiple target features in an image to iteratively update the adjacency information of a single target feature.

The fused information in equation (4) is encoded using the global weighting module in Figure 3b, among which the single weighting module is calculated as follows:

where σ is the SoftMax function; W, H, U are learnable parameters for the fully connected layer; fc is the fully connected operation.

3.3.2.

Long Short-Term Memory Networks Based Context Module.

Based on the bidirectional LSTM network, context information transfer in MOTIFS¹⁸ is carried out. But the study ignores the important information in the key nodes and edges, and the generated scene graph is with ambiguous main relationships and unclear key targets. BiLSTM-Attention network is applied to improve the relationship inference in this paper inspired by the relation extraction in natural language processing^{24, 25}. Attention mechanism based model focuses more on the primary relationship between the targets and obtains the information representation of key nodes by following the hidden layer information of the neighboring target nodes. The hidden layer feature information is as follows:

where A₁ is the hidden layer feature information of the target node after encoding. The target information A₂ in the context of each node feature is

where Q and K are randomly initialized trainable parameters, and tanh is the activation function. To weigh the importance of a node, Q is used as a similarity measure represented by advanced query representation. The context relationship information space is obtained by multiplying A₁ with the normalized weight matrix.

3.4

Relationship fusion module

We decode the relational spatial information of the inference module by the bidirectional LSTM and get the context features of the edges:

where E_o and E_s are the semantic features of the subject and object, E_o, E_s ∈ℝ⁴⁰⁹⁶. Subject and object features are included in the obtained context space. In previous studies, the probability of relationship classification is mainly obtained by multiplying the separated subject and object features. We can see from reference²⁶ that nonlinear projection on fusing visual feature x and problematic feature y to measure the difference between them. The equation is followed:

where ReLU is the activation function; W_x, W_y are learnable parameters. Similarly in this paper, we take different information of the subject and object to measure the probability distribution of their corresponding predicate relations, which is called R_i and calculated as follows:

4.1

Dataset and evaluation indicators

In this paper, we use the same dataset Visual Genome and evaluation indicators as references^{14, 18, 27, 28}. Specifically, considering the influence of long-tail distribution, 150 common object categories and 50 relationship categories are selected for evaluation. After the dataset pre-processing, the scene graph for each image has an average of 11.6 objects and 6.2 relationships. For the comparison with existing methods, the dataset is separated into a training set, a validation set, and a test set. The training set contains 75651 images, in which 5000 images are in the validation set, and rest of the 32422 images are in the test set. Three subtasks are set to evaluate the effect of the scene graph generation model:

PredCls (Predicate Classification): set the bounding borders and labels of the correct location of entities; classify relationships between the targets;

SGCls (Scene Graph Classification): set the bounding borders of the targets; predict the labels of the targets in the border, and then classify the relationships between the target pairs;

SGGen (Scene Graph Generation): take only one original image; detect the targets in the image for their bounding border information and labels; classify the relationship between the target pairs.

In this paper, we use the recall rate Top-K as evaluation indicator, denoted as Recall@K, which represents the proportion of correctly predicted classifications in the first K predicted relationships and K is set to 20, 50 and 100 respectively.

4.2

Experimental setting

A two-stage training method is applied in the scene graph generation in this paper. We use the Faster-RCNN target detection model pre-trained by VGG16²⁹ for entity detection. Images are all with a uniform size of 592 × 592, and normalized with mean and variance. When we get the global information, the abstract visual feature map is obtained by mean-pooling and full connection on VGG16 output. The GloVe²⁰ model is used to convert the categories into word vectors at the time of category embedding, and a fully connected layer with an output dimension of 256 is applied for the fusion of the output features of the self-attentive structure and the bidirectional LSTM. The stochastic gradient descent (SGD) is set as the convergence algorithm, with the parameter batch set to 1, the base learning rate to 0.001, the weight decay to 0.0001, and the momentum to 0.9. The model is trained on RTX-2080tiGPU based on PyTorch framework for single card training.

4.3

Quantitative analysis

As shown in table 1, the method proposed in this paper containing three subtasks on Visual Genome dataset are compared with the scene graph generation methods of IMP¹⁵, AE³⁰, TFR³¹, G-RCNN¹⁴, and MOTIFS¹⁸. We use the same target detection pre-trained model for each method. The experiment results show that the GFCF model in this paper performs better in handling the three subtasks than the other methods. As to PredCls, the recall rates of Recall@20\50\100 are 61.6%，66.9% and 68.4% respectively, with an improvement of more than 1%. This certifies that the model has a good performance on relationship prediction classification, especially in SGCLs.

Table 1.

Recall rates of GECF and the existing models.

4.4

Ablation experiments

In order to obtain the specific impact of the method proposed in this paper on scene graph generation, we design ablation experiments on embedding global information, introducing attention mechanisms and the fusion way in the final relationship inference respectively.

4.5

Qualitative results

To qualitatively validate the constructed scene graph and visual relationship model, we put some examples of visualizations in the VG dataset based on the SGCls subtask in Figure 4. In the scene images in Figure 4, the green bounding box is the correct prediction zone and matches with the correct label, and the orange bounding box is the correct bounding box without a match. In the scene graphs, the black edge represents the correct predicate classification, the orange includes not only the negation of detected correct predicate, but also negation of relationships between targets that exist but are not detected, while the red represents the wrong predicate prediction classification. For example, in Figure 4a, several different types of targets are correctly detected and relationships are correctly predicted, where the unlabeled <bus-has-window>, <windshield-on-bus> conforming to common sense relationships are also predicted. For targets that cannot be accurately predicted, their associated relationships cannot be predicted either, such as <person-wearing-shirt>, <girl-near-woman> in Figure 4e. Besides, we find that there are conflicting predictions on some targets with accurate predictions, like <table-with-chair> and its correct relationship <chair-near-table> shown in Figure 4b and also wrong prediction on non-significant relationship between targets. There are false relationship detections from the model proposed in this paper, most of which are caused by target detector failure and incorrect detection, like <sign-behind-tree> in Figure 4d. On the premise of accurately detecting the targets, the relationship prediction model in this paper still performs good.

Figure 4.

Scene graph visualization.

To validate the improvement of the model in this paper on qualitative results, the scene graph generation is visualized and compared with part of the qualitative results of Neural¹⁸ method in the SGCls subtask. As shown in Figure 5, green boxes are the correctly predicted target classifications while the red boxes are the incorrectly predicted target classifications, with the correct classification in brackets. Green edges are the correct relationship predictions of the while the red edges are the wrong relationship predictions, with the correct relationship in brackets. In Figure 5, although there are errors in single target and relationships, the overall target detection results and relationship Recognition performance are better than those of the neural method.

Figure 5.

Comparisons of graph structure visualization between GECF and Neural methods.