Most background subtraction algorithms developed to detect moving objects are potentially problematic in that they experience performance degradation when weather conditions are adverse. We solve this problem by proposing a refinement method using a consistent motion detection method, the performance of which is robust to weather related changes in video images captured by a static camera. The proposed algorithm reduces the number of false-positive regions and fills parts that are missing as a result of the nature of the background subtraction methods. We show the extent of the improvement afforded by our algorithm in the handling of moving object detection in adverse weather conditions.

We present a residual learning-based convolutional neural network, referred to as DeepResPore, for detection of pores in high-resolution fingerprint images. Specifically, the proposed DeepResPore model generates a pore intensity map from the input fingerprint image. Subsequently, the local maxima filter is operated on the pore intensity map to identify the pore coordinates. The results of our experiments indicate that the proposed approach is effective in extracting pores with a true detection rate of 94.49% on test set I and 93.78% on test set II of the publicly available PolyU HRF dataset at a false detection rate of 8.5%. Most importantly, the proposed approach achieves state-of-the-art performance on both test sets.

This guest editorial summarizes the Special Section on Advanced and Intelligent Vision Systems.

An in-socket sensory system enables the monitoring of transfemoral amputee movement for a microprocessor-controlled prosthetic leg. User movement recognition from an in-socket sensor allows a powered prosthetic leg to actively mimic healthy ambulation, thereby reducing an amputee’s metabolic energy consumption. This study established an adaptive neurofuzzy inference system (ANFIS)-based control input framework from an in-socket sensor signal for gait phase classification to derive user intention as read by in-socket sensor arrays. Particular gait phase recognition was mapped with the cadence and torque control output of a knee joint actuator. The control input framework was validated with 30 experimental gait samples of the in-socket sensory signal of a transfemoral amputee walking at fluctuating speeds of 0 to 2  km  ·  h^−  1. The physical simulation of the controller presented a realistic simulation of the actuated knee joint in terms of a knee mechanism with 95% to 99% accuracy of knee cadence and 80% to 90% accuracy of torque compared with those of normal gait. The ANFIS system successfully detected the seven gait phases based on the amputee’s in-socket sensor signals and assigned accurate knee joint torque and cadence values as output.

A belly reconstruction and measurement scheme is conducted on a shape-flexible mannequin with digital image correlation technique. We adopted an integer subpixel image matching process. First, a compound feature including Gaussian combined moment and parameters extracted from gray-level co-occurrence matrix is proposed to track an integer pixel between reference and target deformed images. Second, a mutual learning co-operative particle swarm optimization algorithm is employed to locate the subpixel precisely. Each subpopulation does its own optimization independently. The subpopulations keep information communication and knowledge sharing for co-operative evolution to enhance global searching capability. In addition, the previous optimal information from the former interest point is adopted fully in initializing the particles’ positions of the next interest point, which effectively improves the speed of the convergence. Experimental results indicate that under the high measurement accuracy without any loss, the time-consumption of this scheme is significantly superior to that of the conventional method, particularly at a large number of interest points.

Video-based human action recognition is a challenging task in computer vision. In recent years, the convolution neural network (CNN) and its extended versions have shown promising results for video action recognition. However, most of the existing methods cannot deal with the global motion information effectively, especially for long-term motion which is crucial to represent complex none-periodic actions. To address this issue, a stacked trajectory energy image (STEI) is proposed by extracting trajectories from motion saliency regions and stacked them onto one grayscale image. This will result in an STEI with discriminative texture feature which can effectively characterize the global motion from multiple consecutive frames. Then, a three-stream CNN framework is proposed to simultaneously capture spatial, temporal, and global motion information of the action from RGB frames, optical flow, and STEI. Moreover, a trajectory-aware convolution strategy is introduced by incorporating local and long-term motion information so as to learn the motion features directly and effectively from three complementary action-related regions. Finally, the learned features are aggregated and categorized by a linear support vector machine. The experimental results on two challenging datasets (i.e., HMDB51 and UCF101) demonstrate that our approach statistically outperforms a number of state-of-the-art methods.

The trade-off between accuracy and computational cost is a central aspect of the problem of real-time vehicle detection, especially for resource-limited platforms. We propose a vehicle-detection method based on template aggregate channel features (TACFs) and soft channel cascade to address the computational requirements of a monocular rearview vehicle detection system. The TACFs is a system of template features designed based on the original aggregate channel features, which is aimed at fully describing the characteristics of the vehicle’s appearance. The proposed soft channel cascade is a pipeline of multiple single-channel classifiers and a full-channel classifier, in which the single-channel classifiers are used to quickly filter the background window and reduce the area of the region of interest, and the full-channel classifier is used to verify the candidate window to improve the detection precision. The proposed method is evaluated on three different datasets with a variety of scenarios involving dense traffic, poor lighting, bad weather conditions, and partial occlusion. Detailed evaluations show that the proposed method enables high-precision detection and high recall rate, and performs better than the existing state-of-the-art vehicle detection methods. In addition, the soft channel cascade can rapidly reject most windows that do not contain any vehicles and enables real-time detection.

We present the discrete separable shearlet transform (DSST) separability assessment system for recognizing facial expressions. DSST is an image multiscale geometric analysis method. We use a separability assessment to evaluate the separability of different scales and directions of the coefficients after DSST transformation. First, all test and training images are normalized and equalized. Then, all preprocessed images are DSST-transformed, and all low- and high-frequency coefficients are obtained. Next, the separability of different scales and directions of the coefficients is evaluated, and we use only those that have a large separability index. Then, we combine the low- and high-frequency coefficients for the best separability direction and scale as the extracted features. Finally, we use a support vector machine to classify seven expressions (i.e., happiness, sadness, surprise, disgust, fear, anger, and neutrality) from the Japanese Female Facial Expression, Extended Cohn–Kanade, MMI, and Psychological Image Collection at Stirling datasets. The experimental results show that the recognition rate of the proposed method is better than those of state-of-the-art methods.

Convolutional neural networks (CNN) have given rise to a new generation of video super-resolution (SR) technique. However, most existing CNN-based video SR algorithms treat the consecutive frames as a series of feature maps, just as the procedure performed in single image SR algorithms. We propose an end-to-end three-dimensional (3-D) CNN video SR framework. The input frames are considered as a cube in our framework. 3-D convolution is performed on it to extract features along spatial and temporal dimension. Image prior knowledge, such as optical flows, is introduced in reconstruction. A combination of mean square error loss and multiscale structure similarity index (MS-SSIM) loss is used to optimize the model. Experimental results show that the proposed method reconstructs high-resolution frames with more accurate and visually pleasant structures compared with state-of-the-art video SR algorithms. We also achieve comparable PSNR/SSIM results with less computation time.

We propose an improved object tracking algorithm based on kernelized correlation filter (KCF), which can overcome the drawback of traditional KCF algorithms in that they cannot effectively adapt to target-scale variations and target occlusion in tracking. First, the target-scale pyramid is built, whose histogram of oriented gradients feature extracted of every layer multiplied by the correlation filters; the maximum response of current scale of the filters is the best target scale. In addition, the improved algorithm is combined with the improved correlation filter framework, and the background information around the target is appropriately increased. When the target is occluded, the background information can be effectively used to track the target. The proposed algorithm is validated on the benchmark evaluation and compared with the traditional algorithms, such as KCF and circulation structure of tracking-by-detection with kernel. The results indicate that our tracking accuracy can reach 66.9% and the success rate can be 58.2. When the target is scale variation, the accuracy and success rate increase by 1.1% and 10.3%, respectively, compared with KCF. If the tracked target is occluded, a second improved algorithm is compared with the detection algorithm which only adds the scale detection, the tracking accuracy increases by 8%, and the success rate increases by 4.9%.

Seismic data visualization and analysis can help the domain experts, e.g., geologists and oil or gas exploration experts, to explore the distribution of petroleum or gas. It assists them to get a better understanding of stratigraphic structures and the distribution of the geological materials, e.g., underground flow path (UFP) and the contexts of UFPs (river delta, floodplain, slump fan, oil well, etc.). UFPs are one of the significant stratigraphic structures according to the domain experts, because they are closely related to the distribution and the migration of oil or gas. We design a quadratic-surface distance query scheme to explore UFPs and their contexts within a local region. First, it just needs to share parameters of quadratic surfaces to the rendering modules instead of all volume data or all subvolume data to conduct distance queries, and it is flexible to perform multiple complex logic operations through the quadratic surface-based queries. Second, it enables one to perform domain-specific interactions after distance queries such as the flexible switching of multiple display modes. Third, it enables one to perform local transfer function on different subvolumes different query results or their arbitrary combinations. We have evaluated the approach by comparing them with existing methods by performance evaluation and result evaluation. Results show that the proposed approach is capable of performing complex distance queries and fulfilling the domain-specific interactions getting better results and timing performance.

To reduce the matching ambiguities and explore the matching potential of repetitive features from the scenes with repetitive elements, we present a robust framework of Delaunay triangulation matching based on feature saliency analysis. The feature saliency is computed based on the Gaussian invariance model of the selected reliable features and used as guidance for the triangle-constraint region matching. Due to the seed-sensitive nature of the triangulation matching, the framework integrates the effectiveness validation steps to predict whether the extracted features and the found seeds are proper for matching the specific scenes. By suppressing the surround feature influence, we provide an opportunity for the repetitive features to enhance their feature saliency in the local regions and increase the discriminative power to identify their correspondences. We benchmark the matching performance on the tradeoff between the mean precision and recall. The promising results manifest its effectiveness on the ambiguity reduction.

Abnormal event detection in crowded scenes is a challenging task in the computer vision community. A hybrid motion descriptor named the multiscale histogram of first- and second-order motion is proposed for abnormal event detection. The second-order motion describes the change in motion and is extracted by optical flow-based instantaneous tracking, which avoids object tracking in crowded scenes. For the modeling of normal events, a kernel null Foley–Sammon transform (KNFST) is introduced. KNFST makes a projection in the null space, where normal samples of all types are treated jointly instead of considering each known class individually. Experiments conducted on two benchmark datasets and comparisons to state-of-the-art methods demonstrate the superiority of the proposed method.

We present a method based on deep learning for detecting and localizing abnormal/extreme events in sea surface temperature (SST) of the Red Sea images using training samples of normal events only. The method operates in two stages; the first one involves features extraction from each patch of the SST input image using the first two convolutional layers extracted from a pretrained convolutional neural network. In the second stage, two methods are used for training the model from the normal training data. The first method uses one-class support vector machine (1-SVM) classifier that allows a fast and robust abnormal detection in the presence of outliers in the training dataset. In the second method, a Gaussian model is defined on the Mahalanobis distances between all normal training data. Experimental tests are conducted on satellite-derived SST data of the Red Sea spanning for a period of 31 years (1985–2015). Our results suggest that the Gaussian model of Mahalanobis distances outperformed 1-SVM by providing better performance in terms of sensitivity and specificity.

With the development of video services available on various devices (mobile, PC, tablet, television, etc.), the end user experiences and requirements have changed. Objective video quality assessment tools and metrics are increasingly complex and adapted to different types of service context. Compared with other reviews of quality metrics, the full-reference (FR) video quality metrics reviewed (particularly, ViS3, SSIMplus, video multimethod assessment fusion, and open perceptual video quality metric) are more recent and have not been compared with each other yet. We compare and evaluate the performance of 10 FR metrics with respect to their accuracy in predicting the subjective perceived quality for the videoconferencing application. Emphasis is given to the metrics evaluating the visual quality (from human perception). A detailed statistical study is carried out on four subjective databases: École polytechnique fédérale de Lausanne database, live mobile database, and two Orange Lab databases, with a large sample of distortion types (transmission errors, encoding bit rates, etc.).

Dictionary learning for sparse representation has attracted much attention among researchers in image denoising. However, most dictionary learning-based methods use a single dictionary which has limitation in sparse representation ability. To improve the performance of this methodology, we propose a multichannel color image denoising algorithm based on multiple dictionary learning. Compared with a fixed dictionary, multiple dictionaries have more powerful representation ability. The algorithm first uses a Gaussian mixture model to model the generic patch prior of an external natural color image dataset. Then, the multiple orthogonal dictionaries are initialized with the generic prior by applying singular value decomposition to the covariance matrix of each Gaussian component. The sparse coding coefficients and the multiple dictionaries are alternately updated for better fitting the desired image. Considering the difference of the noise levels in RGB channels, we use a weight matrix to adjust the contributions of different channels for the denoised result. The desired image is estimated based on maximum a posteriori framework. The extensive experiments have demonstrated that our proposed method outperforms some state-of-the-art denoising algorithms in most cases.

Visual tracking and reidentification has gathered a lot of attention in real-world surveillance applications. We explore on-board visual tracking and reidentification of a person from a moving camera, such as a drone. We develop an android application which can communicate with the drone and transmit roll, pitch, and vertical upthrust instructions. We use a kernelized correlation filter tracker for tracking the person, and for the reidentification task we apply a deep three-dimensional ConvNet. The experiments demonstrate a robust performance.

Many fusion methods have been developed to improve the performance of action recognition with RGB and depth data, where learning conjoint representation of heterogeneous modalities by a single network has not been paid enough attention. We present an associated representation method for RGB-D action recognition using the siamese network with contrastive-center loss. First, some samples of each class and modality data are selected as the references to construct positive and negative pairs. Each positive pair consists of a training sample and its class reference, whereas the negative pair only involves different classes reference. Then these pairs are inputted to a two-stream siamese network to learn the collaborative representation of RGB and depth data. Two ranking losses, namely intramodal and cross-modal contrastive-center loss, are developed to impose similarity/dissimilarity metric on those pairs. Specifically, the intramodal contrastive-center loss measures the relationship between samples and references from RGB or depth data. The cross-modal contrastive-center loss measures the relationship of visual and depth features in a same low-dimensional space. Finally, the ranking losses and a softmax loss are jointly optimized for action recognition. The proposed method is evaluated on two large action datasets, LAP IsoGD and NTU RGB+D, and a smaller dataset, Sheffield Kinect gesture. The experimental results demonstrate that the proposed method surpasses most of the state-of-the-art methods.

A spatial domain multifocus image fusion method is proposed using a structure-preserving filter. In particular, the latest recursive filter (RF) is introduced as the structure-preserving filter in the proposed spatial domain method. Moreover, a focused region detection method is presented to determine initial weight maps based on an average low-pass filter. Then a fused image can be generated by the final weight maps, which are obtained using the RF to refine the initial weight maps and can well preserve the structures of source images. Experimental results show that the proposed method is superior to the state-of-the-art multifocus fusion methods in terms of subjective and objective evaluation.

An important issue in statistical modeling is to determine the complexity of the model based on the scale of data so as to effectively mitigate the model’s overfitting problems without big data. We adopt a data-driven approach to automatically determine the number of components of the model. In order to better extract robust features, we propose a framework of data-driven two-layer structure visual dictionary learning (DTSVDL). It works by dividing the visual dictionary structure learning into two levels: the attribute layer and the detail layer. In the attribute layer, the attributes of the image dataset are learned, and these attributes are obtained by a data-driven Bayesian nonparametric model. Then, in the detail layer, the detailed information over attributes is further explored and refined, and the attributes are weighted by the number of effective observations associated with each attribute. Our proposed approach has three main advantages: (1) the two-layer structure makes our building visual dictionary be more expressive; (2) the number of components in the attribute layer can be determined automatically from the data; (3) the components are automatically determined based on the scale of visual words; therefore, our model can well mitigate the overfitting problem. In addition, by comparing with stacked autoencoders, stacked denoising autoencoders, LeNet-5, speeded-up robust features, and pretrained deep learning model ImageNet-VGG-F algorithms, we find that our approach achieves satisfactory image categorization results on two benchmark datasets. Specifically, higher categorization performance is achieved than by the classical approaches on 15 scene categories and action datasets. We conclude that the resulting DTSVDL possesses a good generality derived from attribute information as well as an excellent distinction derived from detailed information. In other words, the visual dictionary learned by our algorithm is more expressive and discriminatory.

Zero-shot learning (ZSL) has recently attracted increasing attention in visual tasks like action recognition. We propose a spatiotemporal visual-semantic embedding network (STVSEM) for zero-shot action recognition. First, given the fact that two-stream architecture based action recognition algorithms have achieved excellent results in recent years, the module is assembled to our designed network by simultaneously using the spatial features (e.g., RGB appearance) and optical flow in time domain as visual features to significantly improve the visual expression capability. Then, in order to slightly alleviate the problem of semantic loss that typically occurs in the case of using embedding-based ZSL methods, an autoencoder is introduced to get a better semantic representation and complement semantic relationship information for unseen classes by seen classes. Last but not least, a joint embedding mechanism that explores and exploits the relationships of the visual data and semantic information in an intermediate space is employed to ameliorate the gap between vision and semantics. The experimental results on Charades and UCF101 datasets indicate that the proposed method outperforms the state-of-the-art methods in accuracy, which further demonstrates the effectiveness of our method.

As a result of people taking more and more pictures in their lives, image assessment technology, which can automatically help people choose high quality pictures quickly, has become particularly important. Most algorithms use peak signal-to-noise ratio (PSNR) to assess image quality. However, images with high scores on PSNR are not as beautiful as individuals think. Image aesthetic assessment technology can come closer to human aesthetic standards. We report on a method named saliency symbiosis network for image aesthetic assessment. This is significant because we improved the conventional convolutional neural networks (CNN) method, which gets very close to the human visual mechanism after adding saliency features in CNN. Owing to considering limitations of CNN input size, we also proposed a pooling strategy to improve the ability of the model to accept arbitrary input sizes. Afterward, we propose an effective mean Huber loss function, which becomes less sensitive to outliers and can quickly train the model to being optimal. The experiment results proved that the proposed method, by using very small training data, performed the highest accuracy in image aesthetic assessment and classification.

In the deep learning-based video action recognition, the function of the neural network is to acquire spatial information, motion information, and the associated information of the above two kinds of information over an uneven time span. We propose a network for extracting semantic information of video sequences based on the deep fusion feature of local spatial–temporal information. Convolutional neural networks (CNNs) are used in the network to extract local spatial information and local motion information, respectively. The spatial information is in three-dimensional convolution with the motion information of the corresponding time to obtain local spatial–temporal information at a certain moment. The local spatial–temporal information is then input into the long- and short-time memory (LSTM) to obtain the context relationship of the local spatial–temporal information in the long-time dimension. We add the ability of the regional attention mechanism of video frames in the neural network mechanism for obtaining context. That is, the last layer of convolutional layer spatial information and the first layer of the fully connected layer are, respectively, input into different LSTM networks, and the outputs of the two LSTMs at each time are merged again. This enables a fully connected layer that is rich in categorical information to provide a frame attention mechanism for the spatial information layer. Through the experiments on the three action recognition common experimental datasets UCF101, UCF11, and UCFSports, the spatial–temporal information deep fusion network proposed has a high correct recognition rate in the task of action recognition.

Image semantic segmentation understands and parses the image scene from the pixel-level. We work on the weakly supervised image semantic segmentation method on bounding box annotations. Inspired by Roberts edge detection operator, we develop an edge gradient module, which includes the max-pooling layer and average-pooling layer to extract edge gradient features. We combine the pyramid pooling (PP) module extracting global features with the atrous spatial PP module building the relationships between nodes to form long-range context dependency module. It can strengthen the ties between various parts of the object, just like a simplified fully connected conditional random field. To obtain more accurate feedback from bounding box annotations, we design a random label transformation algorithm. Finally, we demonstrate the validity of our module on PASCAL VOC 2012 and MS-COCO datasets, and our whole model has achieved a better performance than other mainstream methods.

With the continuous growth of the global population, large-scale public gatherings have become more common, and crowd management at these gatherings has become an urgent problem for public safety management. Crowd motion analysis and early warning based on crowd motion segmentation in video surveillance systems has become an important research topic in computer vision. A translation domain segmentation (TDS) model based on improved cosine similarity (ICS) is proposed to segment moving crowds with different crowding levels and complex motion modes. The method reconstructs the objective function of the basic TDS model by ICS to simultaneously measure the difference between both magnitude and direction of two vectors; thus, undersegmentation due to the magnitude difference can be avoided. By switching between “localization” and “globalization” modes of the objective function, the algorithm can be applied to segment crowds with different densities and motion states. Moreover, by simultaneously introducing local motion magnitude and local frame difference magnitude thresholds, nonforeground regions can be excluded from the initial regions during region evolution. Experimental results show that the proposed method achieves superior performance and higher accuracy compared to existing flow field-based methods when applied to complex scenes containing moving crowds.

The emergence of low-cost depth cameras creates potential for RGB-D based human action recognition. However, most of the existing RGB-D based approaches simply concatenate original heterogeneous features without discovering the latent relations among different modalities. We propose a discriminative common structure learning (DCSL) model for human action recognition from RGB-D sequences. Specifically, we extract deep learning-based features and hand-crafted features from multimodal data (skeleton, depth, and RGB). In particular, we propose a deep architecture based on 3-D convolutional neural network to automatically extract deep spatiotemporal features from raw sequences. The proposed DCSL model utilizes a generalized version of collective matrix factorization to learn shared features among different modalities. To perform supervised learning and preserve intermodal similarity, we formulate a graph regularization term by considering both label information and similar geometric structure of multimodal data, which intends to improve the discriminative power of shared features. Moreover, we solve the objective function using an iterative optimization algorithm. Then, an improved collaborative representation classifier is employed to perform computationally efficient action recognition. Experimental results on four action datasets demonstrate the superior performance of the proposed method.

Although existing research on classification of Chinese paintings is limited to consideration of the relationship between paintings and labels, we propose a convolutional neural network (CNN)-based feature description, feature-weighted, and feature-prioritized algorithm to achieve overwhelmingly better classification performances. In comparison with the existing research on Chinese painting classifications, where the distribution information of paintings is often ignored and the influence of the feature importance on the calculation of distribution information is not considered, we extract the features of Chinese paintings by CNN models and propose a joint standard and normalized mutual information to allow features being prioritized via their level of importance. Following that, an embedded machine learning is further integrated to formulate an embedded classification algorithm, namely joint mutual information-based and data-embedded classification (JMIDEC), and the support vector machine is finally applied as the classifier to optimize the classification results. Extensive experiments show that the proposed JMIDEC algorithm outperforms a number of representative methods with stronger robustness.

The production of edible bird’s nest (EBN) is an important income-generating agricultural activity in Malaysia. However, cleanliness inspection during EBN production poses certain difficulties due to issues such as labor dependency, human error, and inconsistency. Automated inspection remains a challenging task as there is no effective algorithm for impurities segmentation. Impurities, such as feathers, have similar color properties as EBN, making segmentation difficult due to the low contrast between the two features. We propose and demonstrate an experimental design-based lighting solution for improving the contrast ratio between the impurities and EBN in a scientific and systematic way. Lighting parameters, such as the configuration, angle, wavelength, and intensity of lighting, are investigated using a full factorial experimental design. The contrast ratio between the impurities and the EBN is chosen as the response variable in the experimental design. The main effects plot of the lighting parameters is obtained to identify the most optimal lighting parameters. The optimal lighting setup yields an improvement of 36.2%, with a contrast ratio of 0.79, compared to the normal fluorescent lighting setup, which has a contrast ratio of 0.58. This improvement would produce distinct intensity regions for impurities and EBN within an EBN image, therefore facilitating the segmentation operation.

Videos and images captured in poor weather conditions not only have seriously degraded visual quality, but they can also restrict the performance of many computer vision algorithms. One such severe weather condition is rain, which causes complex local intensity fluctuations and fuzzy background images. We study the single-image rain removal to improve the visual quality of images captured in rain. We propose a rain streak removal method with a double fidelity terms unidirectional variational model based on wavelet transform, which is utilized to separate the rain streaks. The rain streaks component is only included in the approximate component and the corresponding high-frequency components. The double fidelity terms unidirectional variational model is then used for removing the rain streaks. The regularization term mainly introduces image-prior information by unidirectional variation. The numerical algorithm is realized by the augmented Lagrange algorithm. Experimental results on a variety of simulated and real rain images show that the proposed method can efficiently remove rain streaks. Experimental results demonstrate the good performance of the proposed model, removing rain streaks while reducing information loss from the background. The proposed method also requires much less computation.

We propose a deep learning network L1-2D²PCANet for face recognition, which is based on L1-norm-based two-dimensional principal component analysis (L1-2DPCA). In our network, the role of L1-2DPCA is to learn the filters of multiple convolution layers. After the convolution layers, we deploy binary hashing and blockwise histogram for pooling. We test our network on some benchmark facial datasets, including Yale, AR face database, extended Yale B, labeled faces in the wild-aligned, and Face Recognition Technology database with the convolution neural network, PCANet, 2DPCANet, and L1-PCANet as comparison. The results show that the recognition performance of L1-2D²PCANet in all tests is better than baseline networks, especially when there are outliers in the test data. Owing to the L1-norm, L1-2D²PCANet is robust to outliers and changes of the training images.

With the development of intelligent transportation and parking, license plate detection in open environments is in great demand. However, due to the clutter of background and variation of license plates, the existing methods could not make a good balance between accuracy and efficiency. A method based on semantic region proposals is presented. By thinking from the pixel level, this method first adopts a semantic segmentation convolutional network for license plate candidate region extraction. To improve accuracy of segmentation, an enhanced loss function is designed. Afterward, a classification and regression network based on the oriented bounding box regression algorithm is used for region verification and refinement. Experiments on three public datasets show that the proposed method can be adapted to license plate images captured under different scenarios and can achieve better performance than the state-of-the-art methods.

The aim of multifocus image fusion technology is to produce an all-in-focus image, in which clear parts of different source images are integrated to a single image. Traditional image fusion methods usually suffer from some problems, such as block artifacts, artificial edges, halo effects, contrast reduction, and sharpness reduction. To address these problems, a multifocus image fusion method based on a convolutional neural network (CNN) is proposed. First, the CNN is trained using a large number of multifocus image samples to obtain a model that can correctly distinguish between clear and blurred pixels. Then the sharpness of the image to be detected is predicted using the model to form a focus map. After small-region filtering and guided filtering, a final decision map is formed. Finally, the multifocus source images are fused into a fully focused image according to the final decision map. Experimental results show that the proposed image fusion method outperforms other ones in terms of visual effects and objective evaluation.

In recent years, codetecting objects through the use of contextual information across multiple images has attracted considerable attention. We introduce an object codetection method that exploits contextual information among multiple images through a higher-order conditional random field (CRF). First, we obtain object candidates from each image of a test set by using a pretrained detector. Second, we feed the object candidates into a higher-order CRF that captures the appearance similarity using pairwise potentials and object category co-occurrence constraints using higher-order potentials. Finally, we jointly predict the category labels of all object candidates through the mean field inference in the CRF. Experimental results on the Caltech Pedestrian, PASCAL VOC 2007, PASCAL VOC 2012, and COCO datasets demonstrate the effectiveness of the proposed method compared to the baseline method.

Generative adversarial networks (GANs) are a class of techniques widely applied in image synthesis, recovery, compensation, and other related fields. We propose and introduce a novel, improved conditional model antimode collapse GANs (AMCGANs). Through a newly designed network architecture and optimizing strategy, the function of the class label information is moderately constrained and it no longer directly influences the discriminator, and hence the synthesized images belonging to the same classes will not be excessively concentrated due to the attraction of the same labels. Thus, the mode collapse problem, namely generating homogeneity, which always hinders image synthesization approaches can be effectively restrained. On the other hand, by sharing the feature extraction part and only updating its weights during the training of discriminator, AMCGANs achieves relatively efficient computing performance. In addition, it works well both for supervised and for semisupervised learning circumstances. Extensive experiments have been conducted on the Fashion-MNIST and CIFAR10 data sets to verify the effectiveness of the proposed approach.

A deep neural network-based image copyright protection scheme is presented. Instead of modifying the images like traditional watermarking methods, the proposed scheme trains a neural network to extract the robust features from image blocks and then classify them to represent the copyright message. First, the original image and its multiple attacked images are divided into nonoverlapping blocks, a part of which will be selected as candidate blocks. Then, the copyright message bits, together with candidate blocks, constitute the training dataset for the network, enabling the trained network to serve as a copyright message extractor. With the proposed scheme, no quality loss will be caused, and moreover, superior robustness can be achieved due to the adaptive robust feature extraction. Our study also offers further insight into the rationalities and considerations in design. Extensive experiments on a wide range of images show that the proposed scheme possesses strong robustness to many attacks, including additive noise corruption, JPEG compression, filtering, and resizing.

In long short-term memory (LSTM) neural networks, the input gates and output gates control information flowing into and out of memory cells. For sequence-to-sequence learning problems, each element is input into the network only once. If the input gates are closed at a certain step, the information is lost and is not input again. The same problem exists for the output gates. Therefore, the input and output gates do not fully support the roles of gating. An LSTM network with external memories, in which separate memories are installed for the input and output gates, is proposed. Information that is blocked by the input gates is preserved in the input memories, enabling the cells to read these memories when necessary. Similarly, information blocked by the output gates is preserved in the output memories and flows out to hidden units of the network at an appropriate time. In addition, a dynamic attention model is proposed to take into account the attention history. It provides guidance when predicting the attention weights at each step. The proposed model exploits attention-based encoder–decoder architecture to generate image captions. Experiments were conducted on three benchmark datasets, namely Flickr8k, Flickr30k, and MSCOCO, to demonstrate the effectiveness of the proposed approach. Captions generated by the proposed method are longer and more informative than those obtained with the original LSTM network.

Infrared and visible light image fusion is an important issue in the field of infrared technology. A variational model based on saliency preservation is proposed to help fusion process for infrared and visible light image by enhancing the saliency information of source images. The proposed variational model is composed by the weighted fusion term and the gradient fusion term. A weight function constructed by the saliency map of the source images is proposed to form the weighted fusion term. While in the gradient fusion, the target gradient is obtained by the maximum eigen-value and corresponding eigen-vector of the weighted structure tensor that also contains the salient information of the source images. Experimental results indicate that the proposed variational model better emphasizes the target information of the source images and outperforms other state-of-the-art fusion methods in both aspects of subjective visual and objective performance.

Recently, numerous methods have been proposed to tackle the problem of fine-grained image classification (FGIC). Most of them follow a two-step strategy that contains detecting the object regions and classifying with the features extracted from these regions. For the feature extraction, the most popular method is directly cropping the feature maps according to the location of detected part regions. However, one challenge of such a method is that the direction of the semantic parts may vary in different images, therefore, it is necessary to capture such differences for better classification. We propose a CNN architecture by aligning semantic parts (ASP-CNN) for FGIC, aiming to increase the interclass variance and meanwhile reduce the intraclass variance in fine-grained datasets. Extensive experiments on CUB-200-2011 and CUB-200-2010 show the effectiveness of our ASP-CNN.

The distinctiveness of image regions is widely used as the cue of saliency. Generally, the distinctiveness is computed according to the absolute difference of features. However, according to the image quality assessment (IQA) studies, the human visual system is highly sensitive to structural changes rather than absolute difference. Accordingly, we propose the computation of the structural dissimilarity between image patches as the distinctiveness measure for saliency detection. Similar to IQA models, the structural dissimilarity is computed based on the correlation of the structural features. The global structural dissimilarity of a patch to all the other patches represents saliency of the patch. We adopt two widely used structural features, namely the local contrast and gradient magnitude, into the structural dissimilarity computation in the proposed model. Without any postprocessing, the proposed model based on the correlation of either of the two structural features outperforms 11 state-of-the-art saliency models on three saliency databases.

Labanotation is one of the most widely used notation systems and plays a powerful role in recording and archiving traditional folk dance. We propose an end-to-end method for generating Labanotation from motion capture data by identifying the movement of each part of the human body and by assigning corresponding Labanotation symbols. Our method is mainly highlighted in the following aspects: first, we design simple yet highly discriminative skeleton features that can accurately represent human movements; and second, for the recognition of upper limb movements, we adopt fast and efficient extreme-learning neural networks, and for the recognition of lower limb movements, we employ powerful long short-term memory networks. It is worth mentioning that this is the first time that neural networks have been applied to the field of Labanotation generation. Experimental results show that our approach achieves much better recognition accuracy than previous work.

An iterative joint bilateral filter is used to obtain a natural weight map. Images from different modalities are merged by a weighted-sum rule in the spatial domain. Saliency maps are determined by the gradient of the pairwise raw images. Comparing the pairwise values of saliency maps, a coarse weight map is attained to determine which pixel is preferred. Since such a coarse weight map obtained by pairwise comparison is not a natural weight map subjectively, i.e., it is inconsistent with human visual system, the weight map is modified by using an iterative joint bilateral filter. With the iterative joint bilateral filter, the weight map becomes natural. We use the refined weight map to obtain the fused image and we seamlessly merge images from different modalities effectively. Experiments were conducted on several pairs of multimodal images to verify the effectiveness and superiority of the proposed image fusion algorithm compared to the state-of-the-art methods.

Disparity refinement based on a regression model continues to be challenging for specified function with a weak generalization ability. An invalid disparity refinement method to improve the quality of the disparity map is proposed. This method includes two committed steps: outliers disparity removal and redefinition of invalid pixels. To obtain a more accurate initial disparity map, removal of outliers based on Chauvenet’s criterion method is proposed, using the distribution of disparity values on a segmentation region. Then, the least square support vector machine model is applied to every horizontal line of the obtained initial disparity map to model the valid disparity values, corresponding image color values, and co-ordinates of pixels. Finally, invalid pixels are redefined by the regression model. Experimental results demonstrate that the dense disparity maps of the proposed method show superior performance compared with current state-of-the-art methods.

With the rapid development of digital video, shot boundary detection (SBD) has attracted much attention since it is the fundamental preprocessing for video indexing, annotation, retrieval, and other content-based operations. However, most state-of-the-art SBD methods are based on the spatial features of video image, and the overall characteristics of video shots are not fully considered. We propose a feature extraction method based on shot characteristics and a more robust SBD process. First, a video is divided into several segments, the segments containing consecutive video frames inside a shot are considered as training segments and others are called candidate segments. Afterward, using block-wise principal component analysis on the training segments, shot eigenspaces are established. The video frames in the candidate segment are then projected onto the corresponding shot eigenspace to extract the feature vectors. Finally, analysis and pattern matching for feature vectors are performed to extract the video shot boundary. Experiments on TRECVID test data demonstrate that the mean values of F1 in cut transition detection and gradual transition (GT) detection of our method are 0.901 and 0.866, respectively, obviously higher than the values of the compared methods, especially in GT detection, thus providing better accuracy in SBD.

In computer vision, the epipolar geometry embeds the geometrical relationship between two views of a scene. This geometry is degenerated for planar scenes as they do not provide enough constraints to estimate it without ambiguity. Nearly planar scenes can provide the necessary constraints to resolve the ambiguity. But classic estimators such as the 5-point or 8-point algorithm combined with a random sampling strategy are likely to fail in this case because a large part of the scene is planar and it requires lots of trials to get a nondegenerated sample. However, the planar part can be associated with a homographic model and several links exist between the epipolar geometry and homographies. The epipolar geometry can indeed be recovered from at least two homographies or one homgraphy and two noncoplanar points. The latter fits a wider variety of scenes, as it is unsure to be able to find a second homography in the noncoplanar points. This method is called plane-and-parallax. The equivalence between the parallax and the epipolar lines allows to recover the epipole as their common intersection and the epipolar geometry. Robust implementations of the method are rarely given, and we encounter several limitations in our implementation. Noisy image features and outliers make the lines not to be concurrent in a common point. Also off-plane features are unequally influenced by the noise level. We noticed that the bigger the parallax is, the lesser the noise influence is. We, therefore, propose a model for the parallax that takes into account the noise on the features location to cope with the previous limitations. We call our method the “parallax beam.” The method is validated on the KITTI vision benchmark and on synthetic scenes with strong planar degeneracy. The results show that the parallax beam improves the estimation of the camera motion in the scene with planar degeneracy and remains usable when there is not any particular planar structure in the scene.

Smile intensity estimation plays important roles in applications such as affective disorder prediction, life satisfaction prediction, camera technique improvement, etc. In recent studies, many researchers applied only traditional features, such as local binary pattern and local phase quantization (LPQ) to represent smile intensity. To improve the performance of spontaneous smile intensity estimation, we introduce a feature set that combines the saliency map (SM)-based handcrafted feature and non-low-level convolutional neural network (CNN) features. We took advantage of the opponent-color characteristic of SMs and the multiple convolutional level features, which were assumed to be mutually complementary. Experiments were made on the Binghamton-Pittsburgh 4D (BP4D) database and Denver Intensity of Spontaneous Facial Action (DISFA) database. We set the local binary patterns on three orthogonal planes (LBPTOP) method as a baseline, and the experimental results show that the CNN features can better estimate smile intensity. Finally, through the proposed SM-LBPTOP feature fusion with the median- and high-level CNN features, we obtained the best result (52.08% on BP4D, 70.55% on DISFA), demonstrating our hypothesis is reasonable: the SM-based handcrafted feature is a good supplement to CNNs in spontaneous smile intensity estimation.

Images captured from astronomical telescopes always have aliasing artifacts due to atmospheric turbulence, which typically conveys additional high-frequency content containing extra details of the astronomical scene. Super-resolution (SR) reconstruction can exploit the information existing in the aliasing to create a higher-resolution astronomical image, where the precise registration of input observations is generally regarded as a necessary premise. We consider SR reconstruction as a multichannel sampling scheme with uncertain deviations mathematically and present a frequency domain technique to achieve accurate alignment of aliased astronomical images by utilizing their low-frequency and aliasing-free part of the spectrum. Moreover, a planar motion model including three parameters is allowed to describe the relative uncontrolled motions between the acquisition system and the astronomical scene. Finally, a sharp high-resolution astronomical image can be reconstructed by utilizing the adaptive normalized convolution algorithm. The technique proposed is tested in simulations and practical experiments and shows very excellent SR reconstruction capabilities in the case of aliased astronomical images.

Crowd motions usually play a basic role of analyzing and understanding abnormal events. However, only using these visible features cannot fully describe the scenarios because the influence caused by an abnormal event is not considered from the social psychological point of view. Although the invisible influences cannot be directly observed through the video, they objectively exist and have precise definitions and specific analytical methods in sociology and psychology. Therefore, a mid-level representation named global event influence (GEI) for global anomaly detection in dense crowds is introduced. The proposed GEI integrates the crowd motions and social psychology attributes to improve the description of crowds. For this, low-level motion features are abstracted as crowd attributes of scale, velocity, and disorder. Then, the detailed definitions and mathematical expressions of GEI are presented through calculating the convolution of rise factor and decay factor. Based on GEI, a model for global anomaly detection is proposed. Compared with most previous methods, our proposed model is robust to detect not only the occurrence of anomalous events but also elimination time of the event influence. Accordingly, strategies for event occurrence detection and influence elimination detection are proposed, respectively. In addition, a dataset of dense crowds is introduced and used for evaluation. The experimental comparison on benchmark datasets shows that the performance of our GEI model has not only the competitive accuracy of event occurrence detection but also the claimed effectiveness of influence elimination, which is more advanced than others.

Different turfs have different growth characteristics, engendering differences in the number of maintenance cycles and amounts of pesticides used; therefore, studying their subtle color and shape differences through image recognition is crucial. Our study proposes an improved least squares support vector machine (LS-SVM) pixel classification method for this purpose. The sensitivity to local color changes in the hue, saturation, and value color space is considered, and the Sobel operator is used to extract the homogeneity as pixel-level color features. The maximum local energy, gradient, and second-order moment matrix of image pixels are obtained as texture features using a Gabor filter. Seven shape features of different plant leaves are calculated, multiple extracted features are used as LS-SVM classifier inputs, and samples are selected and trained with a dynamic threshold. The trained classifier can be used for segmentation. The experiments showed that it could use the local information of the color images and the excellent generalization ability of LS-SVM to segment lawn plants effectively. Under different weather conditions, the penalty coefficient, C, and kernel parameters with optimal generalization were Bayesian optimized to obtain a segmentation rate exceeding 95%. This algorithm yields a higher classification rate for plants with less obvious differences in texture and shape and optimizes space and time complexities.

Deep convolutional neural networks have shown prominent performance in stereo matching. However, current network architectures lack performance in exploiting context and global information to finding corresponding points in ill-posed regions. A stereo matching network without postprocessing is proposed to solve this problem. This network combines the improved multilevel feature pyramid pooling module with the light two-dimensional (2-D) convolution subnetwork to efficiently utilize multilevel information and global information. In the multilevel feature pyramid pooling module, the base image feature is extracted by cascading three small convolution filters. Features of a stereo image pair are calculated by hierarchically fusing and pooling features information of the same scale after using the residual network. Multilevel semantic information is fully utilized to improve the robustness of image feature representation in multilevel feature pyramid pooling module. In the light 2-D convolution subnetwork, low-level structural information is obtained from the target image by three concatenated convolution layers with small convolution filters. Low-level information is used to rectify matching cost with global view to improve matching accuracy. The experimental results on the Scene Flow dataset, the MPI Sintel dataset, and the Middlebury dataset show that the performance obtained by the proposed network can be improved in the ill-posed regions. Matching accuracy is competitive compared to other results obtained by end-to-end networks without postprocessing.

Video-based person reidentification is a challenging and important task in surveillance-based applications. Toward this, several shallow and deep networks have been proposed. However, the performance of existing shallow networks does not generalize well on large datasets. To improve the generalization ability, we propose a shallow end-to-end network which incorporates two stream convolutional neural networks, discriminative visual attention and recurrent neural network with triplet and softmax loss to learn the spatiotemporal fusion features. To effectively use both spatial and temporal information, we apply spatial, temporal, and spatiotemporal pooling. In addition, we contribute a large dataset of airborne videos for person reidentification, named DJI01. It includes various challenging conditions, such as occlusion, illuminationchanges, people with similar clothes, and the same people on different days. We perform elaborate qualitative and quantitative analyses to demonstrate the robust performance of the proposed model.

As an effective and efficient way to graphically portray ideas and concepts, free-hand sketches are playing an increasingly significant role in today’s image retrieval systems. There are many factors that contribute to a successful sketch-based image retrieval (SBIR) system, but perhaps the most essential ones are the design of powerful image feature descriptors and effective perceptual similarity metrics in the feature space. However, the painting experiences and styles of users vary hugely, which poses a great challenge for consistently accurate feature representation and matching. We introduce an approach for SBIR by characterizing homogeneous sketch drawing styles. Specifically, given a sketch query, the proposed method first infers the user’s painting style from it by analyzing contour features. Then, top K-nearest sketches of historical users with similar drawing characters are identified from the database. Finally, the matched images of the query can be obtained by reranking those of the K-nearest sketches stored in the historical recordings. To evaluate the effectiveness of the proposed method, we conducted quantitative comparisons with several peer SBIR methods. Experimental results indicate that the proposed method is 15% higher than state-of-the-art in terms of mean average precision.

Content-based image retrieval (CBIR) systems use multiple image features to represent an image. These systems suffer from curse of dimensionality since a high-dimensional feature vectors are formed to represent images in the target dataset. Reduction in length of feature vector can speed-up the retrieval process, but it reduces the retrieval accuracy. To overcome the aforementioned problem, an efficient layer-wise feature incremental approach for the CBIR system has been proposed. The proposed approach uses three primitive image features namely color, texture, and shape. The retrieval process is accomplished in three layers, at first a layer-complete dataset is searched but only single-feature space is used. Top 10% of images most similar to query image are retained in the second layer. The second layer uses two features for similarity computation, and only 50% of the most similar images are passed to the third layer. Finally, the third layer uses all three features to compute the similarity. Our aim is to reduce the search space at subsequent layers and use multiple features for a reduced dataset at the final layer. The proposed approach is evaluated on four publicly available image datasets. The retrieval results on the basis of precision, recall, and f-score show the performance improvement in comparison to state-of-the-art CBIR systems.