1.

Introduction

In the field of computer vision, object detection has long been a challenging and important task. State-of-the-art algorithms deliver satisfactory results for two-dimensional images. Nonetheless, these methods suffer from limited variation and cluttered backgrounds.¹ Highly accurate RGB-D cameras that have recently been developed can easily provide high-quality three-dimensional (3-D) information (color and depth information).² Objects can thus be examined by acquiring color and depth information together, which is better than using only raw color images to learn feature representations.³

In order to solve the problems of rotation and scale, part-occlusions, and nonrigid transformations, recent RGB-D object detectors have featured the following tools:⁴ feature extraction using a rotation-invariant descriptor,⁵ part-based coding scheme using the generalized Hough transform,⁶ feature matching using machine learning frameworks,⁷ and so on.

A number of researchers have paid attention to object representation through depth images in order to improve detection performance, such as the incorporation of information regarding the shape and spatial geometry of objects.⁸ A representative spare feature is the integration of 3-D coordinates with the color fast feature.⁵ However, the computational cost of local feature extraction and matching increases with the number of classifiers. Hence, this paper involves the computation of a fast RBG-D local binary feature (LBF) in polar coordinates, which have yielded remarkable results for object categorization under challenging conditions such as rotation variation and cluttered backgrounds. This is because it is easier to rotate the coordinates of the descriptor at a certain polar angle relative to the patch orientation in a polar coordinate system.

The generalized Hough transform has been successfully adapted to the problem of part-based object detection because it is robust against partial occlusion and slightly deformed shapes.⁹ Moreover, it is tolerant to noise, and can find multiple occurrences of a shape in the same processing pass. Its main disadvantage is that it requires a considerable amount of storage and extensive computation. However, it has been reported that Hough voting efficiency during object categorization can be improved using a highly efficient classifier.¹⁰

With regard to invariant Hough random ferns (IHRF),¹¹ this paper applies a random ferns classifier (RFC)¹² to a Hough transform to improve search speed and reduce the need for a large storage space for data. Furthermore, the Hough voting is performed in rotation-invariant Hough space, since each support point shares a stable polar angle and scalable displacement related to the center of the relevant object.

This paper is structured as follows: the framework for RGB-D object detection is presented in Sec. 2. Experimental results, including a comparison of the proposed method with state-of-the-art techniques, are provided in Sec. 3. The contributions of this paper and ideas for future research are discussed in Sec. 4.

2.

Methodology

This section describes the procedure for RGB-D object detection based on IHRF. Figure 1 outlines the procedure for this approach.

Fig. 1

An overview of training, represented by “a,” and detection, represented by “b.” (a-1) Positive samples. (a-2) Depth value recording. (a-3) Local coding. (a-4) Negative samples. (a-5) Feature extraction. (b-1) Corner detection. (b-2) Scale transform. (b-3) Feature extraction. (b-4) RFC. (b-5) Hough voting. (b-6) Finding local maxima in 2-D Hough space. (b-7) Back projection. (b-8) Detection results.

Figure 1 formulates the rotation-invariant and multiscale object detection problem as a probabilistic Hough voting procedure. For this example, the IHRF is trained on images of coffee cups (color and depth) obtained from the RGB-D Object Dataset.¹³ Some positive [Fig. 1(a-1)] and negative [Fig. 1(a-4)] samples including color and depth images were provided for training. The depth value of the positive samples in their modeling center should then be recorded as $d_0$ [Fig. 1(a-2)]. For the positive image, the system extracts a large number of scanning windows within the color and depth images, and forms the local coding [Fig. 1(a-3)]. Following this, the rotation-invariant LBF [Fig. 1(a-5)] is extracted and used to train the RFC [Fig. 1(b-4)]. When presented with the image for detection, the system extracts a large number of interest points [Fig. 1(b-1)] and computes the scale value of the scanning windows, which is equal to the depth value $d_i$ at the time divided by the original depth value $d_0$ [Fig. 1(b-2)]. The rotation-invariant LBF is then extracted in these scanning windows [Fig. 1(b-3)] and used for instance recognition [Fig. 1(b-4)]. The local scanning windows then cast probabilistic votes containing the locations of the centroid of the objects, which are collected in the voting space [Fig. 1(b-5)]. As a visualization of this space in Fig. 1(b-6) shows, the system searches for local maxima in the voting space and returns the correct detection as the strongest hypothesis. By back-projecting the contributing votes [Fig. 1(b-7)], the system retrieves support for the hypothesis in the image [Fig. 1(b-8)], and roughly separates the location of the object from the background. All the key steps are described in detail in subsequent sections.

2.1.

Rotation-Invariant RGB-D Local Binary Feature

Random fern descriptors, also called LBF, consist of some logical pairwise comparisons of the intensity or gradient levels of randomly selected pixels in images.¹⁴ However, such comparisons are not robust against rotation and scale variations because each pairwise pixel is randomly generated offline while remaining fixed in runtime. Therefore, a rotation-invariant descriptor with a high degree of stability in RGB-D images is defined as

Eq. (1)

$$f(i,j,n)=\{\begin{array}{ll}1& I_n(x_i,y_i)>I_n(x_j,y_j)\\ 0& I_n(x_i,y_i)\le I_n(x_j,y_j)\end{array},d(i,j)=\{\begin{array}{ll}1& I_D(x_i,y_i)>I_D(x_j,y_j)\\ 0& I_D(x_i,y_i)\le I_D(x_j,y_j)\end{array},$$

where $I_n(x,y)$ and $I_D(x,y)$ are the $n$’th ($n\in [\mathrm{1,16}]$) feature channel¹⁵ obtained from a color image and the $n$’th depth channel obtained from the depth image, respectively, with both centered at pixel locations $x$ and $y$. $(x_i,y_i)$ and $(x_j,y_j)$ are random pairwise pixel locations, and each comparison returns 0 or 1. In general, the pairwise pixels $S$ that are chosen map an image patch to a $2^{2S}$-dimensional space of binary descriptors in each fern. According to Eq. (1), RGB-D LBF can be computed as

Eq. (2)

$$F_m={\{f_1,f_2,\dots ,f_S,d_1,d_2,\dots ,d_S\}}_2,$$

where $F_m$ is the $m$’th fern and $f$ (or $d$) is the $i$’th binary feature. Therefore, the entire set of random ferns can be denoted by $F=\{F_1,F_2,\dots ,F_K\}$. A trade off between performance and memory can be made by changing the number of ferns $K$ and their sizes $S$.

To achieve orientation invariance, the pairwise pixels used in Eq. (1) can be calculated by the polar coordinates

Eq. (3)

$$\{\begin{array}{l}{x}_1=R_1\cos ({\theta }_1+{\theta }_{\mathrm{m}})\\ y_1=R_1\sin ({\theta }_1+{\theta }_{\mathrm{m}})\end{array},\{\begin{array}{l}{x}_2=R_2\cos ({\theta }_2+{\theta }_{\mathrm{m}})\\ y_2=R_2\sin ({\theta }_2+{\theta }_{\mathrm{m}})\end{array},$$

where gradient orientations are rotated relative to the maximum gradient orientation (MGO) ${\theta }_{\mathrm{m}}$, and the polar coordinates $R_i$ and ${\theta }_i$ can be converted to Cartesian coordinates $x_i$ and $y_i$, respectively, by using the trigonometric functions sine and cosine.

Note that the fixed pole is located at the center of the image, and the fixed polar axis has the same direction as the MGO. This allows pairwise pixels to be matched correctly under arbitrary orientation changes between the two images. Therefore, by assigning a consistent orientation to each LBF based on local image properties, the rotation-invariant RGB-D LBF can be represented simply relative to this orientation and, therefore, be rendered invariant to image rotation. Furthermore, by reserving typical features and reducing redundancy features, satisfactory generalization performance and training efficiency of the classifier are guaranteed. Figures 2(a) and 2(b) show the results of rotation-invariant RGB-D LBF on color and depth images.

Fig. 2

The results of rotation-invariant RGB-D LBF on (a) color and (b) depth images.

2.2.

Training the Random Ferns Classifier

Random ferns are of great interest in computer vision because of their speed, parallelization characteristics, and robustness against noisy training data. They are used for various tasks, such as keypoint recognition and image classification.¹² When they are applied to a large number of input vectors of the same class $C$, the output of each fern is a frequency distribution histogram, which is shown in Fig. 3. In the histogram, the horizontal axis represents a $2^S$-dimensional space of binary descriptors and the vertical axis represents the number of times the binary code appears in class $C$, also called the class conditional probability $p(F_i/C)$, where $i\in [1,K]$.

Fig. 3

Classification using an RFC, where “$\times$” is the symbol for multiplication.

Random ferns replace trees in random forests (RFs) with nonhierarchical ferns and pool their answers in a naive Bayesian manner to yield better results and improve classification rates in terms of the number of classes. As discussed in Sec. 2.1, the set of $F({\theta }_{\mathrm{mt}})$ located in a local patch with the MGO ${\theta }_{\mathrm{mt}}$ is regarded as a class. Thus, a randomly selected patch detected in another image is assigned to the most likely class $\mathrm{C͡}$ by calculating the posterior probability

Eq. (4)

$$\mathrm{C͡}=\underset{k}{\mathrm{argmax}}\prod _{L=1}^{K}p(F_L({\theta }_{\mathrm{mt}})/C_k).$$

For a given a test input, one can simply apply the binary representations to account for the ferns and look up the corresponding probability distribution over the class label, as shown in Fig. 3. Finally, the RFC selects the class with the highest posterior probability as the categorized result. RFC is a remarkable classification algorithm that randomly selects and trains a collection of ferns. In this way, classifying new inputs involves only simple lookup operations.

3.

Experiments and Results

This experimental section evaluates our method’s performance and compares it with that of state-of-the-art approaches. It applies these methods to the challenging RGB-D Object Dataset¹⁷ and adheres to the experimental protocols and detection accuracy criteria established for each of the datasets in previous works. All experiments were conducted on a standard 3.2 GHz PC with 2 GB of RAM. For IHRF, the settings were as follows: the RFC consisted of $K=10$ ferns, and picked $S=13$ pairwise pixels for RGB-D LBF. Note that while using more ferns achieves higher recognition rates, it also requires more memory to store the distributions and results in a higher computational cost.¹² So, the Fern size of 10 and 13 features used for each Fern have proved to be a good compromise in our recent experiments.¹¹ The RGB-D Object Dataset¹⁷ is a large dataset of 300 common household objects, eight of which were used for training and detection, as shown in Fig. 4.

Fig. 4

Eight RGB-D objects used for training and detection. (a) Cap_1. (b) Bowl_4. (c) Flashlight_2. (d) Flashlight_5. (e) Cereal_box_2. (f) Coffee_mug_5. (g) Soda_can_1. (h) Soda_can_6.

The goal of the experiment was to assess the accuracy of instance recognition using our method. According to the leave-sequence-out method,¹⁷ the first experiment involved training on the video sequences of each object, where the camera was mounted at 30 deg and 60 deg above the horizon, and evaluations on a 45-deg video sequence. The eight RGB-D objects shown here formed the largest multiview dataset, where both RGB and depth images were provided for each view. Therefore, this part tested whether combining RGB and depth is helpful when well-segmented or cropped images are available.

The recall-precision curve (RPC; see Fig. 5) was generated by changing the probability threshold on the vote strength of the hypothesis. Table 1 lists a performance comparison with the recognition results obtained by RGB, depth, and RGB-D images, respectively.

Fig. 5

RPC on eight RGB-D objects. All curves were generated by RGB, depth and RGB-D images, respectively. (a) Cap_1. (b) Bowl_4. (c) Flashlight_2. (d) Flashlight_5. (e) Cereal_box_2. (f) Coffee_mug_5. (g) Soda_can_1. (h) Soda_can_6.

Table 1

Performance of RGB, depth, and RGB-D images on eight objects in terms of recall-precision equal error rate (EER: %).

As shown in Fig. 5 and Table 1, RGB images attained an 87.4% EER, which was better than that for depth images (73.5%). This means that RGB images are more useful than depth images for instance-level recognition. This result showed that objects can easily use different textures and colors to distinguish among one another. The RGB-D approach achieved an impressive 93.7% EER for the objects, outperforming both RGB and depth. Hence, the most significant conclusion is that combining RGB and depth images yields better performance. The leave-sequence-out evaluation was much more challenging and showed that combining shape and visual features significantly improves accuracy.

The second part of the experiment related to a set of quantitative experiments comparing our method with other relevant algorithms, including linear support vector machine (LinSVM),¹⁷ RF,¹⁷ Gaussian kernel SVM (kSVM),¹⁷ RGB-D kernel descriptors,¹⁸ and hierarchical matching pursuit.¹⁹ Table 2 shows a comparison between the EER obtained using our method with IHRF and the results of other methods on the RGB-D dataset. All EER results were consistent with the first set of conclusions, whereby RGB-D images outperformed RGB and depth images regardless of classification technique. As can be seen from the results, our method attained impressive detection results for RGB-D images with an EER performance of 93.7%, which presents an improvement over results of past approaches. This situation is the same as the depth’s EER 73.5% since the use of rotation-invariant LBF. Note that the EER of RGB images was 87.4%, which is slightly less than the corresponding values for the fourth and the fifth methods. This was because the RFC was $K=13$, which refers to a trade off between performance, and memory and computational cost. If the value of $K$ increases, performance improves efficiently.

Table 2

Comparison of different methods on the eight datasets using RGB, depth, and RGB-D images (EER: %).

Some examples of multi-instance detection are shown in Fig. 6. The results show that the RGB-D-based IHRF not only detects the object despite partial occlusion but also can often even deal with rotation variations and large perspectival changes. In Table 2, the first four methods all take approximately more than 2.5 s to label each scene from the RGB-D dataset, and the running time for hierarchical matching pursuit is 0.5 s. However, our method only required 0.4 s (no less than $200\times 150\text{pixel}$ resolution) for object recognition. For conventional methods, the required amounts of memory increase linearly with the number of samples for each class. By contrast changing joint probability for features in each Fern does not result in increasing memory usage.

Fig. 6

The detection results on RGB-D datasets including the (a) first, (b) second, and (c) third scene.

4.

Conclusions

This paper proposed a 3-D object detection method using RGB-D images based on IHRF. It relies on a rotation-invariant LBF based on RFCs that can cast probabilistic votes within the Hough transform framework. Experiment results show that such RGB-D-based IHRF can be efficiently used to detect instances of classes in large, challenging images with an accuracy that is superior to that of previous methods, and achieves the best results. This approach also allows for an efficient implementation in terms of time and space in comparison with related techniques. In the future, the authors intend to use the recognition results to improve the precision of object segmentation.