1.

INTRODUCTION

In recent years, Deep Learning (DL) shows us powerful capabilities for image processing. But it cannot output exact photometric process parameters, such as the 6-DoF object position, and it shows non-interpretable results. Due to such reasons this paper presents a robot vision system based on Convolutional Neural Networks (CNN) and Monte Carlo algorithms. As an example to discuss about how to apply DL in industry. A robust, efficient and accurate 3D object detection system could show potential solutions for human-machine collaboration systems in future. The approach solves the 3D object detection problem in real time 6-DoF pick-and-place operations for a wide variety of objects in clutter. First thanks to PointNet++, keypoints of target objects are extracted from their template point clouds. Since the difficulty and time cost of point cloud analysis a CNN based on YOLO architecture is used to segment the scene point cloud. Then outliers will be removed using a cluster analysis based on Euclidean distance. After such preprocessing objects are tracked using a particle filter approach. Distance of nearest point, color feature and geometric feature are used as three criteria for Maximum Likelihood Estimation to find the 6-DoF position of objects. An experiment was designed to verify, due to the new approach the difficulty within point cloud analysis can be faster and easier realized. Another experiment will show that the three criteria used for Maximum Likelihood Estimation are essential. In the end the efficiency (20 fps) and accuracy (mean error of position: 3.57 mm and orientation: 0.103 rad) of 6-DoF position detection and success rate (97.3%) of object grabbing in a concrete pick-and-place application will be shown.

2.

RELATED WORK

In the past 20 years the research on object detection can be divided into two categories: traditional methods and novel neural network approach. Many researches are on manually defined feature or descriptor for 2D image analysis, e.g.: [1][2][3]. For the field of 3D feature someone has made such studies: [4][5][6][7]. Image processing based on multispectral images is generally used for the analysis of Remote Sensing image [8][9]. Others someone have studied on edge detection based on multispectral image analysis [10]. All the above studies are on the features or descriptor. Then with the help of feature map and some functional modules (such as classifier SVM) can be some functional applications realized: e.g.: [11][12][13].

In recent years due to the rise of neural network development many researches based on this novel method have emerged. [14][15] are the researches on 3D object detection based on RGB-D image processing. For 3D object detection tasks can be solved by the neural network based on bird view from Lidar view or point cloud from stereo camera [16][17][18].

It is worth mentioning that three papers of Ross Girshick [19][20][21]. In these papers he studies to combine the traditional method with neural network to solve the task of 2D object detection. Then in the researches [22][23] his idea has extended to more efficient methods. Inspired by them, in this case we also discuss to perform the 3D object detection based on classic method and neural network approach.

3.

METHOD OVERVIEW

In this section we explain the 3D object detection system based on deep learning approach and classic method. This system is to support a pick and place robot. Due to the combination of the two approach the difficult problems of 3D image processing are solved.

Because the computational cost of point cloud processing is large. As shown in Figure 1 our system consists of three functional modules: (1) Target point cloud creation, (2) Scene point cloud segmentation and (3) 6DoF object detection. The first two modules are offline and online preprocessing, respectively. Some deep learning approach are used in these two modules. Then the third module is the core functional block. In this module we have used classic method to find the exact photometric process parameters (6DoF object position).

Figure 1.

3D object detection system overview

3.2

Scene point cloud segmentation

The important two aims of this part are: (1) Classification of objects in scene, (2) Rough object localization (bounding box). Thanks to YOLO2 [22], based on the analysis of color images the object will be segmented from scene. Then only the pixels in the bounding box will be converted to point cloud, of course with classification information. Finally, outliers can be segmented by Euclidean clustering, as shown in figure 2.

Figure 2.

2D, 2.5D and 3D Preprocessing

4.

EXPERIMENTS

Our experiment environment is based on a pick and place robot application. Due to hand eye calibration approach the position and orientation (pose) of the camera frame with respect to the robot frame is identified. Then the pose of the object in the camera frame is obtained by our 3D object detection system. Finally, the pose of the robot grasper for target object grasping is calculated. As shown in figure 3, we have three different rigid body objects as test object.

Figure 3.

Experiment environment

5.

CONCLUSION

With the help of two deep learning approach (PointNet++, YOLO) the efficiency, stability and robustness are greatly improved. A method that deep learning in industrial image processing applied is proposed: preprocessing (segmentation) and offline tasks (keypoint extraction). Classic method such as particle filter approach is used to compensate for the disadvantage of deep learning. Finally, traditional methods and novel deep learning combine to bring the optimal performance for this 3D object detection system.