This paper presents the sensor planning strategy in a robot vision system in which 3D information of the object is obtained from different view points by the structured light sensor. Sensor planning strategy has a great influence on the efficiency and reliability of the vision system. Both the 3D data obtained from the object in the scene and the vision model provide important information for the recognition and localization of the object. The sensor planning module is responsible for making good use of this information in finding the next view point so that the most useful information can be acquired as soon as possible. Rules are made and adopted in the sensor planning module to identify the object or to determine its position. Different surface types of the objects have different features that need special measures to be detected. The rules are designed to deal with variations of the object's appearance so that the vision system can go ahead to the right solution. The sensor planning strategy works in two steps: first it analyzes what features are to be detected according to the model guidance and the current hypotheses of the object, next it tries to select the proper rule to estimate where to find the predicted features and the corresponding next view point of the sensor.