Vision-based motion tracking is commonly used in surveillance, human-machine interfaces in robotics and automation, virtual and augmented reality applications and biomechanics. Most techniques require markers, use a predefined motion sequence or user-intervention for initialization, and do not process in real-time. This paper describes the implementation of a vision-based non-invasive technique for markerless real-time tracking of human-arm motion. Human-arm motion is tracked by processing images from two calibrated cameras in real-time to estimate the position of the 3D joint centers of the wrist and elbow, and determine the orientation of the hand from the 3D positions of the index finger and thumb. Tracking of the hand and arm was carried out without any prior knowledge of subject's arm length, texture, width and distance from the camera.
The control of a robot manipulator by a human operator is often necessary in unstructured dynamic environments with unfamiliar objects. Remote teleoperation is required when human presence at the robot site is undesirable or difficult, such as in handling hazardous materials and operating in dangerous or inaccessible environments. Previous approaches have employed mechanical or other contacting interfaces which require unnatural motions for object manipulation tasks or hinder dexterous human motion. This paper presents a non-contacting method of teleoperating a robot manipulator by having the human operator perform the 3D human hand-arm motion that would naturally be used to compete an object manipulation task and tracking the motion with a stereo-camera system at a local site. The 3D human hand-arm motion is reconstructed at the remote robot site and is used to control the position and orientation of the robot manipulator end-effector in real-time. Images captured of the robot interacting with objects at the remote site provide visual feedback to the human operator. Tests in teleoperation of the robot manipulator have demonstrated the ability of the human to carry out object manipulator tasks remotely and the teleoperated robot manipulator system to copy human-arm motions in real-time.