In this paper, a new stereo object-tracking system is proposed in which the variable window mask and the optical binary phase extraction joint transform correlator (BPEJTC) are used. Firstly, using the distance information from the stereo camera to the tracking object, the area of the tracking object is digitally extracted through a variable window mask. And, at the second step, by carrying out an optical BPEJTC between the reference image obtained from a variable window mask and the stereo input image, the coordinates of the tracking object's location are acquired, and then with these values, the convergence angle and the pan/tilt of the stereo tracking camera can be finally controlled. Some experiments show that the proposed system is able to effectively extract the area of the target object from the input image having the background noises and easily control the convergence angle and the pan/tilt of the stereo cameras with the obtained location values of a tracking object. From these experimental results, a feasibility of real-time implementation of the adaptive stereo object tracking system using the proposed algorithm is also suggested.