In this paper, we proposed a stereo object tracking system that can control the convergence angle and pan/tilt of cameras by using optical binary phase extraction joint transform correlator (BPEJTC) and can extract the tracking object from a complex background and foreground noises by using the block matching-based window mask. It is used to perceive and extract the tracking object from the foreground and complex background by using window mask of the block matching-based SAD(sum of absolute difference), and by using the optical BPEJTC of the phase type, which has improved the correlating properties of the conventional optical JTC(joint transform correlator) with the adaptive object tracking ability, the position values of moving target on the left and right images can be calculated. And, we can be controlling the convergence angle and pan/tilt of cameras by using this values. Therefore, real time stereo object tracking system, which could adapt to the changes in surrounding, can be implemented. From the experimental results, the proposed stereo tracking system is found to track the object adaptively under the complex circumstances and changing background noises and the possibility of real-time implementation of the proposed system by using the optical system is also suggested.