Recently, pan-tilt-zoom(PTZ) camera is widely used in extensive-area surveillance applications. A number of background modeling methods have been proposed within existing object detection and tracking systems. However, conventional background modeling methods for PTZ camera have difficulties in covering extensive field of view(FOV). This paper presents a novel object tracking system based on a spherical background model for PTZ camera. The proposed system has two components: The first one is the spherical Gaussian mixture model(S-GMM) that learns background for all the view angles in the PTZ camera. Also, Gaussian parameters in each pixel in the S-GMM are learned and updated. The second one is object tracking system with foreground detection using the S-GMM in real-time. The proposed system is suitable to cover wide FOV compared to a conventional background modeling system for PTZ camera, and is able to exactly track moving objects. We demonstrate the advantages of the proposed S-GMM for object tracking system using PTZ camera. Also, we expect to build a more advanced surveillance applications via the proposed system.