Omnidirectional vision with the advantage of a large field-of-view overcomes the problem that a target is easily lost due to the narrow sight of perspective vision. We improve a target-tracking algorithm based on discriminative tracking features in several aspects and propose a target-tracking algorithm for an omnidirectional vision system. (1) An elliptical target window expression model is presented to represent the target’s outline, which can adapt to the deformation of an object and reduce background interference. (2) The background-weighted linear RGB histogram target feature is introduced, which decreases the weight of the background feature. (3) The Bhattacharyya coefficients-based feature identification method is employed, which reduces the computation time of the tracking algorithm. (4) An adaptive target scale and orientation measurement method is applied to adapt to severe deformations of the target’s outline. (5) A model update strategy is put forward, which is based on similarity measurements to achieve an effective and accurate model update. The experimental results show the proposed algorithm can achieve better performance than the state-of-the-art algorithms when using omnidirectional vision to perform long-term target-tracking tasks.