In visual tracking, the object's appearance may change over time due to illumination changes, pose variations, and partial or full occlusions. This variability makes tracking difficult. This paper proposes an adaptive appearance model for visual tracking. The model can adapt to changes in object appearance over time. The value of each pixel is modeled by a Gaussian mixture distribution. A novel update scheme based on the expectation maximization algorithm is developed to update the appearance model parameters. In designing the tracking algorithm, the observation model is based on the adaptive appearance model, and a particle filter is employed. Outlier pixels and occlusions are handled using a robust-statistics technique. Numerous experimental results demonstrate that the proposed algorithm can track objects well under illumination changes, large pose variations, and partial or full occlusions.