In this paper, we describe a method for estimating and compensating 3D camera motion in image sequences for applications to video coding systems. To improve the coding efficiency, a two-stage motion compensation (MC) method is used, consisting of global MC for predicting camera motion and local MC for predicting object motion. For global MC, a new linear motion parameter model is presented to describe the camera motion parameters: zoom, pan, tilt, swing, and focal length. To estimate camera motion parameters based on the proposed model, the mixed least squares-total least squares (LS-TLS) method is used. The proposed estimation procedure consists of feature correspondence establishment, finding the parameters by fitting the correspondence data to the proposed model equation, and outliers rejection to reduce feature matching error. Unlike the existing linear techniques, the proposed method accurately estimates even the large rotation angles and the focal length. Experimental results show that the proposed method outperforms the conventional methods under identical conditions, especially for large rotation images.