This paper addresses the problem of removing spatially varying blur caused by camera motion with the help of inertial measurements recorded during exposure time. By utilizing a projective motion blur model, the camera motion is viewed as a sequence of projective transformations on the image plane, each of which can be estimated from the corresponding inertial data sample. Unfortunately, measurement noise leads to temporally increasing drift in the estimated motion trajectory and can significantly degrade the quality of recovered images. To address this issue, this paper employs capturing a small sequence of images with different exposure settings along with the recorded inertial data. A special arrangement of exposure settings is designed to anchor the correct position of the camera trajectory, followed by a drift correction step, which makes use of the sharp image structures preserved in one of the captured images. The effectiveness of our approach is demonstrated by conducting comparison experiments on both synthetic images and real images.