Handheld electro-optical imaging devices usually suffer from shaky problems. In this paper, we present a fast robust approach for real-time image stabilization. Since the perform1ance of image stabilization mainly depends on global motion estimation and the accuracy of motion estimation will be affected when foreground motion happened, sudden image jitters will be introduced during stabilization. To solve this problem, conventional methods detect and remove the foreground objects in motion estimation but this way works inefficiently and fails when foreground moving objects occupy large part of image. Our method is based on the following improvements: modified ORB feature points(FPs) processing, adaptive calculation of affine transformation matrix and joint utilization of two Kalman filters. It can solve the sudden image jitter problem even when there are large foreground moving objects in the image. Qualitative and quantitative evaluations demonstrate the merits of our method. Experiments show that our method solves large foreground motion problem and achieves 35 FPS for 640*480 image on Intel Core i5-4590 CPU@3.30 GHz on the windows.