This paper introduces a virtual synthesis technology based on handheld augmented reality. The technology can display the synthesized renderings of a three-dimensional background scene, portraits in a picture and a three-dimensional model of exhibits in real time using blue screen matting, coordinate registration, image synthesis and other methods. And the technology supports adjustment of shooting angle. This paper improves some functions of ARToolKit and develops a mobile system based on ARToolKit, OpenGL ES and jpct-AE libraries. The system has been applied in Zhejiang provincial museum and shows good results.