In order to overcome the high cost of manufacture and large volume or weight limitations, one solution is to arrange multiple sub-aperture optical systems in accordance with certain spatial rules. The stacking image of sub-aperture beam on focal plane is equivalent to large aperture optical system. However, due to the discretization of pupil distribution of sparse-aperture optical system, the signal-to-noise ratio of image is reduced, the modulation transfer function decreases at midband spatial frequencies, and the optical system errors increase. Aiming at the poor imaging quality of sparse-aperture optical system, in this article, the method of restoration algorithm based on improved Wiener filter and optimization of adjacent frames is proposed, which makes the restored video image have higher definition. The restoration algorithm based on improved Wiener filter and evaluate as well as optimize of adjacent frames in this article mainly contains four aspects, including the analyze of image degradation process, the establish of image restoration model, the evaluate of restored image’s definition, and the optimize of adjacent frame image. Firstly, Synthesize the effect of atmospheric transmission and array structure on image degradation, we have constructed an image degradation model and have calculated the degradation function under the model. Then, the restoration model based on Wiener filter is established and improved. Moreover, the definition evaluation factor of no reference image is built to measure the quality of the restoration image. Finally, construct the mapping relation between the adaptive constant K and the definition evaluation factor in Wiener filter, constantly optimize image restoration quality. In high altitude reconnaissance, remote sensing imaging and other fields, cameras are required to have very high resolution, so the algorithm in this article has great research value.