A novel measurement system of digital optical modulation transfer function based on CCD is introduced. This system is mainly made of slight source , collimator, detected system, magnify object lens set, CCD, image collect card ,computer and so on. Different algorithms are proposed for two different sighting vanes of the pinhole and the slit. For pinhole sighting vane, three levels of wavelet transformation algorithm are used to denoise, and by the relation of initial point spread function and line spread function, two-dimensional pinhole image transforms two vertical direction line spread functions. Through the algorithms of Fourier transform, the test can be realized. Slit sighting vane adopts the cubic fitting interpolation algorithm to smooth the curve, and Fourier transform to realize the test. The normalized way is used to process data for the two sighting vane algorithm results. Different model photography lenses are separately measured on the test system. It is showed by the test result that the denoised signals for the pinhole sighting vane are quite smooth through three levels of wavelet transform algorithms, and the point spread function transforms to two vertical-directional line proliferation functions, which makes the Fourier frequency spectrum transform simple; and it has better effect for the slit sighting vane to employ the cubic fitting interpolation algorithm to smooth the curve. MTF test precision is 0.04mm on axis and 0.08mm out of axis.