As a new type of light dispersion device, Acousto-Optic Tunable Filter (AOTF) based on the acousto-optic interaction principle which can achieve diffractive spectral, has rapidly developed and been widely used in the technical fields of spectral analysis and remote sensing detection since it launched. The precise measurement of AOTF’s optical performance parameter is the precondition to ensure spectral radiometric calibration and data inversion in the process of quantitation for spectrometer based on AOTF. In this paper, a kind of AOTF performance analysis system in 450~3200nm wide spectrum was introduced, including the fundamental principle of the basic system and the test method of the key optical parameters of AOTF. The error sources and the influence of the magnitude of the error in the whole test system were analyzed and verified emphatically. The numerical simulation of the noise in detecting circuit and the instability of light source was carried out, and based on the simulation result, the method for improving the measuring accuracy of the system were proposed such as improving light source parameters, correcting and changing test method by using dual light path detecting, etc. Experimental results indicate that: the relative error can be reduced by 20%, and the stability of the test signal is better than 98%. Finally, this error analysis model and the potential applicability in other optoelectronic measuring system were also discussed in the paper.