As the demand for ultrahigh density information storage continues to grow, recording mark size of several tens nanometer which is smaller than the optical diffraction limit is required in optical memory. Functional film super-resolution technique is one of practical approaches to overcome the optical diffraction limit. Optical constants are important parameters to optical films as super-resolution masks. In this paper, the influence of film thickness on optical constants of antimony-based bismuth-doped super-resolution mask layer is investigated. The structure of the samples with different thickness was studied by X-ray diffraction. The transmission spectrum was measured by spectrophotometry. The optical constants of the films in the range of 300-800 nm were measured by spectroscopic ellipsometry. The results show that the structure of the film transforms from amorphous state to crystal state when the thickness increases from 7 nm to 300 nm. In the range of 300-800 nm, the refractive index and extinction coefficient increase with increasing wavelength. The transmission decreases rapidly when the thickness increases from 7 nm to 30 nm. The influences of film thickness on optical constants are more significant in the thickness range of 7-50 nm than that in the thickness above 50 nm.