In ideal optical system, one pixel on the image surface corresponds to a unit area on the object surface. However, in actual optical system, the image of a unit area on object will be extended to a spot around the central pixel due to the response of optical system and the process of electrical signal. The spot is also called point extended function. In this paper, the extension of imaging is simulated in laboratory, which shows the brightness variation of the image. In the experiment, an object that corresponds to 1*1 pixel is placed in a uniform background at appropriate distance, and the radiance is measured on the image surface. Then we change the object to 2*2, 4*4, 8*8, 16*16 pixels and the radiance is also measured. It is verified that the image of unit area will be extended to several pixels. After a careful analysis and examination of the data, a new method of estimating the transmission function of optical system is presented based on optical imaging theory. The method is verified in laboratory by measuring blackbody using AGEMA900 thermo-vision imaging device, and can be applied to estimating the transmission function of unknown optical system.