In many modern optical systems, the resolution is limited not only by the diffraction caused by physical dimensions of the optics lens, but also by the CCD’s nonzero pixel size. Especially for the traditional incoherent illumination, the restriction of CCD pixel is greater than that of optical diffraction. Here we develop a novel approach to enhancing resolution beyond the limit set by CCD’s pixels, in which a two-dimensional and orthogonal encoding mask is attached before the imaging lens to modulate frequency on input target spectrum. Here we focus on the design about a 4f optical imaging system, considering the ability of Fourier transformation to achieve the equivalent conversion between space and frequency domain. And to prevent the loss of frequency in the overlapping regions when sampled by classical CCD, there must be some proportion between the spatial range of object plane and corresponding frequency plane. Meaning while, the wavefront aberration of Fourier lens needs to be controlled to fulfill the mathematical features of Fourier transformation. We apply to improving and revising the theoretical design for the encoding mask based on the design limit of opticalmechanical engineering, and we analyze the different orthogonal forms of encoding masks which can bring the spectra diffraction to the imaging area. According to the theoretical discussion, revision and algorithm simulation, the results in the preliminary testing system show that the encoding mask can be used to produce enhancement of resolution by a factor of 2 in-exchange for decreasing the field of view by the same factor.