Digital holography combines the advantages of the optical holography and the computers. It can implement an all-digital processing and has the quasi real-time property. With lensless Fourier transform recording architecture, the limited bandwidth of CCD camera can be utilized sufficiently, and the sampling theorem is satisfied easily. Therefore, high-resolution can be achieved. So it is preferred in the microstructure imaging. In the paper, based on the Fresnel diffraction theory and the off-axis lensless Fourier transform recording architecture, the experimental optimization and correspondingly the digital reconstruction was investigated. Also, the lateral resolution of the reconstructed image was analyzed and improved by the proposed techniques. When the USAF test target was imaged without any pre-magnification, the lateral resolution of 3.1μm was achieved, which matched the theoretical prediction very well. The key points to achieve high resolution image are to use the smaller object and to arrange the distance between the object and the CCD plane as short as possible. Meanwhile, properly overlapping the reconstructed image with the DC term was helpful to improve the resolution. The noise in the reconstructed image could be reduced greatly by choosing the optical elements precisely and adjusting the beam path finely. The experimental results demonstrated that it is possible for the digital holographic microscopy to produce the high resolution image without the objective pre-magnification. The results also showed that, with a high quality hologram, the special image processing during the reconstruction may be unnecessary to obtain a high quality image.