In this manuscript we describe the design and optimization of ultrahigh resolution spectral/Fourier domain OCT systems for three applications in retinal imaging: imaging of the normal retina, three-dimensional (3D) imaging of retinal pathologies, and 3D imaging of the rodent retina. Seven spectrometer configurations were tested for resolution and sensitivity drop with depth, and CCD pixel crosstalk was characterized. The human retina was imaged in vivo with five different axial resolutions between 2 and 10 microns, and with three different transverse resolutions. Information from these experiments enabled the optimization of OCT systems for the above applications. Results include clinical 3D data of retinal pathologies, high quality cross-sectional images of the normal retina with different axial and transverse resolutions and 3D data from the rat and mouse retinas. Factors affecting the sensitivity fall-off are discussed and theoretical predictions are compared with experimental measurements. Different retinal imaging applications necessitate different system designs, depending on the requirements of speed, axial resolution, axial measurement range, transverse resolution, and field of view. While axial resolution is the dominant factor in image quality, a smaller transverse spot size can reduce speckle size and improve contrast at boundaries such as the boundary between the ganglion cell layer and the inner plexiform layer. The effect of reducing the transverse spot size is most pronounced in images with 5-10 um axial resolution. In addition, we characterize all factors responsible for the sensitivity drop with depth in spectral/Fourier domain OCT.