Optical coherence tomography (OCT) is an interferometric technique using the low coherence property of light to axially image at high resolution in biological tissue samples. Transverse imaging is obtained with two-dimensional scanning and transverse resolution is limited by the size of the scanning beam at the imaging point. The most common metrics used for determining the axial resolution of an OCT system are the full-width-at-half-maximum (FWHM), the absolute square integral (ASI), and the root-mean-square (RMS) width of the axial PSF of the system, where the PSF of an OCT system is defined as the envelope of the interference fringes when the sample has been replaced by a simple mirror. Such metrics do not take into account the types of biological tissue samples being imaged. In this paper we define resolution in terms of the instrument and the biological
sample combined by defining a resolution task and computing the associated detectability index and area under the receiver operating characteristic curve (AUC). The detectability index was computed using the Hotelling observer or best linear observer. Results of simulations demonstrate that resolution is best quantified as a
probability of resolving two layers, and the impact on resolution of variations in the index of refraction between the layers is clearly demonstrated.
The departure between a reference range image and an acquired one is assessed using correlation. Decision criteria have been defined. Performance of coding and filtering are discussed. Results obtained on a Vander Lugt correlator with two twisted nematic spatial light modulators are given.