Optical methods of measurement do not require contact of a probe and the object under study, and thus have found use in a broad range of applications such as nondestructive testing (NDT), where noninvasive measurement is crucial. Measuring the refractive index of a material can give a valuable insight into its composition. Low‑coherence radiation sources enable measurement of the sample’s properties across a wide spectrum, while simultaneously measuring the absolute value of optical path difference between interfering waves, which is necessary to calculate the refractive index. The measurement setup used in this study consists of a fiber‑based Fabry‑Perot interferometer, illuminated by a low‑coherence infrared source. The samples under measurement are located in the cavity of the interferometer, and their transmission spectra are recorded using an optical spectrum analyzer. Additional reference measurements are performed with the cavity filled with air, in order to precisely measure the geometrical length of the cavity. The purpose of the study was to develop a digital signal processing algorithm to improve the resolution of analysis of the spectra of radiation measured at the output of the interferometer. This goal was achieved by decreasing the broadening of the signal in the Fourier domain caused by dispersion of the medium filling the cavity. The Fractional Fourier Transform is a generalization of the Fourier transform allowing arbitrary rotation of the signal in the time-frequency domain, allowing more precise analysis of signals with variable frequency. This property makes this transformation a valuable tool for the analysis of interferometric signals obtained from measurements of dispersive media, as the variable rate of change of the optical path length with respect to wavenumber in such media results in varying frequency of the modulation of measured spectra. The optical path difference inside the material under measurement is used together with the geometrical length obtained from the reference measurement in order to determine the refractive index. The parameters of the transformation are found by iterative adjustment to the signal under analysis. The developed algorithm was tested using both real measured spectra and simulated signals based on a theoretical model of the interferometric setup, and its effectiveness was compared to previously used methods of analysis. It was found to increase the resolution of analysis up to the Fourier limit that occurs in signals with no dispersion.