Spatial-spectral interference carries the spectral phase difference information between short pulses. We propose a new method of time delay retrieval via the slope of spatial-spectral interference fringe in the case of only time delay without high-order spectral phase difference between short pulses. The analytical expression is deduced based on the principle of spatial-spectral interference. The simulation results show that the slope of spatial-spectral interference fringe and the crossing angle between short pulses are both important for the calculation accuracy. This proposed method has advantages of no direction-of-time ambiguity, simple principle and calculation process, which are helpful for the measurement and control of the time delay between short pulses in coherent combination, plasma parameter diagnosis and so on.