The performance of target detection and tracking algorithms generally depends on the signature, clutter, and noise that are usually present in the input scene. To evaluate the effectiveness of a given algorithm, it is necessary to develop performance metrics based on the input plane as well as output plane information. We develop two performance metrics for assessing the effects of input plane data on the performance of detection and tracking algorithms by identifying three regions of operation—excellent, average, and risky intervals. To evaluate the performance of a given algorithm based on the output plane information, we utilize several metrics that use primarily correlation peak intensity and clutter information. Since the fringe-adjusted joint transform correlation (JTC) was found to yield better correlation output compared to alternate JTC algorithms, we investigate the performance of two fringe-adjusted JTC (FJTC)-based detection and tracking algorithms using several metrics involving the correlation peak sharpness, signal-to-noise ratio, and distortion invariance. The aforementioned input and output plane metrics are used to evaluate the results for both single/multiple target detection and tracking algorithms using real life forward-looking infrared (FLIR) video sequences.