Fatigue cracks developed in metallic materials are of critical safety concerns for mechanical, aerospace, and civil engineering structures. For fracture-critical structures, if not appropriately inspected, excessive growth of fatigue cracks can lead to catastrophic structural failures. Current crack detection technologies developed for nondestructive testing (NDT) or structural health monitoring (SHM) often require costly equipment, extensive human involvement, or complex signal processing algorithms. Recently, computer vision-based methods have shown great promise in damage detection for being contactless, low cost, and easy-to-deploy. In this paper, we propose a novel computer vision-based method for detecting fatigue cracks in a video stream. This method is based on tracking the surface motion of structural members under crack opening and closing, and identifying fatigue cracks by extracting discontinuities in the surface motion caused by cracking. The effectiveness of this method was validated through an experimental test of a steel compact, C(T), specimen. Results indicate that the proposed approach can robustly detect the fatigue crack under ambient lighting condition, despite the crack was surrounded by other crack-like edges, covered by complex surface textures, or invisible to human eyes under crack closure.