Bolted steel joints are one of the most common types of connections in steel structures. Due to significant loads carried over long-term operation, bolted steel joints are prone to structural damage. Monitoring bolted steel joints is critical to ensure their functionality and structural safety. Among all factors related with joint damage, bolt loosening has been reported as a main cause of the damage of bolted joints. Detecting bolt loosening is therefore critical for the heath assessment of bolted steel joints. Recently, computer vision-based structural health monitoring (SHM) methods have been proposed in many research fields due to the benefits of being low-cost, easy-to-deploy, and contactless. In this study, we propose an image-based feature tracking approach to detect bolt loosening in steel connections. The method relies on a feature tracking algorithm, through which densely distributed feature points can be automatically detected and tracked from multiple images taken at different times. A novel algorithm is established to rapidly search feature points and track the movement of these feature points between images. If the bolt is loosened, feature points associated with the loosened bolt would exhibit a unique rotational movement pattern. By highlighting these feature points, the loosened bolt can be successfully localized. The effectiveness of the proposed approach was verified by a laboratory test of a steel joint using a consumer-grade digital camera.