Failure-free operation of solar panels is of fundamental importance for modern commercial solar power plants. To achieve higher power generation efficiency and longer panel life, a simple and reliable panel evaluation method is required. By using thermal infrared imaging, anomalies can be detected without having to incorporate expensive electrical detection circuitry. In this paper, we propose a solar panel defect detection system, which automates the inspection process and mitigates the need for manual panel inspection in a large solar farm. Infrared video sequences of each array of solar panels are first collected by an infrared camera mounted to a moving cart, which is driven from array to array in a solar farm. The image processing algorithm segments the solar panels from the background in real time, with only the height of the array (specified as the number of rows of panels in the array) being given as prior information to aid in the segmentation process. In order to “count” the number the panels within any given array, frame-to frame panel association is established using optical flow. Local anomalies in a single panel such as hotspots and cracks will be immediately detected and labeled as soon as the panel is recognized in the field of view. After the data from an entire array is collected, hot panels are detected using DBSCAN clustering. On real-world test data containing over 12,000 solar panels, over 98% of all panels are recognized and correctly counted, with 92% of all types of defects being identified by the system.