Automatic patterned fabric defect detection is a promising technique for textile manufacturing due to its low cost and high efficiency. The applicability of most existing algorithms, however, is limited by their intensive computation. To overcome or alleviate the problem, this paper presents a convolutional matching pursuit (CMP) dual-dictionary algorithm for patterned fabric defect detection. A preprocessing with mean sampling is performed to eliminate the influence of background texture of fabric defects. Subsequently, a set of defect-free image blocks are selected as a sample set by sliding window. Dual-dictionary and sparse coefficiencies of the defect-free sample set are obtained via CMP and the K-singular value decomposition (K-SVD) based on a Gabor filter. Then we employ the defect-free and defective fabric image’s projections onto the dual-dictionary as features for defect detection. Finally, the test results are determined by comparing the distance between the features to be measured. Experimental results reveal that the proposed algorithm is effective for patterned fabric defect detection and an acceptable average detection rate reaches by 94.2%.