With the growing demand for more efficient wound care after surgery, there is a necessity to develop a machine learning based image analysis approach to reduce the burden for health care professionals. The aim of this study was to propose a novel approach to recognize wound infection on the postsurgical site. Firstly, we proposed an optimal clustering method based on unimodal-rosin threshold algorithm to extract the feature points from a potential wound area into clusters for regions of interest (ROI). Each ROI was regarded as a suture site of the wound area. The automatic infection interpretation based on the support vector machine is available to assist physicians doing decision-making in clinical practice. According to clinical physicians’ judgment criteria and the international guidelines for wound infection interpretation, we defined infection detector modules as the following: (1) Swelling Detector, (2) Blood Region Detector, (3) Infected Detector, and (4) Tissue Necrosis Detector. To validate the capability of the proposed system, a retrospective study using the confirmation wound pictures that were used for diagnosis by surgical physicians as the gold standard was conducted to verify the classification models. Currently, through cross validation of 42 wound images, our classifiers achieved 95.23% accuracy, 93.33% sensitivity, 100% specificity, and 100% positive predictive value. We believe this ability could help medical practitioners in decision making in clinical practice.