This paper presents a method to optimize the SVM committee used in a colonic polyp CAD system to achieve high detection performance and efficiency. In our CAD system, characteristic features of polyp candidates are fed into a committee of SVMs to determine if one detection is a true polyp. The committee consists of M different SVMs, and each of them is established by an N-feature vector. A progressive feature vector selection scheme was proposed to select a population of feature vectors, in which N-feature vectors are composed progressively in N stages. To optimize the SVM committee configuration, two-way ANOVA is performed to analyze the effect of committee-member-number (M) and feature-vector-length (N). The area under the ROC curve (AUC) in a ten-fold cross validation is used as the performance metric. Pairwise Tukey’s tests are performed to reveal if the performance differences between two configurations are statistically significant. The experiments were tested on 29 patients with 53 polyps. The committee configuration in comparison are N=1 to 7 and M=1, 3, 5, 7, or 9. ANOVA showed that N = 3 has statistically significant performance improvement over N=1 and 2, but is statistically equivalent with N= 4 to 7. It also showed that there is statistical improvement from M = 1 to 7, while M = 7 and 9 are statistically equivalent. Based on the result, we chose a committee configuration with N = 3 and M = 7 since it is the most efficient committee with statistically best performance.