The problem of designing ternary phase and amplitude filters (TPAF5) that reduce the probability of image misclassification for a two-class image set is studied. The Fisher ratio is used as a measure of the correct classification rate, and an attempt is made to maximize this quantity in the filter designs. Given the nonanalytical nature of the design problem, a simulated annealing optimization technique is employed. Computer simulation results are presented for several cases including single in-class and out-of-class image sets and multiple image sets corresponding to the design of synthetic discriminant function filters. Significant improvements are found in expected rates of correct classification in comparison to binary phase-only filters and other TPAF designs. Approaches to accelerate the filter design process are also discussed.