A compact design of an acousto-optic high-speed infiniteimpulse- response filter for line enhancement or notch filtering is introduced. Such filters can find applications in communication systems for signal-processing tasks such as carrier and bit recovery, signal prefiltering for noise reduction, interference or jammer cancellation, and channel equalization. Stability of such filters is an important concern because of the feedback and the nonlinearity in the acousto-optic cell response. Stability conditions are derived. Theoretical analysis and numerical experiments show that an acousto-optic-cell-based infinite-impulseresponse filter can enhance the signal-to-noise ratio at the filter output by 15 to 30 dB or more, depending on the design.