Finding defects with automatic visual inspection techniques is an essential task in various industrial fields. Despite considerable studies to achieve this task successfully, most previous methods are still vulnerable to ambiguities from diverse shapes and sizes of defects. We introduce a simple yet powerful method to segment defects on various texture surfaces in an unsupervised manner. Specifically, our method is based on the multiscale scheme of the phase spectrum of Fourier transform. The proposed method can even handle one-dimensional long defect patterns (e.g., streaks by scratch), which have been known to be hard to process in previous methods. In contrast to traditional inspection methods limited to locating particular sorts of defects, our approach has the advantage that it can be applied to segmenting arbitrary defects, because of the nonlinear diffusion involved in the multiscale scheme. Extensive experiments demonstrate that the proposed method provides much better results for defect segmentation than several competitive methods presented in the literature.