Point matching under illumination changes is significant for many vision information applications. However, the uneven and dramatic illumination variations model is rarely considered in existing point matching algorithms. Therefore, a method to match features efficiently under uneven and dramatic illumination changes is presented. This method extracts and describes illumination invariant interesting points from matched multibrightness layers that are obtained by a set of contrast stretching functions and prior information based on original images. Layers matching is insensitive to large unevenness of illumination changes and provides similar images in brightness and structure, so the effects of large uneven illumination changes can be reduced greatly. This algorithm is compatible with most detectors and descriptors. To accelerate the computing speed, the features from the accelerated segment test detector and the improved speeded up robust features descriptor are chosen in this paper. In addition, the combination of priority Hamming distance matching and Lowe’s matching algorithms is first proposed to increase the matching speed. This method is generic and can be used in most point matching under all varying illumination conditions. Experimental results demonstrate that the proposed method improves the quality of matched points significantly.