Classification of images in many categorized datasets has rapidly improved in recent years. However, methods that perform well on particular datasets typically have one or more limitations, such as insufficient image-transformation invariance or significant performance degradation as the number of classes is increased. We attempt to overcome these challenges by extracting and matching visual features only at the focuses of visual saliency instead of the entire scene. First, we propose a visual-saliency detection method that combines the respective merits of color-saliency boosting and global-region-based contrast schemes to achieve more accurate saliency maps. Using a single feature type, we then obtain good performance on three public datasets when compared to other state-of-the-art approaches. Overall, our results prove that robust and efficient fixation-based classification, in terms of reducing the complexity of feature extraction, is possible.