Scene categorization is an important research topic in computer vision. Although various approaches have been proposed to solve this problem, as far as the authors know, there is little work concerning details in scene images. This paper proposes to manipulate details in scene images to improve scene categorization performance. Specifically, a scene image is first partitioned into super-pixels, and these segments are evaluated to see whether their details need to be enhanced or not. Then, image segments are accordingly processed. After that, scale invariant feature transform features are extracted from processed images for image representation and categorization. Extensive experiments are conducted on the 15-class scene dataset and 67-class indoor scene dataset. Experimental results showed that enhancing details in scene images appropriately can effectively improve scene categorization performance. Furthermore, fusing information from details and structures of scene images leads to state-of-the-art results.