Dimension reduction techniques have become one popular unsupervised approach used towards detecting anomalies in hyperspectral imagery. Although demonstrating promising results in the literature on specific images, these methods can become difficult to directly interpret and often require tuning of their parameters to achieve high performance on a specific set of images. This lack of generality is also compounded by the need to remove noise and atmospheric absorption spectral bands from the image prior to detection. Without a process for this band selection and to make the methods adaptable to different image compositions, performance becomes difficult to maintain across a wider variety of images. Here, we present a framework that uses factor analysis to provide a robust band selection and more meaningful dimension reduction with which to detect anomalies in the imagery. Measurable characteristics of the image are used to create an automated decision process that allows the algorithm to adjust to a particular image, while maintaining high detection performance. The framework and its algorithms are detailed, and results are shown for forest, desert, sea, rural, urban, anomaly-sparse, and anomaly-dense imagery types from different sensors. Additionally, the method is compared to current state-of-the-art methods and is shown to be computationally efficient.