A hyperspectral image (HSI) has high-spectral and low-spatial resolution. As a result, most targets exist as subpixels, which pose challenges during target detection. Moreover, limitations of target and background samples always hinder the detection performance. In this study, a hybrid method for subpixel target detection of an HSI is developed. The scores of matched filter (MF) and adaptive cosine estimator (ACE) are used to construct a hybrid detection space. The reference target spectrum and background covariance matrix are improved iteratively based on the distribution property of targets, using the hybrid detection space. As the iterative process proceeds, the reference target spectra get closer to the central line, which connects the centers of the target and the background, resulting in a noticeable improvement in target detection. One synthetic dataset and two real datasets are used in the experiments. The results are evaluated based on the mean detection rate (DR), receiver operating characteristic curve, and observations of the detection results. For the synthetic experiment, the hybrid method improves more than 10 times with regard to the average DR compared with that of the traditional MF and ACE algorithms, which use N-FINDR target extraction and Reed–Xiaoli detector for background estimation.