High-efficiency video coding (HEVC) introduces a flexible hierarchy of three block structures: coding unit (CU), prediction unit (PU), and transform unit (TU), which have brought about higher coding efficiency than the current national video coding standard H.264/advanced video coding (AVC). HEVC, however, simultaneously requires higher computational complexity than H.264/AVC, although several fast inter-mode decisions were proposed in its development. To further reduce this complexity, a fast inter-mode decision algorithm is proposed based on temporal correlation. Because of the distinct difference of inter-prediction block between HEVC and H.264/AVC, in order to use the temporal correlation to speed up the inter prediction, the correlation of inter-prediction between two adjacent frames needs to be analyzed according to the structure of CU and PU in HEVC. The probabilities of all the partition modes in all sizes of CU and the similarity of CU segmentation and partition modes between two adjacent frames are tested. The correlation of partition modes between two CUs with different sizes in two adjacent frames is tested and analyzed. Based on the characteristics tested and analyzed, at most, two prior partition modes are evaluated for each level of CU, which reduces the number of rate distortion cost calculations. The simulation results show that the proposed algorithm further reduces coding time by 33.0% to 43.3%, with negligible loss in bitrate and peak signal-to-noise ratio, on the basis of the fast inter-mode decision algorithms in current HEVC reference software HM7.0.