Downscaling between various video sizes is an important field in transcoding, especially in the era of communication and consumer electronics. Architectures are proposed to offer the best strategy in downscaling. We combine an intra refresh mechanism with a partial encode architecture. A new intra refresh decision mechanism is then proposed based on visual quality constraint. Not only does the work improve transcoded video quality, but overall complexity is also kept low. The experiment results show that the proposed work achieves a 0.5 to 2.5-dB advance in quality, compared with intra refresh in the open loop and partial encode architectures.