In this paper, we propose a layered complexity-aware encryption scheme to partially encrypt scalable video bitstreams to achieve reasonable security that the specific application entails and the involved computational complexity allows. The proposed scheme naturally combines wavelet-based scalable video coding technique and the concept of selective encryption. We also study the relationship between rate, distortion, and the involved encryption complexity (R-D-EC). Here, distortion is used to measure the level of security of the encrypted video bitstream, and the percentage of encrypted bitstream is used to denote encryption complexity. Our simulation results indicate that selective encryption using a prioritized bitstream structure provided by scalable video coding can achieve almost bitrate-independent encryption complexity control.