For rate-distortion optimized rate allocation in JVT Scalable Video Coding (SVC), the distortion impact of every FGS NAL unit on the global reconstruction quality is calculated by repeatedly bitstream decoding, which leads to high complexity. In this paper, a fast rate allocation algorithm by modeling distortion estimation is proposed. Based on the hypothesis that DCT residual coefficients follow Laplacian distribution, we establish the distortion estimation model by calculating quantization error of each FGS NAL unit and analyzing the prediction in hierarchical B coding structure. Besides, the parameter in the model is updated according to the distribution of residual coefficients decoded at the base layer within every frame. Experimental results show that compared to the existing method of R-D optimized rate allocation in SVC, the proposed method results in a reduction in decoding time of nearly 50%, and save the runtime of rate allocation by 45.3%, while the PSNR loss of decoded sequence is only 0.04 dB on average.