This paper proposes an efficient wavelet-based coding scheme with fine grain scalability, where the base layer is encoded with a novel designed of wavelet-based coder, and the enhancement layer is encoded with progressive fine granularity scalable (PFGS) coding. This algorithm involves multi-frame motion compensation, rate-distortion optimizing strategy with Lagrangian cost function and context-based adaptive arithmetic coding. In order to improve efficiency ofthe enhancement layer coding, an improved motion estimation scheme that uses both information from the base layer and the enhancement layer is also proposed in this paper. The novel designed of wavelet-based coder significantly improves the coding efficiency ofthe base layer compared with MPEG-4 ASP (Advanced Simple Profile) and H.26L TML9. The PFGS coding is a significant improvement over MPEG-4 FGS at the enhancement layer. Experiments show that single layer coding efficiency gain of the proposed scheme is about 2.0-3.0dB and 0.3-1.0dB compared with MPEG-4 ASP and H.26L TML9, respectively. The overall coding efficiency gain of the proposed scheme is about 4.0-5.0dB compared with MPEG-4 FGS.