This paper proposes a robust video transmission system over lossy packet-switched networks with block-based fine granularity scalable (B-FGS) coding including source coding, efficient rate allocation and packetization, which works effectively when packet loss occurs. The compressed B-FGS bit-stream can be truncated in fine grain for its block-based bit-plane coding at enhancement layer. Therefore channel adaptive rate allocation can be implemented and the time-varying bandwidth can be exploited efficiently. Distortion associated rate allocation method is proposed to substitute Rate-Distortion optimal rate allocation in order to reduce the streaming server’s complexity. In addition, packetizing with independent data provides robust video transmission, which restrains the error propagation across packets. Furthermore, in order to mitigate quality flicking of decoded video induced by packets loss, spatial interleaving is applied in packetizing. Thus, in proposed video transmission system, available bandwidth variation and packet loss, two key problems in video streaming applications, are well dealt with. In order to validate the effectiveness of proposed transmission system, MPEG-4 FGS transmission system and that with conditional retransmission are compared in simulations. Proposed transmission system is more robust to packet-loss errors than the other two, especially when packet-loss rate is high.