High Efficiency Video Coding (HEVC) is the state-of-art video compression standard which fully support scalability features and is able to generate layered video streams with unequal importance. Unfortunately, when the base layer (BL) which is more importance to the stream is lost during the transmission, the enhancement layer (EL) based on the base layer must be discarded by receiver. Obviously, using the same transmittal strategies for BL and EL is unreasonable. This paper proposed an unequal error protection (UEP) system using different hierarchical amplitude modulation (HQAM). The BL data with high priority are mapped into the most reliable HQAM mode and the EL data with low priority are mapped into HQAM mode with fast transmission efficiency. Simulations on scalable HEVC codec show that the proposed optimized video transmission system is more attractive than the traditional equal error protection (EEP) scheme because it effectively balances the transmission efficiency and reconstruction video quality.