Based on weak fluctuation theory, the expression of bit-error rate (BER) of Multiple Quadrature Amplitude Modulation (MQAM) with the consideration of detector noise in the downlink of space communication system is discussed in this paper. According to the expression, the performance of three typical modulation schemes, which are 4QAM, 16QAM, 64QAM, are specially analyzed. It is known that the higher the order of the modulation scheme is, the more bits of information per symbol can carry. However, when the transmission power is 1 W and the receiver diameter Dr is 1 m, the BER is 2.12×10-13 for 4QAM, 5.98×10-8 for 16QAM and 6.22×10-5 for 64QAM, which means that a higher order modulation scheme shows a higher bit-error rate (BER). Thus considering bandwidth efficiency as well as bit error rate, 16QAM is highly recommended in the real space optical communication system. In addition, the relationships between BER and optimum divergence angle, transmitter beam radius, receiving aperture for downlink are also suggested respectively in this paper, which has important reference significance for the design of the ground-to-satellite laser communication system.