In order to achieve multigigabit transmission in deep-space optical communication, our study applies a new modulation mode named orbital angular momentum (OAM) modulation, and uses the encoded OAM states of single photon as data information carriers, thus providing a reliable and high-speed transmission of signals. According to the long link characteristic of deep-space communication, we conduct a reasonable deployment for communication nodes in deep-space environment. First, we present the reliability of deep-space channel and analyze the data rate and spectral efficiency of beams with OAM. Second, we study the characteristics and generations of vortex beams with OAM by simulation. Results show that vortex beams have better spatial multiplexing capability of realizing high capacity data transmission. Finally, we propose an encoding method with OAM states of single photon. The transceiver units are based on spatial light modulators to perform the modulation and demodulation of vortex beams. At the receiver, the charged-coupled device camera is used to detect the signal intensity and decodes the OAM states. Our proposal not only ensures the confidentiality of deep-space optical communication, but also greatly increases the transmission rate.