Coherent optical communication systems are an attractive choice for future high-speed optical satellite networks due to the excellent receiver sensitivities that can be achieved by coherent detection techniques. Recently, the interest in coherent optical systems has increased with advances in photonic devices and high-speed digital signal processing (DSP). The best coherent receiver sensitivity (9 photons per bit) is obtained with phase-shift keying (PSK) modulation in combination with homodyne detection. In this paper, we propose coherent duobinary systems for high-speed optical satellite communications. It is shown that duobinary modulation combined with homodyne detection comes very close to the ultimate receiver sensitivity limit set by PSK. Moreover, duobinary modulation benefits from having a relatively narrow signal spectrum. Thus, duobinary systems generally require half the bandwidth of PSK, and are less susceptible to inter-symbol interference. These are advantageous attributes for overcoming the technological difficulties in high-speed systems. The duobinary homodyne receiver is also compatible with high-speed DSP post-processing for ameliorating the atmospheric channel impairments. Computer simulation studies show that the theoretical performance limits of coherent duobinary systems can be achieved with practical devices.