Volume holographic gratings have been used in waveguide displays to implement full-color three-dimensional imaging. Among these, multiplexing gratings are advanced in low energy losses and simple manufacture technologies when used as couplers of color hologram waveguides. A multiplexing holographic grating is designed to realize a uniform red, green, and blue diffraction efficiency and eliminate stray light to the largest extent. Results indicate that the red, green, and blue light incident normal to the grating could be successfully in-coupled into the planar waveguide for total internal reflection with high peak diffraction efficiency, similar energy output, and little stray light. We also analyze the effect of the technical tolerance, including gating thickness, index modulation, grating period, slanted angle, and incident angle. This analysis could help to minimize the optical system and improve the color image quality of waveguide displays.