In experiments, we demonstrated that luminescence quantum yield of single gold nanorods illuminated by continuous wave laser at wavelength of 532 nm depends on the excitation polarization, while that excited by 633 nm laser does not. The electrons in sp-band dominates the luminescence process when the 633 nm laser is applied, resulting in a constant quantum yield under different excitation polarizations. When the 532 nm laser is applied, both the electrons in d-band (interband transition) and sp-band (surface plasmon) involve in the luminescence process. The variation of quantum yield by the 532 nm laser is resulted from different efficiency of d-band interband transition and sp-band plasmon conversion into luminescence. Furthermore, we found that plasmon modes coupling effect can modulate strongly the plasmon emission efficiency by comparing the luminescence of two sets of the nanorods with different size. And smaller size GNRs often results in higher quantum yield of interband transition. These findings make a step to understand the luminescence process of plasmonic nanostructures and point out a rule to control it through plasmon mode coupling effect.