The shot-noise unit (SNU) is a crucial factor for the practical security of a continuous-variable quantum key distribution system. In the most widely used experimental scheme, the SNU should be calibrated first and used as a constant during key distribution. Because the measurement result of quadrature is normalized with the calibration SNU but scaled with practical SNU, which could open loopholes for the eavesdropper to intercept the secret key. In this paper, we report a quantum hacking method to control the practical SNU by using the limited compensation rate for polarization drift. Since the polarization of local oscillator pulses is partially measured, the attack is implemented by manipulating the polarization of the local oscillator pulses without measurement when the system is running. The simulation and experiment results indicate that the practical SNU can be manipulated by the eavesdropper. By making the difference between the calibration and the practical SNU, the excess noise estimated by Alice and Bob could always be lower than the practice which is introduced by the eavesdropper and the distributed keys are not secure.