In this paper, the degradation of signal degree of polarization (DOP) by first and second order polarization mode dispersion (PMD) in 40Gb/s RZ and NRZ optical communication is analyzed by numerical simulation. The simulation results show that the degradation of signal DOP by first order PMD is monotony, but which is fluctuated by second order PMD. The influence of two components of second order PMD on DOP for NRZ code and RZ are also investigated by numerical simulation method. The results also show that the influence of the depolarization component and the PCD component on NRZ and RZ data formats are different. Those results will provide the theory basis for how to select proper compensation arithmetic. A compensation system is founded to compensate the first and high order PMD effectively by DOP as feedback signal and particle swarm optimizer (PSO) as compensation arithmetic. Here, we introduce an intelligent method PSO as a searching algorithm to multi-DOF (degree of freedom) PMD compensation, The PSO algorithm used here is described as: (1) Local neighborhood structure is employed to avoid being trapped into sub-optima. (2) 20 particles are employed. So 20 time units (less than 20 ms) are required in one iteration. (3) The maximum iteration number is set to 50. The experiment result show that the first and second PMD can be compensated at the same times for 10 Gb/s RZ and NRZ by using DOP as feedback signal and PSO as searching algorithm.