In this paper, the optimization algorithms are introduced in adaptive PMD compensation in 10Gb/s optical communication system. The PMD monitoring technique based on degree of polarization (DOP) is adopted. DOP can be a good indicator of PMD with monotonically deceasing of DOP as differential group delay (DGD) increasing. In order to use DOP as PMD monitoring feedback signal, it is required to emulate the state of DGD in the transmission circuitry. A PMD emulator is designed. A polarization controller (PC) is used in fiber multiplexer to adjust the polarization state of optical signal, and at the output of the fiber multiplexer a polarizer is used. After the feedback signal reach the control computer, the optimization program run to search the global optimization spot and through the PC to control the PMD. Several popular modern nonlinear optimization algorithms (Tabu Search, Simulated Annealing, Genetic Algorithm, Artificial Neural Networks, Ant Colony Optimization etc.) are discussed and the comparisons among them are made to choose the best optimization algorithm. Every algorithm has its advantage and disadvantage, but in this circs the Genetic Algorithm (GA) may be the best. It eliminates the worsen spots constantly and lets them have no chance to enter the circulation. So it has the quicker convergence velocity and less time. The PMD can be compensated in very few steps by using this algorithm. As a result, the maximum compensation ability of the one-stage PMD and two-stage PMD can be made in very short time, and the dynamic compensation time is no more than 10ms.