We present a birefringence analysis method based on polarization-sensitive swept-source optical coherence tomography (PS-SS-OCT) for distinguishing pearls. To cope with the round shape of general pearls, a rotation stage was used for the sample scanning. With the system, the birefringence of several cultured pearls including south sea, Akoya, freshwater cultured pearls, and imitation pearls are analyzed and compared. Interestingly, PS-SS-OCT surely shows well developed birefringence patterns of phase retardation and fast axis orientation with the cultured pearls, whereas the pattern does not appear in the imitation pearls. In addition, the intensity image can help to distinguish the cultured pearls. Therefore, PSSS-OCT enables a more accurate interpretation for identifying the cultured pearls from imitation pearls.