Face aging and rejuvenating work effectively in public security criminal investigation, cross-age recognition, and entertainment. However, three main problems still exist: the lack of accurate and sufficient dataset, low aging effect, and poor preservation of personal information. We propose a semi-supervised face aging and rejuvenating method for face aging and rejuvenating. In particular, a conditional encoder is utilized to map an input face into a latent vector, which is used by the generator network with age conditions to produce a new face. The latent vector preserves identity information, whereas the age label controls face aging or rejuvenating. To make generated features closer to prior features, the discriminator network is designed to assist the generator network. In addition, a cycle optimized method is utilized to preserve the personal information of the generated face. Experimental results demonstrate that our network can generate more realistic faces, both in personal identity and age consistency.