A novel genetic algorithm (GA) with a Lamarckian search is proposed for the design of the multiplexed computergenerated hologram (MCGH) with polygonal apertures. The Fraunhofer image of the new MCGH is computed by coherent addition of the subhologram subimages. The subimages are obtained by multiplying the fast Fourier transforms of the subhologram transmittance distributions by layout coefficients computed with the Abbe transform. The division into polygonal apertures is the same for all cells, and defines the polygonal layout of the cells. In our preceding designs of the MCGH with polygonal apertures, only the subhologram transmittances, but not the polygonal layout of the cells, were optimized with our iterative subhologram design algorithm (ISDA). In this paper, we optimize for the first time the polygonal layout of the MCGH cells with a novel GA. For fabrication by e-beam lithography, each cell is composed of a number of stripes. Each stripe is divided into some trapezoidal apertures, which can (i) take a number of different shapes and (ii) belong to a number of different subholograms. The number of possible polygonal layouts for the cells therefore is huge and equal to 264 = 1.85 × 1019 in the case of a MCGH with five subholograms. Each possible layout is coded as a chromosome of bits. Our novel GA performs crossovers and mutations. However, differently from the classical GA, our new GA also uses a Lamarckian search based on a gradient descent, and rapidly determines the optimal polygonal layout for the MCGH cells.