Passive millimeter-wave (PMMW) imagery has tremendous potential for imaging in adverse conditions. However, poor resolution and long acquisition times pose serious limitations to this potential. Therefore, an important issue is the optimization of the sampling pattern. Ordinarily, a focal plane sensor array has sensors placed in a rectangular grid pattern at subNyquist density, and the array must be dithered to sample the image plane at the Nyquist density in each dimension. However, the Nyquist density oversamples the image due to the usually circular support of the diffraction-limited image spectrum. We develop an efficient algorithm for optimizing the dithering pattern so that the image can be reconstructed as reliably as possible from a periodic nonuniform set of samples, which can be obtained from a dithered rectangular-grid array. Taking into account the circular frequency support of the image, we sequentially eliminate the least informative array recursively until the minimal number of arrays remain. The resulting algorithm can be used as a tool in exploring the optimal image acquisition strategy.