Image-based wavefront sensing algorithms are being used to characterize the optical performance for a variety of current
and planned astronomical telescopes. Phase retrieval recovers the optical wavefront that correlates to a series of
diversity-defocused point-spread functions (PSFs), where multiple frames can be acquired at each defocus setting.
Multiple frames of data can be co-added in different ways; two extremes are in "image-plane space," to average the
frames for each defocused PSF and use phase retrieval once on the averaged images, or in "pupil-plane space," to use
phase retrieval on each PSF frame individually and average the resulting wavefronts. The choice of co-add methodology
is particularly noteworthy for segmented-mirror telescopes that are subject to noise that causes uncorrelated motions
between groups of segments. Using models and data from the James Webb Space Telescope (JWST) Testbed Telescope
(TBT), we show how different sources of noise (uncorrelated segment jitter, turbulence, and common-mode noise) and
different parts of the optical wavefront, segment and global aberrations, contribute to choosing the co-add method. Of
particular interest, segment piston is more accurately recovered in "image-plane space" co-adding, while segment tip/tilt
is recovered in "pupil-plane space" co-adding.